Pool and zone storage classes were wrapped in quotes by hand, which
does not escape a quote or a backslash in the value: a quote closes the
string early and the data-plane release values stop parsing, with a
YAML error that does not name the field. They are now quoted like the
volume and admin classes.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The volume and admin storage classes were written unquoted into the
values of the data-plane release. A class name made only of digits,
which Kubernetes allows, arrived there as a number, and a value with a
line break could add keys next to the field, past the schema that is
the tenant's input gate. Both are now quoted. lifecycle.storageClass,
new in this release, also carries the StorageClass name rules in its
schema, so a value that cannot name a class is refused when the
application is written instead of leaving the admin claim pending.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
volume.storageClass is picked for volume server data, and in a
MultiZone instance it can name a class whose volumes exist in one zone
or on one node. Reusing it for the admin volume would tie the admin to
that zone or node, so losing it would stop lifecycle enforcement for
the whole instance, a placement nobody chose for the admin. The admin
volume now takes lifecycle.storageClass, and the cluster default when
that is empty.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The daily walk that expires objects sends its deletes without an object
identity, and the s3 server skips its compare-and-swap when none is
given (walker_dispatcher.go and identityMatches, unchanged from 4.45
through 4.52). It does not re-check the object's age either, so a key
overwritten between the walk's listing and the delete loses its new
content. Backup buckets write unique keys and barely meet that window;
a tenant bucket that rewrites a fixed key does.
Turning enforcement on by default would also start applying, on
upgrade, every rule that was stored and ignored until now. Both are
retention changes an operator should choose, so the admin server and
the worker now run only where lifecycle.enabled is set. On by default
can return once the walk carries identity.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The dashboard reads an application's WorkloadMonitors by name through
its resource-map Role, so the admin and worker monitors the lifecycle
switch adds were invisible to the tenant: the components ran, but the
application page never listed them.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The admin volume and the two lifecycle pods are new demands on tenant
quota, so an upgrade into a namespace with no headroom can wait out its
timeout. Flux's default remediation then rolls the release back, and
with unlimited retries it does so on every attempt: each rollback drops
the [grpc.s3] section and the s3 restart annotation, each retry brings
them back, and the s3 pods restart twice per cycle until quota frees up.
RetryOnFailure keeps what was applied and retries it, as the parent
release that the apiserver builds for the application already does.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
SeaweedFS accepts and stores a bucket lifecycle configuration, but only
the s3_lifecycle job of its plugin worker acts on it, and that job is
scheduled by the admin server. Neither ran here, so a rule returned 200
and no object ever expired, on the platform backup bucket and on tenant
buckets alike, and the documented way to bound backup storage did
nothing.
Both now run in Simple and MultiZone instances, behind lifecycle.enabled
(on by default). A Client instance has no data plane of its own; its
rules are enforced where the master it connects to runs. The worker
registers for s3_lifecycle alone, so enabling lifecycle does not also
start vacuum, balancing or erasure coding. The admin keeps its job state
on a small volume (on volume.storageClass when set), and its UI gets a
generated password, because a tenant namespace admits traffic from the
whole cluster and the server is unauthenticated without one. The
data-plane chart keeps both off unless its parent turns them on, so it
never starts an admin without that password.
Existing rules start applying minutes after the upgrade: the admin's
scheduler runs the first pass within one idle cycle of its lifecycle
lane, then once a day, and objects the rules have already expired are
deleted on that first pass.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
## What this PR does
Seven guides under `docs/` were operator facing and written alongside
the code, and left in the source tree where no user looks. They are on
the website now, so this deletes them here and repoints every reference.
```
docs/operations/backup-classes.md -> operations/services/backup-classes
docs/operations/resource-presets.md -> guides/resource-management
docs/storage-immutability.md -> operations/storage-class-immutability
docs/operations/system-memory-limits.md -> operations/configuration/system-memory-limits
docs/gateway-classes.md -> networking/gateway-classes
docs/observability/distributed-tracing.md -> operations/services/monitoring/distributed-tracing
docs/operations/seaweedfs-capacity-for-backups.md -> operations/services/object-storage/capacity-for-backups
```
`docs/release.md`, `docs/deprecating-applications.md` and the two
SeaweedFS runbooks in `docs/operations/` stay. A workflow, `AGENTS.md`,
a migration and a bats suite read them by path, which is the test from
#3777.
References point at `https://cozystack.io/docs/v1/...`. The site
rewrites that short prefix to the latest version in `layouts/404.html`,
so the links follow releases and pin none. Backup Classes, resource
management and `storageClass` immutability exist in the latest version
already. The other four pages are only in `next/` until 1.7.0 is
promoted, and their links start to resolve then. The generated mongodb
files come from `values.yaml` through `make generate` with
cozyvalues-gen v1.7.0, the version pinned in
`.github/workflows/pre-commit.yml`.
Tests: helm unit tests for mongodb (151), postgres (209), vm-disk (114)
and kubernetes-nodes (223), go tests for backupcontroller, presets and
the operator, rd-presets-check, the migration-50 bats suite.
### Downstream repositories
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#763
Walked the rest of the trigger map against the diff. No provider
follow-up: the mongodb `values.schema.json` diff is one `description`
string, with no field added, removed or renamed, no enum, no default,
and no `kind`, `plural` or `release.prefix` change.
`packages/core/platform/values.yaml` changed in comments only and no
key, so the hand written platform-package table on the site is
unaffected.
Please merge after cozystack/website#763, the links in the code only
resolve once those pages deploy.
### Release note
```release-note
docs: seven guides move from the docs/ directory of this repository to cozystack.io (Backup Classes, resource presets, storageClass immutability, system component memory limits, GatewayClass selection, distributed tracing, SeaweedFS capacity for backups), and in-repo references point there.
```
## What this PR does
The tenant etcd chart has its own copy of
`etcdHighNumberOfFailedGRPCRequests`, and both of its rules count every
`grpc_code` other than `OK` as a failure. `grpc_server_handled_total`
counts a Watch stream once, when it ends, and the streams of a
kube-apiserver that goes away end with `Canceled`. So whenever an API
server on a tenant etcd stops, the Watch method of every member reads as
100 percent failed and the critical rule fires on all of them.
Both rules now count only
`Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded`,
the list the etcd mixin has used since etcd-io/etcd#13127 and the one
the platform copy in
`packages/system/monitoring-agents/alerts/etcd.yaml` already uses. The
grouping `BY (job, instance, grpc_service, grpc_method)`, the
thresholds, `for` and the annotations stay as they are, so the alerts
keep their labels.
Seen on a v1.6.5 cluster while a Kubernetes cluster on a tenant etcd was
created and deleted a few minutes later. vmalert had the critical alert
firing for `grpc_method="Watch"` on all three members with the value
100. At that point
`rate(grpc_server_handled_total{grpc_code!="OK"}[10m])` was 0.63 for
Watch with `Canceled`, and the only other non-OK codes were
`FailedPrecondition` on `RoleGet` and `UserGet` at 0.0017 each, which is
one call each. The same query an hour earlier returned nothing. The
platform copy runs in the same vmalert over the same series, and none of
the firing alerts came from it. Each alert also showed up seven times
there, once per tenant copy of these rules, because the root tenant's
vmalert loads the rules of every tenant namespace. This PR does not
change that.
`hack/etcd-grpc-alerts.bats` renders the chart's PrometheusRule with
`helm template` and replays it through `vmalert-tool` v1.136.0 pinned by
digest, the same image #4963 uses.
`hack/testdata/etcd-grpc-alerts/unittest.yaml` has three cases with
seven checks. Watch streams that end only with `Canceled` must not fire
at 6, 12 and 30 minutes. `Range` calls failing with `Unavailable` at 25
percent must fire the critical rule at 7 minutes and both rules at 12,
with the labels and the summary pinned. Both together must fire only for
`Range`. Against the rules on main the suite fails 4 of the 7 checks,
all of them on Watch alerts with the value 100. Dropping `Unavailable`
from the list, or grouping by `job` and `instance` only, fails 3 checks
each.
```console
$ bats hack/etcd-grpc-alerts.bats
1..1
ok 1 tenant etcd failed gRPC request alerts pass their vmalert-tool cases
```
The same run on the rules from main prints `not ok 1` and the four
failing checks. With Bats 1.14.0 at the commit CI pins, the suite passes
together with `bats-strict-setup`, `bats-no-negated-assert`,
`bats-no-exit-trap`, `bats-runner-coverage`, `bats-runner-contract`,
`alert-severity-contract` and `image-pin-consistency`, 81 of 81. Like
the migration suites it needs docker, and only the image pull touches
the network. Helm 3.17.3 and 4.2.4 render the same rules. I have not run
the new rules in vmalert on a live cluster.
The rules file is the same on release-1.6 and release-1.7. The commit
cherry-picks onto release-1.6 without conflicts, and that backport is a
separate PR with one more commit that drops the `load test_helper` line,
because release-1.6 runs `hack/*.bats` through `hack/cozytest.sh`.
### Screenshots
Not a UI change.
### Downstream repositories
A rule template in `packages/extra/etcd`, a Bats suite under `hack/` and
its test data. No values, schema, README, image or
`ApplicationDefinition` change, and the website does not mention these
rules, so no trigger map entry applies.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(etcd): the tenant etcd alert etcdHighNumberOfFailedGRPCRequests now counts only the gRPC codes that the etcd mixin and the platform copy of the alert count, so the Canceled watch streams of a Kubernetes API server that stops no longer fire it on every etcd member.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Refined etcd gRPC failure alerts to target specific failure status
codes, reducing alerts for non-failure responses while preserving
existing thresholds and timing.
- Added coverage for canceled watch streams, unavailable range requests,
and combined failure scenarios, including warning and critical alert
evaluations.
- **Tests**
- Added integration validation for rendering etcd alert rules and
running them through the alert-testing tool, including failure handling
when prerequisites are unavailable.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Seven guides in docs/ were written next to the code and left in the
source tree, where no operator looks. The website carries them now, so
a second copy here only lets the two drift again: the Backup Classes
page had already fallen behind its original.
docs/release.md, docs/deprecating-applications.md and the two SeaweedFS
runbooks in docs/operations stay. A migration, a bats suite, a workflow
or AGENTS.md reads each of them by path, which is the test for keeping
a document in this repository.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The tenant etcd copy of etcdHighNumberOfFailedGRPCRequests counted
every code other than OK. A Watch stream is counted once, when it ends,
and the streams of a kube-apiserver that stops end with Canceled. So
whenever an API server on that etcd goes away, the Watch method of
every member reads as 100 percent failed and the critical rule fires
on all of them.
The rules now count the codes the etcd mixin counts since
etcd-io/etcd#13127, the same list the platform copy in
monitoring-agents already uses. The grouping by method stays, so the
alert labels do not change.
Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
Five guides in docs/ were written next to the feature code that needed
them and left in the source tree, where no operator looks. They are on
the website, so a second copy here only drifts: edits kept landing in
docs/ after the website copy was made.
docs/vm-external-vlan.md is split rather than moved. Its section on
staging the reference CNI plugins is a contract that
hack/cni-plugins-staging-contract.bats asserts against the multus
Dockerfile, the daemonset manifest and the platform values, so that half
is package documentation and goes to packages/system/multus/README.md,
which the suite now reads. The operator half goes to the website.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The hourly defrag CronJob runs etcd-defrag with its defaults, which compact
to the latest revision before defragmenting. Every cluster already compacts
itself with a periodic five-minute retention, so the second compaction
either fails with "required revision has been compacted" when nothing was
written since, or drops the history that retention keeps for watchers to
resume from.
The same defaults evaluate dbQuotaUsage in the defrag rule against 2 GiB
rather than the quotaBackendBytes the cluster runs with, so the 80 percent
threshold fires at a different database size than intended.
Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
With Cilium at 1.20.2 the tenant gateway comments, README and one
status message still explained behaviour as that of "the pin" at
1.19.5. Several of those claims no longer hold: a Gateway with two
routed passthrough ports now gets an Envoy listener per port, Cilium
judges a route's kinds against the listener the route names, and the
shipped operator accepts an HTTPS and a passthrough listener on one
port when the passthrough hostname is strictly narrower.
Behaviour that is unchanged is re-anchored on the 1.20.2 source,
behaviour that is gone is stated as history, and the refusal of one
hostname on two passthrough ports no longer blames a Cilium limitation
the shipped release does not have.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The TenantGateway controller withheld the HTTPS-terminate listener for a
name a passthrough listener answers only once a TLSRoute was attached
there and resolved a backend, mirroring how Cilium builds its filter
chains. That kept a served endpoint up on Cilium 1.19.5, but it tied
every route's status to one implementation's translator and needed
Service and ReferenceGrant watches to land the deferred decision.
The shipped operator accepts a wildcard terminate listener beside a
narrower passthrough listener on 443 and keeps conflicting every other
overlap. A terminate listener for the passthrough name itself is not
narrower, so on 443 the rule now protects the dataplane outright; on a
native port Gateway API would serve the pair, and the rule holds as a
policy that keeps a platform name with its passthrough listener. Reading
it off the spec makes the outcome the same on every implementation.
Under http01 an HTTPRoute claiming such a name gets no listener and no
certificate whether or not a TLSRoute is attached. A route whose every
hostname is reserved reads Accepted=False with
NoMatchingListenerHostname; one that also claims a free hostname stays
Accepted and is served there, as Gateway API accepts a route whose
hostnames intersect at least one listener. dns01 and existingSecret are
unchanged: their wildcard listener still covers such a name, and the
passthrough listener answers it by its exact server name.
The chart writes the passthrough services on the publishing tenant
alone, so the names reserved by default are the ones the platform
publishes there through its own TLSRoutes. The shape an upgrade
changes is an HTTPRoute on such a name whose platform TLSRoute forwards
nowhere, because it never attached (the package owning the service is
not installed) or because none of its backends resolve: that route kept
a terminate listener and now gets none.
The backend resolution, its watches and the RBAC rules that served them
go away.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The chart wrote the tlsPassthroughServices default onto every non-edge
tenant's TenantGateway, so every tenant Gateway carried tls-api,
tls-vm-exportproxy and tls-cdi-uploadproxy. The TLSRoutes that serve
those names are platform routes: the packages owning the services render
one per name in publishing.exposedServices while gateway.enabled is on,
parented to the publishing tenant's Gateway alone. Everywhere else no
shipped route attached to the listeners, and a passthrough listener
holds its name whether or not anything is routed through it.
The chart now writes the field only on the publishing tenant, with the
same gateway.enabled switch and the same exposedServices filter those
routes use, so a listener exists exactly where a platform route can
attach. A TLSRoute attached by hand to one of those listeners on another
tenant loses its parent.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The refusal existed because Cilium 1.19.5 served every Gateway port
from one Envoy listener matched by SNI alone, so the "*.<apex>"
terminate listener of the wildcard modes and a passthrough listener
under the apex could reach each other's backend. The shipped Cilium
1.20.2 gives each port its own Envoy listener when SNI overlaps across
ports, and Gateway API treats listeners on different ports as
compatible, so the pair is served as declared. edge still refuses the
field: it renders no TLS listener for an entry to bind to.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The limiter refuses for good any request it counts above burst, and
burst was the receiver's 4 MiB cap. A request grows on its way to the
limiter: the sampler adds a tracestate to every span it keeps, and the
limiter counts a request as the collector re-encodes it, which writes
the empty value of every attribute the wire may omit, so a valid
request of spans with empty-valued attributes is counted 1.4x. A
request the receiver admitted near its cap was therefore refused on
every retry.
The limiter now runs right after memory_limiter, so it counts the
request before sampling. The receiver caps a request at 3 MiB and
burst stays at 5 MiB, which holds 1.4x the cap; a larger burst would
also be what each pod start lets through above the rate. The rate is
counted before sampling, so the store keeps only the sampled share of
it. The texts say that a request above the cap gets a 400, which
senders do not retry, and that each collector rollout a tenant causes
lets that burst through again on every pod.
The replica bound that keeps that allowance small now applies only to
a central tenant's collector, whose render fails above ten replicas: a
collector writing to the tenant's own store has no rate, and the
replica counts it shipped with stay valid.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
A tenant's Grafana pod may reach the shared traces store's vmauth, and
the Secret a tenant names in oidc.customConfig.secretRef was mounted
into it whole. The apps API does not prune keys the Monitoring schema
leaves out, so a tenant could also set grafana.image: an image of its
choosing, with the collector's Secret mounted, writes past the rate
limit the collector enforces.
Grafana's image is the platform's, like the collector's, and for a
central tenant only the auth.ini key Grafana reads is mounted from the
named Secret. Other tenants have no collector login to keep out and
keep the whole Secret, which an auth.ini reading a sibling key with
$__file{} relies on.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The collector holds a central tenant to its write rate, but the one
login it writes with was also the login of the tenant's Grafana
datasource, and it could both read and write. vmauth cleans the request
path before it picks a route, so any user the tenant admits to its
Grafana, a Viewer included, could POST through the datasource proxy to
../insert and write into the shared store past the limiter.
Each central tenant now has two VMUsers on its one account: the
collector's, which may only write, and Grafana's, which may only read,
each with its own chart-owned Secret and password. A namespace name
holds no dot, so the reader's username never matches another tenant's.
On upgrade the datasource moves to the new login in the same release
that creates it, so reads may fail until vmauth loads the new user.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Each collector pod starts with a full token bucket, and burst was at
least 64 MiB, the largest request vtinsert accepts. A tenant controls
when its collector pods start, through any Monitoring change or by
deleting them, and how many there are, so it could write tens of
mebibytes above its rate per pod start and gigabytes per rollout of
many replicas, which the two-day bound does not survive.
The limiter only sees requests the receiver admits, so in central mode
the receiver now caps them at 4 MiB, decompressed for HTTP, and burst
drops to that floor; a pod start lets at most 4 MiB through above the
rate. The replica count is capped at 10. The texts now say what the
default rate guarantees: one tenant writes at most a default store's
disk cap over two days counted uncompressed, rather than that it
cannot fill it, and they name the per-pod-start allowance and that a
sender pinned to one replica gets that replica's share.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The design has the per-tenant collector rate-limit its tenant, so that
a noisy or hostile tenant cannot exhaust a shared backend. The disk cap
alone does not do that: VictoriaTraces always keeps the last two days,
so a tenant writing faster than the cap allows over two days fills the
disk for every tenant on the store.
The platform sets the rate with
monitoring.tracingCentralTenantBytesPerSecond, and it reaches every
tenant in _cluster through the cozystack-values Secret. A change there is a change in the values each
monitoring HelmRelease reads, so Flux upgrades them all; a rate read
with lookup at render time would never reach a tenant whose release is
already in sync. The platform writes the key only when it is set, since
every release in every tenant reads that Secret. Without it the
collector holds the tenant to the rate of a default 10Gi store.
Each central tenant's collector splits the rate across its replicas
and refuses data above it before the batch processor, so the sender
sees the refusal. An OTLP/HTTP sender gets a 429 and retries; the
limiter answers gRPC with RESOURCE_EXHAUSTED and no RetryInfo, which
the OTLP specification makes final, so central tenants should export
over HTTP.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The apps API defaults a Monitoring spec from its schema but does not
prune keys the schema leaves out, and the Monitoring chart hands its
values to the inner chart as they are. A tenant could therefore set
tracingCollector.image and run any collector it liked in its
namespace; with a per-tenant rate limit in that collector, the same
key would let a tenant drop the limit. The image is the platform's,
so it is no longer passed through.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
## What this PR does
This moves the root module's `k8s.io/*` from v0.35 to v0.37.1 and
controller-runtime from v0.23 to v0.25.1, and drops the
`k8s.io/apimachinery` fork on the way. Supersedes #3001, #4467 and
#4472; those were a stack of the same commits.
The four commits are meant to be read one by one, and each builds and
passes the tests on its own:
1. **Plain watches stop getting the initial-events-end bookmark.** The
apiserver defaults `sendInitialEvents=true` onto any watch with an empty
or `0` resourceVersion and leaves bookmarks off, and our aggregated
storages took that flag alone as a WatchList request. From
`k8s.io/apiserver` v0.36 the watch handler compiled into cozystack-api
panics on that bookmark, so `kubectl get -w` on any cozystack-api
resource would die after the bump. The marker now needs both flags, the
same condition the watch cache uses. There are tests per storage and one
through the real handler chain.
2. **The apimachinery fork goes.** Every apps kind shares the
`Application` Go type, and stock `Scheme.ConvertToVersion` picks the
first kind registered for it. The fork patched that; the apps group now
encodes through a serializer that keeps the kind the object carries. A
handler-chain test covers create, get, list, update, watch, apply and
table for two kinds and fails without it.
3. **MutatingAdmissionPolicy stays out of admission.** From v0.36 the
plugin is on by default and refuses every write until it has synced
`mutatingadmissionpolicies`, which cozystack-api has no RBAC to list and
which clusters before 1.36 do not serve. Without this, every create,
update and delete would fail after the bump. It is also re-applied in
`Complete`, since `--disable-admission-plugins` replaces the list.
4. **The bump itself**, with the regenerated clients, openapi and
strategy CRDs.
Commits 2 and 4 are where the review effort goes; 1 and 3 are small.
Known changes in behaviour:
- Without the fork, the row objects of a Table requested with
`includeObject=Object` carry the first registered app kind. kubectl asks
for that only with `--sort-by`, and it sorts on fields, not on the kind.
Nothing at this layer can fix it, because the handler converts through
`APIGroupInfo.Scheme`, a concrete `*runtime.Scheme`. Admission webhooks
and ValidatingAdmissionPolicy still see the right kind.
- The alpha PodSpec field `workloadRef` became `schedulingGroup`
upstream. Once the new strategy CRDs are applied, the apiserver prunes
`workloadRef` from stored strategies it serves, and their next write
erases it. It only ever worked behind an alpha feature gate.
The branch is rebased onto the Renovate rollup (#4447), with `go mod
tidy` and codegen rerun on top of it.
### Screenshots
No UI changes.
### Downstream repositories
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
chore(deps): cozystack-api and the controllers build against the Kubernetes 1.37 libraries without the apimachinery fork. Plain watches no longer receive the WatchList initial-events-end bookmark, and objects in Table rows requested with includeObject=Object may carry another app's kind.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Application API responses now preserve the requested resource kind
across operations, including lists and watches.
* Backup strategy resources support additional Kubernetes pod settings,
including probe protocols, volume ownership options, and scheduling
groups.
* Backup strategy schemas include updated Kubernetes pod fields and
descriptions.
* **Bug Fixes**
* Initial-event watch bookmarks are sent only when requested; plain
watches continue without an ending bookmark.
* Mutating Admission Policy is disabled by default, including when
admission plugin options are customized.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The computeplane module deploys the kubernetes chart into the tenant
namespace, and that chart's dashboard Role lets the tenant's "use"
subjects read <release>-admin-kubeconfig. Withholding the Secret from
the tenant secrets view does not help when RBAC grants it directly, so
any tenant user at "use" level or above could fetch the cluster-admin
credential of a cluster the module promises to keep operator-owned.
The kubernetes chart now drops that rule when the internal value
withholdAdminKubeconfig is set, and the module sets it in the cluster
HelmRelease's inline values. The tenant cannot reach those values: the
module reads only nodeGroups from them and splices an allowlisted
subset into the pool releases, and Flux applies spec.values over
valuesFrom. A regular kubernetes app keeps the rule, so its dashboard
can still offer the kubeconfig.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The tracingStorages description said a tenant with shared-central
tracing gets the collector without a local entry, which reads as if
tracingCentral alone deploys it. tracingCollector.enabled still
decides whether it renders.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
The distributed tracing design requires each traces backend to be
disk-bounded, not only age-bounded, so a full volume cannot block
ingest. retentionDiskUsageBytes was optional, so a store without it
filled its volume and stopped taking spans; with a shared-central store
that stops ingest for every tenant at once.
A store without a byte cap now drops its oldest day once its
filesystem is 80% full. VictoriaTraces refuses to start with both
limits set, so a byte cap, when given, replaces the percentage one.
It always keeps the last two days, which the values text now says.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
A per-tenant traces backend costs every tenant its own VictoriaTraces
stack. A tenant can now send new spans to one shared store in
tenant-root instead, switched on by _namespace.tracingCentral, which
the tenant chart publishes and the apps API does not let an
application set. tenant-root decides whether it hosts that store, with
tracingCentralHost next to a cluster-mode store named generic.
VictoriaTraces takes the tenant account from request headers and does
no authorization of its own, so other tenants reach the store only
through a vmauth in tenant-root; tenant-root itself, with whoever it
admits to its Grafana, stays inside the trust boundary. Each central
tenant gets a VMUser that pins an account derived from its namespace
name and UID for both writes and reads and caps its requests in
flight, which the collector's two export senders stay under; its
collector exports and its Grafana reads with credentials the chart
generates and keeps, and vmauth overrides any account a client sends.
The chart owns those credentials because the operator publishes a
VMUser's own Secret only once a VMAuth selects it; the collector rolls
when they change, since it reads them at start. The vmauth internal
routes, whose /metrics names every tenant's user, are served on a port
tenants cannot reach, and the vmauth is sized by a VPA, spread across
nodes under a disruption budget, has a WorkloadMonitor like the other
monitoring workloads, and may reach the apiserver, where its
config-reloader reads its config Secret.
Renders refuse the states in which spans would be dropped with every
object Ready: a central tenant while tenant-root hosts nothing,
tenant-root dropping the hosting while central tenants remain, and
the host flag without a store to host. Each refusal reads the other
side at render time, so a concurrent opt-in and stop-hosting can pass
both; the values text names that race and its recovery. At run time
tenant-root's release is not Ready while the vmauth has failed.
Only the collector's export moves: local stores the tenant still lists
keep rendering with their datasources, because the switch belongs to
the parent tenant and the stores to this one, and the one-store limit
of the collector holds in both modes, since turning central off makes
it export locally again. tenant-root never counts as central.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Platform alerts need an email delivery path alongside Alerta without
exposing arbitrary namespace Secrets through tenant-controlled values.
SMTP credentials therefore use a fixed Secret name.
Alertmanager parses the addresses and connects to the SMTP host only
when it sends a notification, so a malformed value there drops alerts
instead of failing the release. The chart validates these fields when
it renders. Addresses must be literal dot-atom mailboxes, and a domain
literal must hold a dotted-quad IPv4 address, because IPv6 literal
syntax differs across the supported Go mail parsers. A bracketed
smarthost must be a valid IPv4 or IPv6 address, and its port must be
between 1 and 65535 without leading zeros. The repeat interval follows
the Alertmanager duration grammar and range, so a value that renders is
one Alertmanager loads.
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
Assisted-by: LLM
cozy-lib.resources.sanitize divides the CPU limit by cpuAllocationRatio
and rendered the quotient as a bare float. When the ratio does not
divide the value evenly the result is finer than a milli-CPU, and the
VPA admission webhook rejects it outright:
admission webhook "vpa.k8s.io" denied the request:
minAllowed: CPU [62500u] must be a whole number of milli CPUs
packages/extra/etcd routes its VPA minAllowed/maxAllowed through the
helper, so 250m / 4 = 62.5m left the etcd HelmRelease looping
UpgradeFailed -> RollbackFailed forever: the rollback target renders the
same invalid value. Since v1.6.0 HelmReleases wait on each other, so
that one stuck app also blocks tenant-root, cozystack-basics and
kubevirt-cdi-operator, and without CDI the Talos worker rollover cannot
import the worker OS image. Only the divisors of 250 worked as ratios,
which is an artefact of one chart's hardcoded floor rather than a real
constraint on the operator.
The request is rounded up, never down, so it stays at or above the
intended fraction of the limit. It is rounded only when it has to be:
if the milli-CPU value equals its own ceiling, the float keeps its
current rendering, so nothing changes on the default ratio of 10 or on
any ratio that divides evenly; otherwise it renders as the ceiling in
m. The comparison is exact, with no tolerance. sprig's divf and mulf run
on shopspring/decimal rather than float64, so mulf (divf 0.25 10.0)
1000.0 renders exactly 25 and there is no binary-float noise on this
codepath to absorb. A tolerance would reintroduce the bug: a value
whole-milli only to within it, such as 100.0000004m at ratio 1, would
read as exact and render "0.1000000004", rejected for the same reason
62.5m is. The same predicate covers sub-milli input, since ceil carries
any positive request to at least 1m, and a zero request stays "0".
Fixes#2917
Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Add per-template guard suites so each guarded hostname site is exercised
on its own: harbor harbor.yaml and httproute.yaml, bootbox machines.yaml,
seaweedfs cosi-deployment.yaml, bucket deployment.yaml and httproute.yaml,
monitoring alerta.yaml, grafana httproute.yaml and the grafanaHost helper
via oidc-keycloak.yaml. Each suite renders only its target template, so
removing that file's guard reddens it. Also add a postgres positive case
that renders the Certificate with an apex set (external=true).
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Add host_guard_test.yaml for postgres, vpn, bootbox, seaweedfs,
monitoring and bucket. Each renders its guarded template with a
non-empty injected _namespace whose host is empty and no per-app host
override, and asserts the ns-host-required failure, plus a positive
render case with the apex set. Completes guard coverage alongside the
existing harbor, qdrant and kubernetes tests.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Charts that publish a service compose their hostname as
<sub>.<_namespace.host>. When the injected tenant apex is empty and no
per-app host override is set, they used to emit a malformed trailing-dot
hostname (e.g. "harbor.") into Ingresses, HTTPRoutes and Certificate
dnsNames silently. Source the apex through cozy-lib.ns-host-required at
each hostname-emitting site so the render fails with a clear message
instead, matching the gateway path.
Covered: harbor, postgres (external), qdrant (external), kubernetes
(control-plane endpoint), vpn, seaweedfs, bootbox, monitoring
(grafana, alerta, oidc client), and system/bucket (adds the cozy-lib
subchart symlink it lacked). The guard is placed at the point each
resource is actually emitted, so standalone renders, no-exposure
tenants and the kubernetes no-etcd beacon path are unaffected. The
post-delete cleanup hooks are left unguarded so an empty apex cannot
block uninstall.
Behavior change: a tenant with an empty injected apex and no host
override that enables one of these components now fails its render
(HelmRelease NotReady) instead of deploying a broken hostname. In
practice the apex is always injected (root has root-host, children
inherit it), so this only trips a genuinely misconfigured tenant.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
The package pinned linux/amd64 because its assets were amd64 only.
With arm64 profiles it follows PLATFORM like every other image.
The installer is published as one index under every tag. skopeo
cannot assemble an index from the imager's docker archives, so the
archives go into one OCI layout and a small script writes the index
there for skopeo to push. buildx could build it from OCI layout
sources, but the release job builds on the classic docker driver,
which has no such source.
matchbox carries the kernel and initramfs of both arches whatever
PLATFORM is, rather than those of TARGETARCH: the node matchbox runs
on is not the arch of the machines it boots, so a per-arch image would
hand an amd64 machine an arm64 kernel whenever the pod lands on an
arm64 node. The amd64 assets stay at their old paths, and a group
keyed on the arch iPXE reports sends arm64 machines to the arm64 ones.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Assisted-by: LLM
The root module's k8s.io/* move from v0.35 to v0.37.1 and
controller-runtime from v0.23 to v0.25.1, its release for Kubernetes
1.37. Two things held them back: the apimachinery fork, which had to be
rebased onto every minor, and the watch handler that from
k8s.io/apiserver v0.36 panicked on the initial-events-end bookmark the
aggregated storages sent to plain watches. Both are gone.
The rest of the upgrades are what these releases require: cel-go
v0.29.2, the etcd client v3.7.0, kube-openapi, structured-merge-diff and
the x/ modules. code-generator moves with the libraries.
Regenerating the strategy CRDs that embed core types adds the
Kubernetes 1.37 fields and follows one removal: the alpha PodSpec field
workloadRef became schedulingGroup upstream. Once the new CRD is
applied, the apiserver prunes workloadRef from every stored strategy it
serves, and the object's next write erases it from etcd. It only ever
took effect behind an alpha feature gate.
The gateway README and the tenantgateway CRD tests cite the
apiextensions-apiserver list-type ratchet at the pinned version; it is
byte-identical in v0.37.1, so only the version they name changes.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Phase 2 of distributed tracing (design cozystack/community#38, epic
#3761), built on the phase-1 backend (#3762, merged): a per-tenant
**OpenTelemetry Collector** — the app-facing OTLP ingest gateway. It
lives in the tenant namespace (like a per-tenant vmagent), so
app→collector traffic is intra-namespace and never crosses the tenant
boundary. No OTel operator exists in the platform, so this is a plain
Deployment + ConfigMap + Service (the fluent-bit pattern).
## Changes
- **`templates/vtraces/collector.yaml`** — OTLP receivers (gRPC `:4317`,
HTTP `:4318`) → `memory_limiter` + head sampling
(`probabilistic_sampler`, default 10%) + optional `attributes/redact`
(PII/secret control point) + `batch` → `otlphttp` exporter to the
`tracingStorages` backend (`vtinsert:10481` cluster / `vtsingle:10428`
single, endpoints per upstream docs). PSS-restricted securityContext,
digest-pinned image via `cozy-lib.image`, config-checksum roll.
- **app-facing Service `otel-traces`** (4317/4318).
- **`tracingCollector` values** — `enabled` / `replicas` /
`samplingPercentage` (schema-bounded 0-100) / `redactAttributes` /
`resources`. A **fail-guard** rejects an enabled collector fronting more
than one `tracingStorages` backend (it exports to a single backend by
design).
- **`otel-traces` WorkloadMonitor** and dashboard resource-map entry,
rendered under the collector's own gate. The same resource-map Role also
gets the pre-existing `alermanager` typo corrected to `alertmanager`, so
the alertmanager WorkloadMonitor is readable from the dashboard.
- helm-unittest coverage for the collector and its WorkloadMonitor;
regenerated `values.schema.json` / `README.md` / `monitoring-rd`
ApplicationDefinition.
## Verification
- `helm unittest` — `system/monitoring` 110, `extra/monitoring` 15, all
pass.
- `helm template` renders cleanly; `make generate` idempotent.
- Live check on a test cluster (single-mode backend): the collector
starts with `redactAttributes` containing a numeric, a colon-bearing and
an empty key; an OTLP/HTTP span sent to `otel-traces:4318` is accepted
and is queryable from the backend through the Jaeger API, with the
redacted attributes removed and the others kept.
- Local `/branch-review` — **LGTM** (one blocker + four recommendations
fixed, each with a test: multi-backend guard, memory_limiter cgroup
limit, configurable replicas, sampling bounds, trust-boundary wording).
## Scope / deferred (epic #3761)
`AccountID`/`ProjectID` header injection for the shared-central topology
(phase 4) and engine integrations (phase 3) are follow-up PRs.
### Screenshots
N/A, no UI change.
### Downstream repositories
The diff touches only `packages/extra/monitoring`,
`packages/system/monitoring` and `packages/system/monitoring-rd`. The
website's monitoring reference page is regenerated from the package
README by the release bot, and the Terraform provider models only the
tenant's `monitoring` flag, not these values, so no follow-up is needed.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
feat(monitoring): deploy a per-tenant OpenTelemetry (OTLP) collector for traces once a VictoriaTraces backend is configured. Applications send OTLP to the in-namespace `otel-traces` Service (gRPC `:4317`, HTTP `:4318`); the collector applies head sampling (`samplingPercentage`, default 10%) and optional attribute redaction (`redactAttributes`) before exporting to the backend. Off by default (no backend configured).
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added configurable trace storage in cluster or single-instance modes,
with retention settings and Grafana datasources.
* Added an enabled-by-default OpenTelemetry collector with configurable
replicas, sampling, attribute redaction, and resource limits.
* Added workload monitoring and automatic resource recommendations for
configured trace components.
* Added readiness health checks for both tracing backend modes.
* **Bug Fixes**
* Corrected dashboard permissions for alert monitoring and trace
collector resources.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
I removed the two monitoring OIDC BATS suites that had no runner after
the Chainsaw migration, and brought the runner audit and wrapper OIDC
test from [#3778](https://github.com/cozystack/cozystack/pull/3778).
Also added checks for audience mapper and `OIDC_MODE=System`. The named
`KeycloakRealmGroup` test documents coverage the document-count and kind
assertions already provided.
Runner audit reports unused BATS suites, missing files in runner paths,
and unresolved Make variables (including `$(E2E_SUITE)`).
Live Monitoring API admission, Grafana operator reconciliation and mode
transitions are not tested here.
### Validation
- `make -j4 unit-tests SHELL=/usr/bin/dash` passed.
- Monitoring Helm tests: 53 system-chart tests and 10 wrapper-chart
tests passed.
- Runner audit: 12 cases passed under dash and Bats 1.14.0. The full
shell syntax and BATS sweep passed under dash too.
- Three chart mutations and seven runner mutations failed as expected.
### Downstream repositories
- [x] No downstream repository is affected by this change
### Release note
```release-note
NONE
```
The Role listed the WorkloadMonitor as 'alermanager', so the real
'alertmanager' monitor was never granted: a tenant dashboard is
forbidden from reading its status and the alertmanager health tile
breaks. The test pins the correct name and the absence of the typo.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Applications need one in-namespace OTLP endpoint that also enforces
sampling and redaction before spans reach the VictoriaTraces backend.
No OpenTelemetry operator ships with the platform, so the collector is
a plain Deployment, ConfigMap and Service ('otel-traces', gRPC 4317,
HTTP 4318) in the tenant namespace; app-to-collector traffic therefore
never leaves the tenant, and isolation comes from the existing tenant
network policies.
The collector renders only when it is enabled and a tracingStorages
backend is configured. Tracing is off by default, and an empty backend
list is a clean off-state rather than a render failure; an app exporter
that finds no Service simply retries. It writes to the first backend
only and rejects a list with several, instead of silently dropping
spans for the others.
redactAttributes is the PII and secret control point, so its keys are
rendered quoted and empty keys are dropped: an unquoted key with YAML
structural characters, or an empty one, makes the collector fail at
startup, and a key YAML re-types (numbers, booleans, null) silently
escapes redaction. replicas has a lower bound of 1 so a bad value fails
schema validation instead of the whole release at apply time; a zeroed
collector is expressed with enabled: false.
The collector gets a WorkloadMonitor and a dashboard-resources Role
entry under the same render gate, so it is part of tenant workload
accounting like every other workload in the chart.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
## What this PR does
Phase 1 of distributed tracing (design: cozystack/community#38; epic:
#3761): a **VictoriaTraces backend and Grafana traces datasource** in
the monitoring stack, mirroring the logs (`VLCluster`) implementation.
No new operator: `VTCluster`/`VTSingle` already ship in
victoria-metrics-operator.
- **`tracingStorages`** values in `extra/monitoring` (annotated,
schema-generating) and `system/monitoring` (defaults), per-entry `mode:
cluster|single`. **Empty by default, so tracing is off out of the box**:
nothing produces spans until the collector ships, so no backend is
provisioned until an operator adds an entry.
- **`templates/vtraces/vtraces.yaml`**: `VTCluster` (default) or
`VTSingle` (edge/dev/small); `retentionPeriod` plus optional
`retentionMaxDiskSpaceUsageBytes`; a release-scoped **PVC cleanup
label**, reused by the existing post-delete hook, so no hook change is
needed.
- **Render-time validation** (`_helpers.tpl`): the installed
VictoriaTraces CRDs are the trimmed ones, so a bad value would otherwise
reach the operator and fail after admission. Each `tracingStorages`
entry is checked before anything renders: `name` must be an RFC 1123
label of at most 42 characters (the `vtstorage-<name>-<hash>` pod
revision label is capped at 63) and unique across the list, `mode` must
be `cluster` or `single`, `storage` must be a positive quantity (a zero
size makes the operator mount an EmptyDir in single mode, and a negative
one fails when the PVC is created), and `retentionDiskUsageBytes` must
be a VictoriaMetrics `BytesString` (the operator rejects the
Kubernetes-quantity `8Gi` grammar that `storage` uses). The `storage`
field is also typed as a quantity in the schema, and the rendered name
is quoted so a numeric name stays a string.
- **`templates/vtraces/grafana-datasource.yaml`**: a Jaeger-typed traces
datasource per entry (built-in Grafana core datasource, no plugin).
- **VPA**: a `VerticalPodAutoscaler` for each trace component
(`vtinsert`/`vtselect` Deployments, `vtstorage` StatefulSet, `vtsingle`
Deployment), matching the metrics/logs sizing, with `updateMode:
Initial`.
- **Readiness gate**: `VTCluster`/`VTSingle` `healthCheckExprs` on
`status.updateStatus`, alongside the existing `VLCluster` gate.
- **VictoriaLogs datasource fix**
(`templates/vlogs/grafana-datasource.yaml`): the range over
`logsStorages` emitted no `---` between documents, so two or more
entries collapsed into one document and only the last datasource
survived. Each entry is now its own document.
- **Tests and schema**: helm-unittest suites for off-by-default, cluster
vs single, datasource type and both query URLs, retention, PVC label,
VPA targets, every validation guard with its boundaries, and per-entry
vlogs datasource documents, plus regenerated `values.schema.json`,
`README.md` and the `monitoring-rd` ApplicationDefinition.
#### Upgrade impact
With default values the only rendered object that changes is the
`monitoring-system` HelmRelease: it gains the `VTCluster`/`VTSingle`
`healthCheckExprs` and `tracingStorages: []`. Flux evaluates those
expressions only against objects in the release inventory, so they are
inert while tracing is off. No VictoriaTraces object is created on
upgrade. Installs with two or more `logsStorages` entries additionally
gain the `GrafanaDatasource` objects that were previously dropped (see
the VictoriaLogs fix above); nothing else under VictoriaLogs or
VictoriaMetrics changes.
#### Verification
- `helm unittest`: `system/monitoring` (89 tests) and `extra/monitoring`
(11 tests) pass. Each validation guard and each branch of the
zero-storage pattern was mutation-checked: reverting it turns its own
test red.
- `helm template` renders `VTCluster`/`VTSingle`/datasource/VPA cleanly.
`tracingStorages: []` (the default) renders nothing. An invalid,
duplicate or over-long `name`, a zero or negative `storage`, or a bad
`retentionDiskUsageBytes` (`8Gi`) is refused at render with an
actionable message; a numeric `name` renders as a string. `make
generate` is idempotent.
- **Datasource URLs verified against upstream docs** ([VictoriaTraces
and
Grafana](https://docs.victoriametrics.com/victoriatraces/querying/grafana/)):
cluster `vtselect:10471/select/jaeger`, single
`vtsingle:10428/select/jaeger`. `jaeger` is the stable, built-in Grafana
datasource; the Tempo Query API (`/select/tempo`, `type: tempo`,
VictoriaTraces v0.9.4+) has a richer TraceQL UX but is
upstream-experimental, so Jaeger is the default. Both URLs are pinned by
unit tests.
- The `retentionDiskUsageBytes` pattern, the zero-storage check and the
name limit were checked against victoria-metrics-operator v0.68.4
(`BytesString.UnmarshalJSON`, `isStorageEmpty`, the `vtstorage-<name>`
StatefulSet name) and the Kubernetes `ControllerRevisionName` naming.
#### Scope / deferred (tracked in #3761)
Per-tenant OTLP Collector, engine integrations, shared-central topology,
and docs are follow-up PRs. This PR intentionally does **not** enable
the OTLP/gRPC listener: apps can use `vtinsert`'s native OTLP/HTTP, and
the gRPC listener lands with the collector.
Builds on the now-merged design proposal cozystack/community#38.
### Screenshots
Not a UI change.
### Downstream repositories
Walked against the diff, no box ticked yet:
- cozystack/website: `monitoring` is in the website `Makefile`
`SERVICES` list, so its reference page is regenerated from
`packages/extra/monitoring/README.md` automatically on the next stable
tag. A user guide for tracing belongs to the docs follow-up in #3761; no
website issue or PR exists for it yet.
- cozystack/terraform-provider-cozystack: the provider has no
`Monitoring` resource (only the tenant's `monitoring` flag), so the new
schema field does not reach it.
- The remaining repositories in the trigger map are not reached by this
diff: no platform values, variants, installer values, `hack/` tooling,
node prerequisites, ApplicationDefinition semantics or telemetry names
change.
- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
feat(monitoring): add a VictoriaTraces backend and a Grafana traces datasource to the monitoring stack (distributed tracing, phase 1). Tracing is off by default; enable it per tenant by adding a `tracingStorages` entry (`cluster` or `single` mode). Also fix the VictoriaLogs Grafana datasources: with two or more `logsStorages` entries, every entry except the last was silently dropped, so existing installs with several log stores gain the missing datasources on upgrade.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added configurable trace storage with cluster or single-node
deployment modes.
* Supports retention periods, persistent storage sizing, storage
classes, and optional disk-usage limits.
* Automatically configures Grafana tracing data sources for each enabled
trace backend.
* Monitoring readiness reflects the health of configured trace storage
services.
* Invalid trace storage settings are rejected during configuration.
* **Bug Fixes**
* Every VictoriaLogs storage now gets its own Grafana datasource; with
several log storages, all but the last were previously dropped.
* **Tests**
* Added coverage for storage rendering, retention settings, validation
guards, Grafana endpoints, health checks, and cleanup behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
cloudflareProxy passed the Cloudflare ranges to the controller under
set_real_ip_from, which is an nginx directive and not an ingress-nginx
ConfigMap key, so the controller dropped it without a warning. With
use-forwarded-headers on, the realip module then trusted every source
from the default proxy-real-ip-cidr of 0.0.0.0/0, and any client that
reached the load balancer directly could set CF-Connecting-IP to an
address of its choice. That address is what whitelist-source-range
checks, so the tenant whitelist, and any per-Ingress one, could be
walked past with one header.
proxy-real-ip-cidr is the key the controller reads for the trusted
ranges.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Rendering the inner chart alone cannot catch a wrapper dropping tenant
OIDC settings. The audience mapper and System-mode reconcile flag also
need assertions before the unused live suites are removed.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The timeout, waitStrategy and healthCheckExprs rationale in the
monitoring-system HelmRelease was written as `#` comments, which Helm
renders verbatim, so every tenant's HelmRelease carried about thirty lines
of prose. Move it into template comments like the traces paragraph beside
it: the reasoning stays in the chart source and the rendered object is
unchanged apart from losing the comments.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
A tracingStorages name is the identity of everything the entry renders:
the VTCluster/VTSingle, its Services and PVCs, and its Grafana datasource.
Renaming an entry therefore deploys an empty backend and removes the old
one, exactly like changing mode, but only mode said so. Give name the same
treat-as-immutable note, and state the limits the render guard enforces.
Reword the neighbouring mode and retentionDiskUsageBytes descriptions
without arrows and dashes, which render awkwardly in the dashboard and in
the generated README.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
Two tracingStorages values render cleanly and only fail once the operator
acts on them.
- A zero storage is silently destructive in single mode: the operator's
isStorageEmpty gate checks IsZero() and mounts an EmptyDir instead of a
PVC, the VTSingle reports operational and the release goes Ready, and every
stored span is lost on the next reschedule. Type the storage field as a
quantity so the schema refuses non-quantities at admission, and refuse
every zero spelling ("", 0, 0Gi, 0.0, .0, 0e3, 0m, +0, -0Mi) in the render
guard, since the quantity pattern accepts those.
- A name over 42 characters wedges cluster mode: the operator names the
StatefulSet vtstorage-<name>, and each pod gets the label
controller-revision-hash: vtstorage-<name>-<hash>, whose value is capped at
63 with up to 10 characters of hash. Past that no pod is created, the
VTCluster goes failed and the release stays not-Ready under retries: -1.
Cap the length in the same guard.
- Quote the rendered name, so a numeric name such as 2024 stays a string
instead of failing the apply on metadata.name.
- Rewrite the helper's rationale so it matches the trimmed CRDs and the
pinned operator, and state in the mode field description that the API
cannot enforce immutability on an array-item field, so the note is the only
guard.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
The `#` prose in the traces datasource and in the HelmRelease health-check
block is not stripped at render the way `{{/* */}}` is, so it shipped into
every tenant's rendered GrafanaDatasource and HelmRelease. Move it into
template comments: the reasoning stays in the chart source but no longer lands
in the emitted objects.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
The installed VictoriaTraces CRDs are the trimmed ones (crds.plain: false),
so their spec is x-kubernetes-preserve-unknown-fields and a bad value reaches
the operator instead of being refused by the API server. Catch the ones a
render guard can, and document the one it cannot.
- name is used verbatim as the VTCluster/VTSingle and GrafanaDatasource
metadata.name, so an invalid name is a late admission error and a duplicate
silently overwrites the first entry (last write wins in the range). Guard it
against an RFC 1123 label and a uniqueness set.
- retentionDiskUsageBytes maps to the operator's BytesString field, whose
UnmarshalJSON rejects anything outside
^[0-9]+(kb|mb|gb|tb|KB|MB|GB|TB|KiB|MiB|GiB|TiB)?$, notably not the Gi
grammar of the sibling storage field. The operator then marks the CR failed
and the release fails well after the value was written. Refuse it at render
with the upstream pattern instead.
- changing mode re-identifies the object instead of reconfiguring it, so
switching single to cluster prunes the VTSingle and the operator
garbage-collects its PVC, and cluster to single (or dropping the entry)
leaks the vtstorage PVCs
until the release is deleted. A stateless template cannot see the prior
mode, so record the loss and the one-way migration in the field description.
The name, uniqueness, mode and retention checks share one pass in a helper;
the inline mode guard in vtraces.yaml moves there. The retention description
also notes that the cap is per vtstorage replica in cluster mode.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
The healthCheckExprs comment (and the mirroring test comment) still claimed
tracingStorages is render-rejected when empty and that the 'Ready with no
trace store' hole cannot be reached, but defaulting tracing off removed that
guard and made empty the default. Rewrite both to state the actual behaviour:
empty is the default (tracing off), so the exprs are usually inert (Flux
evaluates them only against CRs present in the inventory), and a Ready release
with no trace store is intentional and harmless because nothing produces
spans without a collector, so no fail-on-empty guard is needed.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
tracingStorages had two defaults (extra + system), so 'tracingStorages: null'
silently backfilled the inner default and a VTCluster rendered anyway, while
'tracingStorages: []' aborted the whole monitoring release. There was no clean
'off', and the default shipped a 6-pod VTCluster per tenant for a pipeline
that nothing writes spans into yet.
- Default tracingStorages to [] in both packages/system/monitoring/values.yaml
and packages/extra/monitoring/values.yaml, so absence has nothing to fall
through to and the generated schema advertises the shipped (off) state.
- Drop the fail-on-empty guard in templates/vtraces/vtraces.yaml. Nothing
produces spans without a collector, so an absent backend drops no data. The
guard belongs with the collector, where it becomes correct: an enabled
producer with no backend really would drop spans, the logsStorages #3181
rationale.
- healthCheckExprs stay unconditional (Flux evaluates them only against CRs
present in the inventory, so they are inert when nothing renders).
- Tests: replace the empty-fails case with off-by-default (renders nothing);
set tracingStorages explicitly where cases relied on the old default.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM