mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 14:02:56 +09:00
feat(monitoring-agents): keep application identity in container log streams (#5217)
## What this PR does
Makes Fluent Bit keep the pod lineage labels
`apps.cozystack.io/application.{group,kind,name}` on container log
lines, as the stream fields
`apps_cozystack_io_application_{group,kind,name}`, for both the default
container-log output and the per-tenant routed output.
Why: the per-application logs scope in the design proposal [Tenant
application logs](https://github.com/cozystack/community/pull/95)
(section 6) filters by these stream fields. Today the kubernetes filter
fetches pod labels, but they are dropped together with the rest of the
`kubernetes_*` keys before output.
What changes in the pipeline:
- A second `nest` lift of the pod labels, then a rename of the three
lineage keys to their final names before the existing `Remove_wildcard
kubernetes_` runs. All other labels are still removed. Every lifted key
is prefixed, so no other label can collide into the identity names.
- The three keys are added to `_stream_fields` of both container-log
outputs. Events and audit pipelines are unchanged.
- A parsed JSON log line is kept under `log_processed` (already on
`main`), so a line's content cannot create the identity keys.
- Guest Kubernetes clusters are not affected: their monitoring-agents
release replaces the filters and outputs wholesale. A regression test in
`packages/apps/kubernetes` pins that guest streams carry no identity
fields.
Effect on existing streams: VictoriaLogs treats a missing stream field
like an empty one, and the rename writes nothing when the label is
absent. Pods without lineage labels keep exactly the same stream. Pods
with lineage labels get a new stream once after the rollout; their
earlier lines stay reachable by namespace.
#### How it was tested
- helm-unittest for `monitoring-agents` (lift order, rename before
`Remove_wildcard`, stream fields on both outputs, events and audit
untouched) and for `kubernetes` (guest streams unchanged). The
events/audit guard was mutation-checked.
- Verified on a test cluster: real pod log lines arrive, Redis pods'
lines carry the three `apps_cozystack_io_application_*` stream fields,
and Fluent Bit logs show no errors.
#### Known limitation
Guest clusters write to the parent log store without authentication, so
a guest administrator can forge these stream fields as well as the
namespace field, within the store their tenant shares. This is the
write-path trust risk already documented in the proposal.
### Screenshots
No UI changes.
### Downstream repositories
- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
Downstream: no trigger matched. Candidate: none affected, because the
diff changes `packages/system/monitoring-agents/values.yaml` (a system
package with no schema) and adds tests; the file under
`packages/apps/kubernetes` is a test only, so the app's `values.yaml`
and `values.schema.json` are unchanged.
### Release note
```release-note
feat(monitoring-agents): container log lines of pods that belong to a Cozystack application now carry `apps_cozystack_io_application_{group,kind,name}` stream fields in VictoriaLogs. Streams of pods without application labels are unchanged; streams of application pods start a new stream once after the upgrade.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **New Features**
* Container-log streams now include application group, kind, and name
metadata for root and tenant stores.
* Guest monitoring streams include pod name, container name, and
namespace fields.
* **Bug Fixes**
* Application metadata is added to container-log streams without
changing the existing stream fields for event and audit outputs.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
This commit is contained in:
3 files changed
+123
-2
No files matched your search
@@ -0,0 +1,38 @@
|
||||
suite: guest fluent-bit never ships application identity stream fields
|
||||
|
||||
templates:
|
||||
- templates/helmreleases/monitoring-agents.yaml
|
||||
|
||||
values:
|
||||
- values/common.yaml
|
||||
|
||||
release:
|
||||
name: kubernetes-test
|
||||
namespace: tenant-test
|
||||
|
||||
# Pods in a guest cluster are tenant-controlled, so their
|
||||
# apps.cozystack.io/application.* labels are not a trustworthy application
|
||||
# identity, and the platform must not promote them to the stream fields the
|
||||
# parent scopes application logs by. The guest release replaces the
|
||||
# monitoring-agents filters and outputs, so it is the guest overrides that keep
|
||||
# the parent's label lift out. This does not stop a guest admin who edits the
|
||||
# guest's own fluent-bit or writes to the parent's vlinsert directly; that is
|
||||
# the write-path control the logs proposal leaves open.
|
||||
|
||||
tests:
|
||||
- it: does not lift pod labels or declare lineage stream fields in the guest
|
||||
set:
|
||||
addons.monitoringAgents.enabled: true
|
||||
_namespace.monitoring: tenant-test
|
||||
asserts:
|
||||
- isKind:
|
||||
of: HelmRelease
|
||||
- notMatchRegex:
|
||||
path: spec.values.fluent-bit.config.filters
|
||||
pattern: 'kubernetes_labels|apps_cozystack_io_application_'
|
||||
- notMatchRegex:
|
||||
path: spec.values.fluent-bit.config.outputs
|
||||
pattern: 'apps_cozystack_io_application_'
|
||||
- matchRegex:
|
||||
path: spec.values.fluent-bit.config.outputs
|
||||
pattern: '_stream_fields=stream,kubernetes_pod_name,kubernetes_container_name,kubernetes_namespace_name&'
|
||||
@@ -0,0 +1,67 @@
|
||||
suite: fluent-bit keeps the application identity of a container log line
|
||||
|
||||
# The lineage webhook labels a pod with apps.cozystack.io/application.{group,
|
||||
# kind,name}. A container log line keeps them as top-level
|
||||
# apps_cozystack_io_application_{group,kind,name}, the names the metrics side
|
||||
# uses, and they are stream fields so a reader can scope to one application
|
||||
# with extra_stream_filters. The kubernetes map, labels included, is otherwise
|
||||
# dropped by Remove_wildcard kubernetes_, so the three keys have to be lifted
|
||||
# and renamed out of the kubernetes_ prefix before that filter runs.
|
||||
templates:
|
||||
- charts/fluent-bit/templates/configmap.yaml
|
||||
release:
|
||||
name: monitoring-agents
|
||||
namespace: cozy-monitoring
|
||||
set:
|
||||
global:
|
||||
target: tenant-root
|
||||
kubernetesProvider:
|
||||
scheme:
|
||||
"v1/Namespace":
|
||||
gvr:
|
||||
version: v1
|
||||
resource: namespaces
|
||||
namespaced: false
|
||||
objects:
|
||||
- kind: Namespace
|
||||
apiVersion: v1
|
||||
metadata:
|
||||
name: tenant-a
|
||||
labels:
|
||||
namespace.cozystack.io/monitoring: tenant-a
|
||||
tests:
|
||||
- it: lifts the pod labels after the kubernetes map is lifted
|
||||
asserts:
|
||||
- matchRegex:
|
||||
path: data["fluent-bit.conf"]
|
||||
pattern: '(?s)Operation lift\n\s+Nested_under kubernetes\n.*Name nest\n\s+Match kube\.\*\n\s+Operation lift\n\s+Nested_under kubernetes_labels\n\s+Add_prefix kubernetes_labels_\n'
|
||||
|
||||
- it: renames the three lineage labels out of the kubernetes_ prefix before Remove_wildcard
|
||||
asserts:
|
||||
- matchRegex:
|
||||
path: data["fluent-bit.conf"]
|
||||
pattern: '(?s)Nested_under kubernetes_labels\n.*Rename kubernetes_labels_apps\.cozystack\.io/application\.group apps_cozystack_io_application_group\n\s+Rename kubernetes_labels_apps\.cozystack\.io/application\.kind apps_cozystack_io_application_kind\n\s+Rename kubernetes_labels_apps\.cozystack\.io/application\.name apps_cozystack_io_application_name\n.*Remove_wildcard kubernetes_\n'
|
||||
|
||||
- it: makes the lineage keys stream fields on the default container-log output
|
||||
asserts:
|
||||
- matchRegex:
|
||||
path: data["fluent-bit.conf"]
|
||||
pattern: 'Match kube\.\*\n\s+Host vlinsert-generic\.tenant-root\.svc\n\s+port 9481\n\s+compress gzip\n\s+uri /insert/jsonline\?_stream_fields=log_source,stream,kubernetes_pod_name,kubernetes_container_name,kubernetes_namespace_name,apps_cozystack_io_application_group,apps_cozystack_io_application_kind,apps_cozystack_io_application_name&'
|
||||
|
||||
- it: makes the lineage keys stream fields on a routed tenant's container-log output
|
||||
asserts:
|
||||
- matchRegex:
|
||||
path: data["fluent-bit.conf"]
|
||||
pattern: 'Match tenant\.tenant-a\.\*\n\s+Host vlinsert-generic\.tenant-a\.svc\n\s+port 9481\n\s+compress gzip\n\s+uri /insert/jsonline\?_stream_fields=log_source,stream,kubernetes_pod_name,kubernetes_container_name,kubernetes_namespace_name,apps_cozystack_io_application_group,apps_cozystack_io_application_kind,apps_cozystack_io_application_name&'
|
||||
|
||||
- it: leaves the events and audit outputs without lineage stream fields
|
||||
asserts:
|
||||
- matchRegex:
|
||||
path: data["fluent-bit.conf"]
|
||||
pattern: 'uri /insert/jsonline\?_stream_fields=log_source,reason,metadata_namespace,metadata_name&'
|
||||
- matchRegex:
|
||||
path: data["fluent-bit.conf"]
|
||||
pattern: 'uri /insert/jsonline\?_stream_fields=log_source,stage,user_username,verb,requestUri&'
|
||||
- notMatchRegex:
|
||||
path: data["fluent-bit.conf"]
|
||||
pattern: 'Match (events\.\*|audit\.\*|\*)\n[^\[]*Nested_under kubernetes_labels'
|
||||
@@ -411,7 +411,7 @@ fluent-bit:
|
||||
Host vlinsert-generic.{{ .Values.global.target }}.svc
|
||||
port 9481
|
||||
compress gzip
|
||||
uri /insert/jsonline?_stream_fields=log_source,stream,kubernetes_pod_name,kubernetes_container_name,kubernetes_namespace_name&_msg_field=log&_time_field=date
|
||||
uri /insert/jsonline?_stream_fields=log_source,stream,kubernetes_pod_name,kubernetes_container_name,kubernetes_namespace_name,apps_cozystack_io_application_group,apps_cozystack_io_application_kind,apps_cozystack_io_application_name&_msg_field=log&_time_field=date
|
||||
format json_lines
|
||||
json_date_format iso8601
|
||||
header AccountID 0
|
||||
@@ -445,7 +445,7 @@ fluent-bit:
|
||||
Host vlinsert-generic.{{ $target }}.svc
|
||||
port 9481
|
||||
compress gzip
|
||||
uri /insert/jsonline?_stream_fields=log_source,stream,kubernetes_pod_name,kubernetes_container_name,kubernetes_namespace_name&_msg_field=log&_time_field=date
|
||||
uri /insert/jsonline?_stream_fields=log_source,stream,kubernetes_pod_name,kubernetes_container_name,kubernetes_namespace_name,apps_cozystack_io_application_group,apps_cozystack_io_application_kind,apps_cozystack_io_application_name&_msg_field=log&_time_field=date
|
||||
format json_lines
|
||||
json_date_format iso8601
|
||||
header AccountID 0
|
||||
@@ -467,6 +467,19 @@ fluent-bit:
|
||||
Operation lift
|
||||
Nested_under kubernetes
|
||||
Add_prefix kubernetes_
|
||||
{{- /*
|
||||
The pod's lineage labels name the application it belongs to. lift
|
||||
cannot pick keys, so every label comes up; all but these three keep
|
||||
the kubernetes_ prefix and fall to Remove_wildcard below. An absent
|
||||
label stays absent, and VictoriaLogs builds _stream from the stream
|
||||
fields that are present, so an unlabelled pod keeps its stream.
|
||||
*/}}
|
||||
[FILTER]
|
||||
Name nest
|
||||
Match kube.*
|
||||
Operation lift
|
||||
Nested_under kubernetes_labels
|
||||
Add_prefix kubernetes_labels_
|
||||
[FILTER]
|
||||
Name modify
|
||||
Match kube.*
|
||||
@@ -507,6 +520,9 @@ fluent-bit:
|
||||
Rename kubernetes_pod_name keep_kubernetes_pod_name
|
||||
Rename kubernetes_namespace_name keep_kubernetes_namespace_name
|
||||
Rename kubernetes_container_name keep_kubernetes_container_name
|
||||
Rename kubernetes_labels_apps.cozystack.io/application.group apps_cozystack_io_application_group
|
||||
Rename kubernetes_labels_apps.cozystack.io/application.kind apps_cozystack_io_application_kind
|
||||
Rename kubernetes_labels_apps.cozystack.io/application.name apps_cozystack_io_application_name
|
||||
[FILTER]
|
||||
Name modify
|
||||
Match kube.*
|
||||
|
||||
Reference in new issue
Block a user