fix(talos-log-collector): document KmsgLogConfig requirement for kernel logs

machine.logging.destinations only carries the JSON-lines service logs; Talos
builds its kmsg destination list solely from KmsgLogConfig documents and the
talos.logging.kernel= kernel argument. The previous README claimed
machine.logging alone forwarded the kernel stream, so an operator following it
collected service logs only and the kernel (kmsg) logs, including the DRBD
messages this package targets, silently never arrived. Add the required
KmsgLogConfig document to the node configuration and explain both paths.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
This commit is contained in:
IvanHunters committed 2026-07-22 22:14:49 +03:00
1 parent 17ab07d31e
commit db6c26681f
1 file changed
+29 -8
+29 -8
View File
@@ -5,9 +5,11 @@ forwards them to the in-cluster VictoriaLogs.
Talos keeps its system and kernel (kmsg) logs inside the OS and does not write
them to a host file that the default Fluent Bit `tail` input could read, so
those logs are never picked up by `monitoring-agents`. This package runs a
Vector receiver on every node; Talos pushes its logs into it over the node
loopback, and Vector forwards them to the tenant VictoriaLogs
those logs are never picked up by `monitoring-agents`. Fluent Bit there is
tail-only and the vendored chart offers no way to add a loopback TCP receiver,
so extending it is not an option; this package runs a dedicated Vector receiver
on every node instead. Talos pushes its logs into it over the node loopback,
and Vector forwards them to the tenant VictoriaLogs
(`vlinsert-generic.<target>.svc`) next to the container, audit and event logs
already collected by `monitoring-agents`.
@@ -17,14 +19,25 @@ Vector runs as a DaemonSet on the pod network, so cluster DNS resolves the
vlinsert service. It listens on a TCP socket published on the node loopback via
a `hostPort` bound to `hostIP: 127.0.0.1`. Because the pod is not on the host
network, the outbound path to vlinsert keeps working; the inbound loopback
socket is provided by the hostPort. `machine.logging` already forwards the
runtime kernel (kmsg) stream as the `kernel` service (verified on Talos v1.12),
so no separate `KmsgLogConfig` is needed for runtime kernel logs.
socket is provided by the hostPort.
Talos exposes its logs through two independent config paths, and both must
point at this socket to collect everything:
- `machine.logging.destinations` forwards the JSON-lines **service** logs
(kubelet, etcd, apid and the other machine services).
- A `KmsgLogConfig` document forwards the **kernel** (kmsg) stream, including
driver messages such as DRBD. `machine.logging.destinations` does not carry
kmsg: Talos builds the kmsg destination list only from `KmsgLogConfig`
documents and the `talos.logging.kernel=` kernel argument, so without one the
kernel logs never leave the node. Both endpoints may point at the same
socket; `KmsgLogConfig` only supports the `json_lines` format.
## Talos configuration (required)
This package deploys only the receiver. Each Talos node must be told to push
its logs to the loopback socket. Add this to the machine config and apply it:
both its service logs and its kernel logs to the loopback socket. Add this to
the machine config and apply it:
```yaml
machine:
@@ -32,8 +45,16 @@ machine:
destinations:
- endpoint: "tcp://127.0.0.1:5170/"
format: "json_lines"
---
apiVersion: v1alpha1
kind: KmsgLogConfig
name: talos-log-collector
url: "tcp://127.0.0.1:5170/"
```
> Omitting the `KmsgLogConfig` document collects service logs only; kernel
> (kmsg) logs, including the DRBD messages, will silently never arrive.
> Early-boot kernel panics, before the network is up, are not captured by any
> in-cluster collector. Use the node BMC serial console for those.
@@ -62,7 +83,7 @@ machine:
| `image.tag` | Image tag (digest-pinned by the build). | `string` | `0.56.0-alpine@sha256:0eb66216f5f9322264e2ba83f4606428ef77a2cd4dea619a5bea610ac80ccc43` |
| `image.pullPolicy` | Image pull policy. | `string` | `IfNotPresent` |
| `resources` | Compute resources for the collector. | `object` | `{}` |
| `resources.cpu` | CPU request and limit. | `quantity` | `100m` |
| `resources.cpu` | CPU request. | `quantity` | `100m` |
| `resources.memory` | Memory request and limit. | `quantity` | `128Mi` |