mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-22 12:43:32 +09:00
fix(talos-log-collector): document KmsgLogConfig requirement for kernel logs
machine.logging.destinations only carries the JSON-lines service logs; Talos builds its kmsg destination list solely from KmsgLogConfig documents and the talos.logging.kernel= kernel argument. The previous README claimed machine.logging alone forwarded the kernel stream, so an operator following it collected service logs only and the kernel (kmsg) logs, including the DRBD messages this package targets, silently never arrived. Add the required KmsgLogConfig document to the node configuration and explain both paths. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
This commit is contained in:
1 file changed
+29
-8
@@ -5,9 +5,11 @@ forwards them to the in-cluster VictoriaLogs.
|
||||
|
||||
Talos keeps its system and kernel (kmsg) logs inside the OS and does not write
|
||||
them to a host file that the default Fluent Bit `tail` input could read, so
|
||||
those logs are never picked up by `monitoring-agents`. This package runs a
|
||||
Vector receiver on every node; Talos pushes its logs into it over the node
|
||||
loopback, and Vector forwards them to the tenant VictoriaLogs
|
||||
those logs are never picked up by `monitoring-agents`. Fluent Bit there is
|
||||
tail-only and the vendored chart offers no way to add a loopback TCP receiver,
|
||||
so extending it is not an option; this package runs a dedicated Vector receiver
|
||||
on every node instead. Talos pushes its logs into it over the node loopback,
|
||||
and Vector forwards them to the tenant VictoriaLogs
|
||||
(`vlinsert-generic.<target>.svc`) next to the container, audit and event logs
|
||||
already collected by `monitoring-agents`.
|
||||
|
||||
@@ -17,14 +19,25 @@ Vector runs as a DaemonSet on the pod network, so cluster DNS resolves the
|
||||
vlinsert service. It listens on a TCP socket published on the node loopback via
|
||||
a `hostPort` bound to `hostIP: 127.0.0.1`. Because the pod is not on the host
|
||||
network, the outbound path to vlinsert keeps working; the inbound loopback
|
||||
socket is provided by the hostPort. `machine.logging` already forwards the
|
||||
runtime kernel (kmsg) stream as the `kernel` service (verified on Talos v1.12),
|
||||
so no separate `KmsgLogConfig` is needed for runtime kernel logs.
|
||||
socket is provided by the hostPort.
|
||||
|
||||
Talos exposes its logs through two independent config paths, and both must
|
||||
point at this socket to collect everything:
|
||||
|
||||
- `machine.logging.destinations` forwards the JSON-lines **service** logs
|
||||
(kubelet, etcd, apid and the other machine services).
|
||||
- A `KmsgLogConfig` document forwards the **kernel** (kmsg) stream, including
|
||||
driver messages such as DRBD. `machine.logging.destinations` does not carry
|
||||
kmsg: Talos builds the kmsg destination list only from `KmsgLogConfig`
|
||||
documents and the `talos.logging.kernel=` kernel argument, so without one the
|
||||
kernel logs never leave the node. Both endpoints may point at the same
|
||||
socket; `KmsgLogConfig` only supports the `json_lines` format.
|
||||
|
||||
## Talos configuration (required)
|
||||
|
||||
This package deploys only the receiver. Each Talos node must be told to push
|
||||
its logs to the loopback socket. Add this to the machine config and apply it:
|
||||
both its service logs and its kernel logs to the loopback socket. Add this to
|
||||
the machine config and apply it:
|
||||
|
||||
```yaml
|
||||
machine:
|
||||
@@ -32,8 +45,16 @@ machine:
|
||||
destinations:
|
||||
- endpoint: "tcp://127.0.0.1:5170/"
|
||||
format: "json_lines"
|
||||
---
|
||||
apiVersion: v1alpha1
|
||||
kind: KmsgLogConfig
|
||||
name: talos-log-collector
|
||||
url: "tcp://127.0.0.1:5170/"
|
||||
```
|
||||
|
||||
> Omitting the `KmsgLogConfig` document collects service logs only; kernel
|
||||
> (kmsg) logs, including the DRBD messages, will silently never arrive.
|
||||
|
||||
> Early-boot kernel panics, before the network is up, are not captured by any
|
||||
> in-cluster collector. Use the node BMC serial console for those.
|
||||
|
||||
@@ -62,7 +83,7 @@ machine:
|
||||
| `image.tag` | Image tag (digest-pinned by the build). | `string` | `0.56.0-alpine@sha256:0eb66216f5f9322264e2ba83f4606428ef77a2cd4dea619a5bea610ac80ccc43` |
|
||||
| `image.pullPolicy` | Image pull policy. | `string` | `IfNotPresent` |
|
||||
| `resources` | Compute resources for the collector. | `object` | `{}` |
|
||||
| `resources.cpu` | CPU request and limit. | `quantity` | `100m` |
|
||||
| `resources.cpu` | CPU request. | `quantity` | `100m` |
|
||||
| `resources.memory` | Memory request and limit. | `quantity` | `128Mi` |
|
||||
|
||||
|
||||
|
||||
Reference in new issue
Block a user