diff --git a/packages/system/talos-log-collector/README.md b/packages/system/talos-log-collector/README.md index 078ee6853..81478dde7 100644 --- a/packages/system/talos-log-collector/README.md +++ b/packages/system/talos-log-collector/README.md @@ -5,9 +5,11 @@ forwards them to the in-cluster VictoriaLogs. Talos keeps its system and kernel (kmsg) logs inside the OS and does not write them to a host file that the default Fluent Bit `tail` input could read, so -those logs are never picked up by `monitoring-agents`. This package runs a -Vector receiver on every node; Talos pushes its logs into it over the node -loopback, and Vector forwards them to the tenant VictoriaLogs +those logs are never picked up by `monitoring-agents`. Fluent Bit there is +tail-only and the vendored chart offers no way to add a loopback TCP receiver, +so extending it is not an option; this package runs a dedicated Vector receiver +on every node instead. Talos pushes its logs into it over the node loopback, +and Vector forwards them to the tenant VictoriaLogs (`vlinsert-generic..svc`) next to the container, audit and event logs already collected by `monitoring-agents`. @@ -17,14 +19,25 @@ Vector runs as a DaemonSet on the pod network, so cluster DNS resolves the vlinsert service. It listens on a TCP socket published on the node loopback via a `hostPort` bound to `hostIP: 127.0.0.1`. Because the pod is not on the host network, the outbound path to vlinsert keeps working; the inbound loopback -socket is provided by the hostPort. `machine.logging` already forwards the -runtime kernel (kmsg) stream as the `kernel` service (verified on Talos v1.12), -so no separate `KmsgLogConfig` is needed for runtime kernel logs. +socket is provided by the hostPort. + +Talos exposes its logs through two independent config paths, and both must +point at this socket to collect everything: + +- `machine.logging.destinations` forwards the JSON-lines **service** logs + (kubelet, etcd, apid and the other machine services). +- A `KmsgLogConfig` document forwards the **kernel** (kmsg) stream, including + driver messages such as DRBD. `machine.logging.destinations` does not carry + kmsg: Talos builds the kmsg destination list only from `KmsgLogConfig` + documents and the `talos.logging.kernel=` kernel argument, so without one the + kernel logs never leave the node. Both endpoints may point at the same + socket; `KmsgLogConfig` only supports the `json_lines` format. ## Talos configuration (required) This package deploys only the receiver. Each Talos node must be told to push -its logs to the loopback socket. Add this to the machine config and apply it: +both its service logs and its kernel logs to the loopback socket. Add this to +the machine config and apply it: ```yaml machine: @@ -32,8 +45,16 @@ machine: destinations: - endpoint: "tcp://127.0.0.1:5170/" format: "json_lines" +--- +apiVersion: v1alpha1 +kind: KmsgLogConfig +name: talos-log-collector +url: "tcp://127.0.0.1:5170/" ``` +> Omitting the `KmsgLogConfig` document collects service logs only; kernel +> (kmsg) logs, including the DRBD messages, will silently never arrive. + > Early-boot kernel panics, before the network is up, are not captured by any > in-cluster collector. Use the node BMC serial console for those. @@ -62,7 +83,7 @@ machine: | `image.tag` | Image tag (digest-pinned by the build). | `string` | `0.56.0-alpine@sha256:0eb66216f5f9322264e2ba83f4606428ef77a2cd4dea619a5bea610ac80ccc43` | | `image.pullPolicy` | Image pull policy. | `string` | `IfNotPresent` | | `resources` | Compute resources for the collector. | `object` | `{}` | -| `resources.cpu` | CPU request and limit. | `quantity` | `100m` | +| `resources.cpu` | CPU request. | `quantity` | `100m` | | `resources.memory` | Memory request and limit. | `quantity` | `128Mi` |