cozystack/hack/dcgm-default-counters.csv at main

0 Fork 0

mirror of https://github.com/cozystack/cozystack.git synced 2026-10-23 03:27:50 +09:00

Files

T

Aleksei Sviridkin d15ec72327 fix(gpu-operator): pin upstream behavior changes; correct DLS licensing example

Addresses follow-up review of the vGPU variant + chart bump.

* values.yaml: pin ccManager.enabled=false and vgpuDeviceManager.enabled=false.
  Upstream chart v26.x flipped both to true. ccManager auto-enables CC
  mode on Hopper-class hardware after upgrade — a non-trivial behavior
  change that should be opt-in. vgpuDeviceManager walks
  /sys/class/mdev_bus/, which does not exist on Ada+ (L4/L40/L40S/B100
  with vGPU 17/20). Pinning both off keeps the default footprint as it
  was on v25.x; users explicitly opt in.

* hack/dcgm-default-counters.csv: refresh from dcgm-exporter tag
  4.5.1-4.8.0 (matches the image bumped earlier in this PR). The
  previous snapshot was pinned to 4.1.1-4.0.4. Adds DCGM_FI_DEV_FB_RESERVED
  and other fields introduced upstream. Header comment updated.
  hack/check-gpu-recording-rules.bats passes against the refreshed
  fixture.

* docs/gpu-vgpu.md (Licensing section):
  - Drop the wrong recommendation to wire the licensing Secret through
    gpu-operator's driver.licensingConfig.secretName. That mounts into
    the host driver pod, where SR-IOV vGPU does not consume a license.
    Document guest-side delivery (cloud-init write_files for
    /etc/nvidia/ClientConfigToken/ and /etc/nvidia/gridd.conf) end to
    end.
  - Correct FeatureType comment. Per the NVIDIA Virtual GPU Software
    Licensing User Guide, FeatureType=0 is the unlicensed state, not
    auto-detect; FeatureType=1 is the value that asks the driver to
    auto-select the license type from the configured vGPU profile.
    Switch the example to FeatureType=1 and list all four valid values
    with their meaning.
  - Add nvidia-smi verification snippet inside the guest.

* docs/gpu-vgpu.md (KubeVirt section): note that L40S SR-IOV VFs share
  the PF device ID (10de:26b9), so a single pciVendorSelector matches
  both. Recommend lspci -nn -d 10de: as a sanity check on other
  generations where PF/VF IDs may differ.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>

2026-04-30 14:52:56 +03:00

6.3 KiB

Raw Permalink Blame History

1	# Snapshot of the upstream DCGM Exporter default-counters.csv used as a
2	# fixture by hack/check-gpu-recording-rules.bats. The test verifies that
3	# every DCGM_FI_* metric referenced by a tracked GPU dashboard is either
4	# declared here (upstream defaults) or in
5	# packages/system/gpu-operator/examples/dcgm-custom-metrics.yaml
6	# (the project's custom CSV).
7	#
8	# Source: https://github.com/NVIDIA/dcgm-exporter/blob/4.5.1-4.8.0/etc/default-counters.csv
9	# Pinned to the DCGM Exporter image tag shipped by the gpu-operator
10	# chart under packages/system/gpu-operator/charts/gpu-operator/values.yaml
11	# (`dcgmExporter.version`).
12	#
13	# Format
14	# If line starts with a '#' it is considered a comment
15	# DCGM FIELD	Prometheus metric type	help message
16	# Clocks
17	DCGM_FI_DEV_SM_CLOCK	gauge	SM clock frequency (in MHz).
18	DCGM_FI_DEV_MEM_CLOCK	gauge	Memory clock frequency (in MHz).
19	# Temperature
20	DCGM_FI_DEV_MEMORY_TEMP	gauge	Memory temperature (in C).
21	DCGM_FI_DEV_GPU_TEMP	gauge	GPU temperature (in C).
22	# Power
23	DCGM_FI_DEV_POWER_USAGE	gauge	Power draw (in W).
24	DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION	counter	Total energy consumption since boot (in mJ).
25	# PCIE
26	# DCGM_FI_PROF_PCIE_TX_BYTES	counter	Total number of bytes transmitted through PCIe TX via NVML.
27	# DCGM_FI_PROF_PCIE_RX_BYTES	counter	Total number of bytes received through PCIe RX via NVML.
28	DCGM_FI_DEV_PCIE_REPLAY_COUNTER	counter	Total number of PCIe retries.
29	# Utilization (the sample period varies depending on the product)
30	DCGM_FI_DEV_GPU_UTIL	gauge	GPU utilization (in %).
31	DCGM_FI_DEV_MEM_COPY_UTIL	gauge	Memory utilization (in %).
32	DCGM_FI_DEV_ENC_UTIL	gauge	Encoder utilization (in %).
33	DCGM_FI_DEV_DEC_UTIL	gauge	Decoder utilization (in %).
34	# Errors and violations
35	DCGM_FI_DEV_XID_ERRORS	gauge	Value of the last XID error encountered.
36	# DCGM_FI_DEV_POWER_VIOLATION	counter	Throttling duration due to power constraints (in ns).
37	# DCGM_FI_DEV_THERMAL_VIOLATION	counter	Throttling duration due to thermal constraints (in ns).
38	# DCGM_FI_DEV_SYNC_BOOST_VIOLATION	counter	Throttling duration due to sync-boost constraints (in ns).
39	# DCGM_FI_DEV_BOARD_LIMIT_VIOLATION	counter	Throttling duration due to board limit constraints (in ns).
40	# DCGM_FI_DEV_LOW_UTIL_VIOLATION	counter	Throttling duration due to low utilization (in ns).
41	# DCGM_FI_DEV_RELIABILITY_VIOLATION	counter	Throttling duration due to reliability constraints (in ns).
42	# DCGM Exporter fields
43	# DCGM_EXP_CLOCK_EVENTS_COUNT	counter	reported clock events
44	# DCGM_EXP_XID_ERRORS_COUNT	counter	reported XIDs during last window
45	# DCGM_EXP_GPU_HEALTH_STATUS	counter	DCGM reported health status
46	# DCGM_EXP_P2P_STATUS	counter	P2P NvLink status
47	# Memory usage
48	DCGM_FI_DEV_FB_FREE	gauge	Framebuffer memory free (in MiB).
49	DCGM_FI_DEV_FB_USED	gauge	Framebuffer memory used (in MiB).
50	DCGM_FI_DEV_FB_RESERVED	gauge	Framebuffer memory reserved (in MiB).
51	# ECC
52	# DCGM_FI_DEV_ECC_SBE_VOL_TOTAL	counter	Total number of single-bit volatile ECC errors.
53	# DCGM_FI_DEV_ECC_DBE_VOL_TOTAL	counter	Total number of double-bit volatile ECC errors.
54	# DCGM_FI_DEV_ECC_SBE_AGG_TOTAL	counter	Total number of single-bit persistent ECC errors.
55	# DCGM_FI_DEV_ECC_DBE_AGG_TOTAL	counter	Total number of double-bit persistent ECC errors.
56	# Retired pages
57	# DCGM_FI_DEV_RETIRED_SBE	counter	Total number of retired pages due to single-bit errors.
58	# DCGM_FI_DEV_RETIRED_DBE	counter	Total number of retired pages due to double-bit errors.
59	# DCGM_FI_DEV_RETIRED_PENDING	counter	Total number of pages pending retirement.
60	# NVLink
61	# DCGM_FI_DEV_NVLINK_CRC_FLIT_ERROR_COUNT_TOTAL	counter	Total number of NVLink flow-control CRC errors.
62	# DCGM_FI_DEV_NVLINK_CRC_DATA_ERROR_COUNT_TOTAL	counter	Total number of NVLink data CRC errors.
63	# DCGM_FI_DEV_NVLINK_REPLAY_ERROR_COUNT_TOTAL	counter	Total number of NVLink retries.
64	# DCGM_FI_DEV_NVLINK_RECOVERY_ERROR_COUNT_TOTAL	counter	Total number of NVLink recovery errors.
65	DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL	counter	Total number of NVLink bandwidth counters for all lanes.
66	# DCGM_FI_DEV_NVLINK_BANDWIDTH_L0	counter	The number of bytes of active NVLink rx or tx data including both header and payload.
67	# VGPU License status
68	DCGM_FI_DEV_VGPU_LICENSE_STATUS	gauge	vGPU License status
69	# Remapped rows
70	DCGM_FI_DEV_UNCORRECTABLE_REMAPPED_ROWS	counter	Number of remapped rows for uncorrectable errors
71	DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS	counter	Number of remapped rows for correctable errors
72	DCGM_FI_DEV_ROW_REMAP_FAILURE	gauge	Whether remapping of rows has failed
73	# Static configuration information. These appear as labels on the other metrics
74	DCGM_FI_DRIVER_VERSION	label	Driver Version
75	# DCGM_FI_NVML_VERSION	label	NVML Version
76	# DCGM_FI_DEV_BRAND	label	Device Brand
77	# DCGM_FI_DEV_SERIAL	label	Device Serial Number
78	# DCGM_FI_DEV_OEM_INFOROM_VER	label	OEM inforom version
79	# DCGM_FI_DEV_ECC_INFOROM_VER	label	ECC inforom version
80	# DCGM_FI_DEV_POWER_INFOROM_VER	label	Power management object inforom version
81	# DCGM_FI_DEV_INFOROM_IMAGE_VER	label	Inforom image version
82	# DCGM_FI_DEV_VBIOS_VERSION	label	VBIOS version of the device
83	# Datacenter Profiling (DCP) metrics
84	# NOTE: supported on Nvidia datacenter Volta GPUs and newer
85	DCGM_FI_PROF_GR_ENGINE_ACTIVE	gauge	Ratio of time the graphics engine is active.
86	# DCGM_FI_PROF_SM_ACTIVE	gauge	The ratio of cycles an SM has at least 1 warp assigned.
87	# DCGM_FI_PROF_SM_OCCUPANCY	gauge	The ratio of number of warps resident on an SM.
88	DCGM_FI_PROF_PIPE_TENSOR_ACTIVE	gauge	Ratio of cycles the tensor (HMMA) pipe is active.
89	DCGM_FI_PROF_DRAM_ACTIVE	gauge	Ratio of cycles the device memory interface is active sending or receiving data.
90	# DCGM_FI_PROF_PIPE_FP64_ACTIVE	gauge	Ratio of cycles the fp64 pipes are active.
91	# DCGM_FI_PROF_PIPE_FP32_ACTIVE	gauge	Ratio of cycles the fp32 pipes are active.
92	# DCGM_FI_PROF_PIPE_FP16_ACTIVE	gauge	Ratio of cycles the fp16 pipes are active.
93	DCGM_FI_PROF_PCIE_TX_BYTES	gauge	The rate of data transmitted over the PCIe bus - including both protocol headers and data payloads - in bytes per second.
94	DCGM_FI_PROF_PCIE_RX_BYTES	gauge	The rate of data received over the PCIe bus - including both protocol headers and data payloads - in bytes per second.

6.3 KiB Raw Permalink Blame History

6.3 KiB

Raw Permalink Blame History