mirror of
https://github.com/cozystack/cozystack.git
synced 2026-10-27 15:17:13 +09:00
Aleksei Sviridkin
d15ec72327
Addresses follow-up review of the vGPU variant + chart bump.
* values.yaml: pin ccManager.enabled=false and vgpuDeviceManager.enabled=false.
Upstream chart v26.x flipped both to true. ccManager auto-enables CC
mode on Hopper-class hardware after upgrade — a non-trivial behavior
change that should be opt-in. vgpuDeviceManager walks
/sys/class/mdev_bus/, which does not exist on Ada+ (L4/L40/L40S/B100
with vGPU 17/20). Pinning both off keeps the default footprint as it
was on v25.x; users explicitly opt in.
* hack/dcgm-default-counters.csv: refresh from dcgm-exporter tag
4.5.1-4.8.0 (matches the image bumped earlier in this PR). The
previous snapshot was pinned to 4.1.1-4.0.4. Adds DCGM_FI_DEV_FB_RESERVED
and other fields introduced upstream. Header comment updated.
hack/check-gpu-recording-rules.bats passes against the refreshed
fixture.
* docs/gpu-vgpu.md (Licensing section):
- Drop the wrong recommendation to wire the licensing Secret through
gpu-operator's driver.licensingConfig.secretName. That mounts into
the host driver pod, where SR-IOV vGPU does not consume a license.
Document guest-side delivery (cloud-init write_files for
/etc/nvidia/ClientConfigToken/ and /etc/nvidia/gridd.conf) end to
end.
- Correct FeatureType comment. Per the NVIDIA Virtual GPU Software
Licensing User Guide, FeatureType=0 is the unlicensed state, not
auto-detect; FeatureType=1 is the value that asks the driver to
auto-select the license type from the configured vGPU profile.
Switch the example to FeatureType=1 and list all four valid values
with their meaning.
- Add nvidia-smi verification snippet inside the guest.
* docs/gpu-vgpu.md (KubeVirt section): note that L40S SR-IOV VFs share
the PF device ID (10de:26b9), so a single pciVendorSelector matches
both. Recommend lspci -nn -d 10de: as a sanity check on other
generations where PF/VF IDs may differ.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
111 lines
6.3 KiB
CSV
111 lines
6.3 KiB
CSV
# Snapshot of the upstream DCGM Exporter default-counters.csv used as a
|
|
# fixture by hack/check-gpu-recording-rules.bats. The test verifies that
|
|
# every DCGM_FI_* metric referenced by a tracked GPU dashboard is either
|
|
# declared here (upstream defaults) or in
|
|
# packages/system/gpu-operator/examples/dcgm-custom-metrics.yaml
|
|
# (the project's custom CSV).
|
|
#
|
|
# Source: https://github.com/NVIDIA/dcgm-exporter/blob/4.5.1-4.8.0/etc/default-counters.csv
|
|
# Pinned to the DCGM Exporter image tag shipped by the gpu-operator
|
|
# chart under packages/system/gpu-operator/charts/gpu-operator/values.yaml
|
|
# (`dcgmExporter.version`).
|
|
#
|
|
# Format
|
|
# If line starts with a '#' it is considered a comment
|
|
# DCGM FIELD, Prometheus metric type, help message
|
|
|
|
# Clocks
|
|
DCGM_FI_DEV_SM_CLOCK, gauge, SM clock frequency (in MHz).
|
|
DCGM_FI_DEV_MEM_CLOCK, gauge, Memory clock frequency (in MHz).
|
|
|
|
# Temperature
|
|
DCGM_FI_DEV_MEMORY_TEMP, gauge, Memory temperature (in C).
|
|
DCGM_FI_DEV_GPU_TEMP, gauge, GPU temperature (in C).
|
|
|
|
# Power
|
|
DCGM_FI_DEV_POWER_USAGE, gauge, Power draw (in W).
|
|
DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION, counter, Total energy consumption since boot (in mJ).
|
|
|
|
# PCIE
|
|
# DCGM_FI_PROF_PCIE_TX_BYTES, counter, Total number of bytes transmitted through PCIe TX via NVML.
|
|
# DCGM_FI_PROF_PCIE_RX_BYTES, counter, Total number of bytes received through PCIe RX via NVML.
|
|
DCGM_FI_DEV_PCIE_REPLAY_COUNTER, counter, Total number of PCIe retries.
|
|
|
|
# Utilization (the sample period varies depending on the product)
|
|
DCGM_FI_DEV_GPU_UTIL, gauge, GPU utilization (in %).
|
|
DCGM_FI_DEV_MEM_COPY_UTIL, gauge, Memory utilization (in %).
|
|
DCGM_FI_DEV_ENC_UTIL, gauge, Encoder utilization (in %).
|
|
DCGM_FI_DEV_DEC_UTIL , gauge, Decoder utilization (in %).
|
|
|
|
# Errors and violations
|
|
DCGM_FI_DEV_XID_ERRORS, gauge, Value of the last XID error encountered.
|
|
# DCGM_FI_DEV_POWER_VIOLATION, counter, Throttling duration due to power constraints (in ns).
|
|
# DCGM_FI_DEV_THERMAL_VIOLATION, counter, Throttling duration due to thermal constraints (in ns).
|
|
# DCGM_FI_DEV_SYNC_BOOST_VIOLATION, counter, Throttling duration due to sync-boost constraints (in ns).
|
|
# DCGM_FI_DEV_BOARD_LIMIT_VIOLATION, counter, Throttling duration due to board limit constraints (in ns).
|
|
# DCGM_FI_DEV_LOW_UTIL_VIOLATION, counter, Throttling duration due to low utilization (in ns).
|
|
# DCGM_FI_DEV_RELIABILITY_VIOLATION, counter, Throttling duration due to reliability constraints (in ns).
|
|
|
|
# DCGM Exporter fields
|
|
|
|
# DCGM_EXP_CLOCK_EVENTS_COUNT, counter, reported clock events
|
|
# DCGM_EXP_XID_ERRORS_COUNT, counter, reported XIDs during last window
|
|
# DCGM_EXP_GPU_HEALTH_STATUS, counter, DCGM reported health status
|
|
# DCGM_EXP_P2P_STATUS, counter, P2P NvLink status
|
|
|
|
# Memory usage
|
|
DCGM_FI_DEV_FB_FREE, gauge, Framebuffer memory free (in MiB).
|
|
DCGM_FI_DEV_FB_USED, gauge, Framebuffer memory used (in MiB).
|
|
DCGM_FI_DEV_FB_RESERVED, gauge, Framebuffer memory reserved (in MiB).
|
|
|
|
# ECC
|
|
# DCGM_FI_DEV_ECC_SBE_VOL_TOTAL, counter, Total number of single-bit volatile ECC errors.
|
|
# DCGM_FI_DEV_ECC_DBE_VOL_TOTAL, counter, Total number of double-bit volatile ECC errors.
|
|
# DCGM_FI_DEV_ECC_SBE_AGG_TOTAL, counter, Total number of single-bit persistent ECC errors.
|
|
# DCGM_FI_DEV_ECC_DBE_AGG_TOTAL, counter, Total number of double-bit persistent ECC errors.
|
|
|
|
# Retired pages
|
|
# DCGM_FI_DEV_RETIRED_SBE, counter, Total number of retired pages due to single-bit errors.
|
|
# DCGM_FI_DEV_RETIRED_DBE, counter, Total number of retired pages due to double-bit errors.
|
|
# DCGM_FI_DEV_RETIRED_PENDING, counter, Total number of pages pending retirement.
|
|
|
|
# NVLink
|
|
# DCGM_FI_DEV_NVLINK_CRC_FLIT_ERROR_COUNT_TOTAL, counter, Total number of NVLink flow-control CRC errors.
|
|
# DCGM_FI_DEV_NVLINK_CRC_DATA_ERROR_COUNT_TOTAL, counter, Total number of NVLink data CRC errors.
|
|
# DCGM_FI_DEV_NVLINK_REPLAY_ERROR_COUNT_TOTAL, counter, Total number of NVLink retries.
|
|
# DCGM_FI_DEV_NVLINK_RECOVERY_ERROR_COUNT_TOTAL, counter, Total number of NVLink recovery errors.
|
|
DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL, counter, Total number of NVLink bandwidth counters for all lanes.
|
|
# DCGM_FI_DEV_NVLINK_BANDWIDTH_L0, counter, The number of bytes of active NVLink rx or tx data including both header and payload.
|
|
|
|
# VGPU License status
|
|
DCGM_FI_DEV_VGPU_LICENSE_STATUS, gauge, vGPU License status
|
|
|
|
# Remapped rows
|
|
DCGM_FI_DEV_UNCORRECTABLE_REMAPPED_ROWS, counter, Number of remapped rows for uncorrectable errors
|
|
DCGM_FI_DEV_CORRECTABLE_REMAPPED_ROWS, counter, Number of remapped rows for correctable errors
|
|
DCGM_FI_DEV_ROW_REMAP_FAILURE, gauge, Whether remapping of rows has failed
|
|
|
|
# Static configuration information. These appear as labels on the other metrics
|
|
DCGM_FI_DRIVER_VERSION, label, Driver Version
|
|
# DCGM_FI_NVML_VERSION, label, NVML Version
|
|
# DCGM_FI_DEV_BRAND, label, Device Brand
|
|
# DCGM_FI_DEV_SERIAL, label, Device Serial Number
|
|
# DCGM_FI_DEV_OEM_INFOROM_VER, label, OEM inforom version
|
|
# DCGM_FI_DEV_ECC_INFOROM_VER, label, ECC inforom version
|
|
# DCGM_FI_DEV_POWER_INFOROM_VER, label, Power management object inforom version
|
|
# DCGM_FI_DEV_INFOROM_IMAGE_VER, label, Inforom image version
|
|
# DCGM_FI_DEV_VBIOS_VERSION, label, VBIOS version of the device
|
|
|
|
# Datacenter Profiling (DCP) metrics
|
|
# NOTE: supported on Nvidia datacenter Volta GPUs and newer
|
|
DCGM_FI_PROF_GR_ENGINE_ACTIVE, gauge, Ratio of time the graphics engine is active.
|
|
# DCGM_FI_PROF_SM_ACTIVE, gauge, The ratio of cycles an SM has at least 1 warp assigned.
|
|
# DCGM_FI_PROF_SM_OCCUPANCY, gauge, The ratio of number of warps resident on an SM.
|
|
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, gauge, Ratio of cycles the tensor (HMMA) pipe is active.
|
|
DCGM_FI_PROF_DRAM_ACTIVE, gauge, Ratio of cycles the device memory interface is active sending or receiving data.
|
|
# DCGM_FI_PROF_PIPE_FP64_ACTIVE, gauge, Ratio of cycles the fp64 pipes are active.
|
|
# DCGM_FI_PROF_PIPE_FP32_ACTIVE, gauge, Ratio of cycles the fp32 pipes are active.
|
|
# DCGM_FI_PROF_PIPE_FP16_ACTIVE, gauge, Ratio of cycles the fp16 pipes are active.
|
|
DCGM_FI_PROF_PCIE_TX_BYTES, gauge, The rate of data transmitted over the PCIe bus - including both protocol headers and data payloads - in bytes per second.
|
|
DCGM_FI_PROF_PCIE_RX_BYTES, gauge, The rate of data received over the PCIe bus - including both protocol headers and data payloads - in bytes per second.
|