How is GPU and memory utilization defined in nvidia-smi results?

Question

I am currently using a tool shipped with nvidia's driver 'nvidia-smi' for performance monitoring on GPU. When we use 'nvidia-smi -a', it will give the information of current GPU information, including GPU core and memory usage, temperature and so on like this:

==============NVSMI LOG==============

Timestamp : Tue

Feb 22 22:39:09 2011

Driver Version : 260.19.26

GPU 0:

    Product Name            : GeForce 8800 GTX
    PCI Device/Vendor ID    : 19110de
    PCI Location ID         : 0:4:0
    Board Serial            : 211561763875
    Display                 : Connected
    Temperature             : 55 C
    Fan Speed               : 47%
    Utilization
        GPU                 : 1%
        Memory              : 0%

I am curious about how are the GPU and memory Utilization defined? For example, GPU core's utilization is 47%. It means there are 47% of SMs active working? Or all the GPU cores are busy in 47% time while idle other 53% time? For memory, the utilization stands for the ratio between current bandwidth and max bandwidth, or the busy time ratio in last time unit?

Solution 1

A post by a moderator on the NVIDIA forums says the GPU utilization and memory utilization figures are based on activity over the last second:

GPU busy is actually the percentage of time over the last second the SMs were busy, and the memory utilization is actually the percentage of bandwidth used during the last second. Full memory consumption statistics come with the next release.

Solution 2

You can refer to this official API document: http://docs.nvidia.com/deploy/nvml-api/structnvmlUtilization__t.html#structnvmlUtilization__t

It says : "Percent of time over the past sample period during which one or more kernels was executing on the GPU."

Comments

[enixon4]

To be more specific: GPU busy is the percentage of time over the last second that any of the SMs was busy, and the memory utilization is actually the percentage of time the memory controller was busy during the last second. You can keep the utilization counts near 100% by simply running a kernel on a single SM and transferring 1 byte over PCI-E back and forth. Utilization is not a "how well you're using the resources" statistic but "if you're using the resources".

[dizcza]

@PrzemyslawZych So how can I then get the real GPU usage like the number of active SMs out of the total number of SMs?

[enixon4]

@dizcza Then we're not talking about monitoring but profiling and for that I'd suggest starting with e.g. NVIDIA Visual Profiler.