Can I stop all processes using CUDA in Linux without rebooting?

Question

Is it possible to stop all running processing using the GPU via CUDA, without restarting the machine?

Solution 1

The lsof utility will help with this. You can get a list of processes accessing your NVIDIA cards with:

lsof /dev/nvidia*

Then use kill or pkill to terminate the processes you want. Note that you may not want to kill X if it's running. On my desktop system, both X and kwin are also accessing the GPU.

Solution 2

Long answer:

lsof /dev/nvidia*

gives you PIDs running on your GPU card which looks something like: lsof: status error on PID: No such file or directory

COMMAND  PID    USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
python  7215 *******  mem    CHR 195,255           434 /dev/nvidiactl
python  7215 *******  mem    CHR   195,0           435 /dev/nvidia0

and

awk '{print $2}'

selects the PID column (in my case it is the second column) and

xargs -I {} kill {}

kills those PID jobs.

Short answer:

You may use the following command to remove them all at once.

Watch out! This command will delete all PIDs showing up for lsof /dev/nvidia*. Do run lsof /dev/nvidia* first to confirm these jobs are the ones you want to delete.

lsof /dev/nvidia* | awk '{print $2}' | xargs -I {} kill {}

Finish the job by a single command.

Solution 3

you can check the processes with nvidia-smi and then

kill -9 <pid>

Comments

[Marm0t]

you could always change the permissions temporarily of /dev/nvidiaxx, I haven't tried it but I believe that would kill the jobs instantly. I don't know anyway of specifying jobs specifically running on the gpu unless you were using some kind of queue or load leveler.

[WY Hsu]

Is it "kill -9 pid" ? Cuz i tred kill -9 <pid>, and not working

[WY Hsu]

I get it, just pid. <> is a like a quote

[thatWiseGuy]

This does not work for me. Killing my kernel process has no effect. The kernel process is indefinitely consuming the GPU and I can't kill it.

[einpoklum]

Suggest you add a few lines about killing them dead with kill -KILL if they can't take a hint.

[Osi]

Good shit dude this just crashed my entire PC and Ubuntu went into repair mode upon startup

[user1165814]

Sorry to hear that. I have changed the answer and add a reminder