Google-colaboratory: No backend with GPU available

Question

Here it is described how to use gpu with google-colaboratory:

Simply select "GPU" in the Accelerator drop-down in Notebook Settings (either through the Edit menu or the command palette at cmd/ctrl-shift-P).

However, when I select gpu in Notebook Settings I get a popup saying:

Failed to assign a backend No backend with GPU available. Would you like to use a runtime with no accelerator?

When I run:

import tensorflow as tf
device_name = tf.test.gpu_device_name()
if device_name != '/device:GPU:0':
    raise SystemError('GPU device not found')
print('Found GPU at: {}'.format(device_name))

Of course, I get GPU device not found. It seems the description is incomplete. Any ideas what needs to be done?

Solution 1

You'll need to try again later when a GPU is available. The message indicates that all available GPUs are in use.

The FAQ provides additional info:

How may I use GPUs and why are they sometimes unavailable?

Colaboratory is intended for interactive use. Long-running background computations, particularly on GPUs, may be stopped. Please do not use Colaboratory for cryptocurrency mining. Doing so is unsupported and may result in service unavailability. We encourage users who wish to run continuous or long-running computations through Colaboratory’s UI to use a local runtime.

There seems to be a cooldown on continuous training with GPUs. So, if you encounter the error dialog, try again later, and perhaps try to limit long-term training in subsequent sessions.

Solution 2

You need to configure the Notebook with GPU device

Click Edit->notebook settings->hardware accelerator->GPU

Solution 3

Add some pictures to make it clearer

Solution 4

My reputation is just slightly too low to comment, but here's a bit of additional info for @Bob Smith's answer re cooldown period.

There seems to be a cooldown on continuous training with GPUs. So, if you encounter the error dialog, try again later, and perhaps try to limit long-term training in subsequent sessions.

Based on my own recent experience, I believe Colab will allocate you at most 12 hours of GPU usage, after which there is roughly an 8 hour cool-down period before you can use compute resources again. In my case, I could not connect to an instance even without a GPU. I'm not entirely sure about this next bit but I think if you run say 3 instances at once, your 12 hours are depleted 3 times as fast. I don't know after what period of time the 12 hour limit resets, but I'd guess maybe a day.

Anyway, still missing a few details but the main takeaway is that if you exceed you'll limit, you'll be locked out from connecting to an instance for 8 hours (which is a great pain if you're actively working on something).

Solution 5

After Reset runtime didn't work, I did:

Runtime -> Reset all runtimes -> Yes

I then got a happy:

Found GPU at: /device:GPU:0

View more solutions

Comments

[Infinite Loops]

I got Found GPU at: /device:GPU:0. Is that means GPU0, that is my Intel HD (R) Graphics Family ? How can I make it detect my 2nd GPU, that is NVIDIA?

[Pallavi]

How to check the usage limit? I mean this is just a morning time here, and I am getting usage limit exceed error for using TPU/GPU. I have only used it for an hour or 2.. and last used till previous night 12:00 AM so have comepleted more than 8 hours.

[DerekG]

I'm not 100% sure on this, but I believe you have at most 12 hours within a 24 hour period.