Google Colaboratory: misleading information about its GPU (only 5% RAM available to some users)

Question

update: this question is related to Google Colab's "Notebook settings: Hardware accelerator: GPU". This question was written before the "TPU" option was added.

Reading multiple excited announcements about Google Colaboratory providing free Tesla K80 GPU, I tried to run fast.ai lesson on it for it to never complete - quickly running out of memory. I started investigating of why.

The bottom line is that “free Tesla K80” is not "free" for all - for some only a small slice of it is "free".

I connect to Google Colab from West Coast Canada and I get only 0.5GB of what supposed to be a 24GB GPU RAM. Other users get access to 11GB of GPU RAM.

Clearly 0.5GB GPU RAM is insufficient for most ML/DL work.

If you're not sure what you get, here is little debug function I scraped together (only works with the GPU setting of the notebook):

# memory footprint support libraries/code
!ln -sf /opt/bin/nvidia-smi /usr/bin/nvidia-smi
!pip install gputil
!pip install psutil
!pip install humanize
import psutil
import humanize
import os
import GPUtil as GPU
GPUs = GPU.getGPUs()
# XXX: only one GPU on Colab and isn’t guaranteed
gpu = GPUs[0]
def printm():
 process = psutil.Process(os.getpid())
 print("Gen RAM Free: " + humanize.naturalsize( psutil.virtual_memory().available ), " | Proc size: " + humanize.naturalsize( process.memory_info().rss))
 print("GPU RAM Free: {0:.0f}MB | Used: {1:.0f}MB | Util {2:3.0f}% | Total {3:.0f}MB".format(gpu.memoryFree, gpu.memoryUsed, gpu.memoryUtil*100, gpu.memoryTotal))
printm()

Executing it in a jupyter notebook before running any other code gives me:

Gen RAM Free: 11.6 GB  | Proc size: 666.0 MB
GPU RAM Free: 566MB | Used: 10873MB | Util  95% | Total 11439MB

The lucky users who get access to the full card will see:

Gen RAM Free: 11.6 GB  | Proc size: 666.0 MB
GPU RAM Free: 11439MB | Used: 0MB | Util  0% | Total 11439MB

Do you see any flaw in my calculation of the GPU RAM availability, borrowed from GPUtil?

Can you confirm that you get similar results if you run this code on Google Colab notebook?

If my calculations are correct, is there any way to get more of that GPU RAM on the free box?

update: I'm not sure why some of us get 1/20th of what other users get. e.g. the person who helped me to debug this is from India and he gets the whole thing!

note: please don't send any more suggestions on how to kill the potentially stuck/runaway/parallel notebooks that might be consuming parts of the GPU. No matter how you slice it, if you are in the same boat as I and were to run the debug code you'd see that you still get a total of 5% of GPU RAM (as of this update still).

Solution 1

So to prevent another dozen of answers suggesting invalid in the context of this thread suggestion to !kill -9 -1, let's close this thread:

The answer is simple:

As of this writing Google simply gives only 5% of GPU to some of us, whereas 100% to the others. Period.

dec-2019 update: The problem still exists - this question's upvotes continue still.

mar-2019 update: A year later a Google employee @AmiF commented on the state of things, stating that the problem doesn't exist, and anybody who seems to have this problem needs to simply reset their runtime to recover memory. Yet, the upvotes continue, which to me this tells that the problem still exists, despite @AmiF's suggestion to the contrary.

dec-2018 update: I have a theory that Google may have a blacklist of certain accounts, or perhaps browser fingerprints, when its robots detect a non-standard behavior. It could be a total coincidence, but for quite some time I had an issue with Google Re-captcha on any website that happened to require it, where I'd have to go through dozens of puzzles before I'd be allowed through, often taking me 10+ min to accomplish. This lasted for many months. All of a sudden as of this month I get no puzzles at all and any google re-captcha gets resolved with just a single mouse click, as it used to be almost a year ago.

And why I'm telling this story? Well, because at the same time I was given 100% of the GPU RAM on Colab. That's why my suspicion is that if you are on a theoretical Google black list then you aren't being trusted to be given a lot of resources for free. I wonder if any of you find the same correlation between the limited GPU access and the Re-captcha nightmare. As I said, it could be totally a coincidence as well.

Solution 2

Last night I ran your snippet and got exactly what you got:

Gen RAM Free: 11.6 GB  | Proc size: 666.0 MB
GPU RAM Free: 566MB | Used: 10873MB | Util  95% | Total 11439MB

but today:

Gen RAM Free: 12.2 GB  I Proc size: 131.5 MB
GPU RAM Free: 11439MB | Used: 0MB | Util   0% | Total 11439MB

I think the most probable reason is the GPUs are shared among VMs, so each time you restart the runtime you have chance to switch the GPU, and there is also probability you switch to one that is being used by other users.

UPDATED: It turns out that I can use GPU normally even when the GPU RAM Free is 504 MB, which I thought as the cause of ResourceExhaustedError I got last night.

Solution 3

If you execute a cell that just has
!kill -9 -1
in it, that'll cause all of your runtime's state (including memory, filesystem, and GPU) to be wiped clean and restarted. Wait 30-60s and press the CONNECT button at the top-right to reconnect.

Solution 4

Restart Jupyter IPython Kernel:

!pkill -9 -f ipykernel_launcher

Solution 5

Im not sure if this blacklisting is true! Its rather possible, that the cores are shared among users. I ran also the test, and my results are the following:

Gen RAM Free: 12.9 GB  | Proc size: 142.8 MB
GPU RAM Free: 11441MB | Used: 0MB | Util   0% | Total 11441MB

It seems im getting also full core. However i ran it a few times, and i got the same result. Maybe i will repeat this check a few times during the day to see if there is any change.

View more solutions

Comments

[stason]

I think I re-connected probably 50 times over the period of a few days and I was always getting the same 95% usage to start with. Only once I saw 0%. In all those attempts I was getting cuda out of memory error once it was coming close to 100%.

[stason]

thank you, but your suggestion doesn't change anything. I'm still getting 5% of GPU RAM.

[Ivan Bilan]

What do you mean with your update? Can you still run stuff with 500Mb? I have the same problem, I am getting RuntimeError: cuda runtime error (2) : out of memory at /pytorch/torch/lib/THC/generated/../THCTensorMathCompare.cuh‌​:84

[Ivan Bilan]

This doesn't help. After killing and reconnecting, the GPU memory is still at 500Mb out of ~12GB.

[Ivan Bilan]

will this help with the memory issue? aren't you killing all other people's runs then?

[Ivan Bilan]

close, but no cigar: GPU RAM Free: 564MB

[Ivan Bilan]

this doesn't help, got same problem: GPU RAM Free: 564MB

[Agile Bean]

as simpler method for restarting the kernel, you can just click Runtime | Restart runtime... or the shortcut CMD/CTRL+M

[Ami F]

Your statement of "As of this writing Google simply gives only 5% of GPU to some of us, whereas 100% to the others. Period." is incorrect - Colab has never worked this way. All diagnosed cases of users seeing less than the full complement of GPU RAM available to them have boiled down to another process (started by the same user, possibly in another notebook) using the rest of the GPU's RAM.

[Ami F]

Future readers: if you think you're seeing this or similar symptoms of GPU RAM unavailability, "Reset all runtimes" in the Runtime menu will get you a fresh VM guaranteeing no stale processes are still holding on to GPU RAM. If you still see this symptom immediately after using that menu option please file a bug at github.com/googlecolab/colabtools/issues

[stason]

Your reality is clearly different from the reality of many others who continue to vote up this post a year later after it was created. It's very likely that some users indeed encounter what you described, but this is not the case for all. So I'm not sure how your statement helps here. Besides when someone asked this exact question in the repo your recommended, he got a BS answer and his ticket was closed: github.com/googlecolab/colabtools/issues/52

[Ami F]

In case it was unclear: I'm not describing what I believe the implementation is based on observation of the system's behavior as a user. I'm describing what I directly know the implementation to be. I posted hoping that users who see less than full availability report it as an issue (either user error or system bug) instead of reading the incorrect statements above and assuming things are working as intended.

[stason]

In other words you're saying you're a Google employee and you're implying that colab stopped discriminating users and from now on if a user doesn't get 100% GPU RAM on their first connect and not due to some previous usage by the same user, there must be a bug in your system, which you request to report. And you will actually look at the problem and not deal with it like I have shown in this example where one of you lied to the user giving him an incorrect reason for the problem. @AmiF.

[Ami F]

No, GPUs have never been shared, and there are no lies in the example you linked (simply a guess at and explanation of the far-and-away most-common reason for the symptom reported).

[Anshuman Kumar]

I can confirm this. I had a 15 gig dataset of mostly HD pictures (my drive has 30 gigs instead of 15gigs) and I ran my code to resize the image dataset to 224,224,3 and I was switched to a high RAM runtime. Then when I began training RAM usage went up to 31.88gigs.

[Anshuman Kumar]

But I would like to add that once I finished that job, I have not been able to access another GPU/TPU for the past 24 hours. It is possible I was blacklisted.

[Jainil Patel]

@AnshumanKumar , give the high load in beginning only otherwise on changing configuration you will lose previously done work which in ram. I didn't used high configuration for 24 hour so I don't know about blacklisting.

[Anshuman Kumar]

Yes, that did happen with me. However the work got done.