google compute engine tool gcloud is exceptionally slow

Question

I tried downloading and using the gcloud bash tool to manage my accounts, however everything I do with the tool is exceptionally slow. It will take MINUTES to reply to a command that is typed.

Is there perhaps a firewall I need to open up on my router or something else to get this to work fast like it's supposed to? For example, the "Installing..." lines in this video https://youtu.be/4y4-xn4Vi04?t=1m21s -- you'll notice they are all complete in the tutorial within a few seconds. This takes over 10 minutes to complete on my machine.

I'm on a newer Macbook Air, and all other internet/etc works really fast. I'm on a decent speed internet connection from AT&T Uverse (30mbps/3mbps). All other browsing is fast and just fine, the only thing in the world I have a problem with is this gcloud tool.

Solution 1

I traced this back to packet filters. When I deleted this rule #5 in my AT&T Uverse modem/router (Motorola NVG589), everything works ok. This was a default setting in the modem which caused all sorts of issues.

Solution 2

I just ran into a similar issue myself, though not as bad as minute long response times. What helped me was turning off the usage reporting.

Looking at some timings:

$ gcloud config set disable_usage_reporting False
$ time gcloud compute -h | tail -1
  real  0m7.058s
  user  0m0.464s
  sys   0m0.088s

A whopping 7 secs to access the help!

Fortunately, this improves greatly after disabling reporting:

$ gcloud config set disable_usage_reporting True
$ time gcloud compute -h | tail -1
  real  0m0.541s
  user  0m0.459s
  sys   0m0.080s

Much better!

Comments

[Anupam]

Are others facing the same issue? I am facing this in on Mac OS X El Captain and cant seem to find a clear solution. Running traceroute www.googleapis.com reveals the delay on remote Google servers and not at the client or router's end. gcloud version is SDK 169.0.0

[John]

using gcloud on an f1-micro instance is slower than actually starting the whole instance. I really hope Google is going to exchange this extremely bulky tool against something FAST

[Andrew Rembrandt]

I'm getting this recently (on Linux), and using strace have seen that it's when refreshing auth tokens with gcloud.py config config-helper - both with the gcloud & kubectl commands - it appears to be hanging during a DNS lookup of metadata.google.internal for me. Using Google's public DNS servers may help.

[Pierre Avérous]

@AndrewRembrandt Had the exact same behaviour (on an M1 Mac), and traced it back to this same DNS lookup that was hanging. Your solution of adding google's public DNS as a secondary DNS server to my laptop fixed the issue !

[Mark]

What version of the SDK are you currently using?

[vtrubets]

gcloud -v returns: "Google Cloud SDK 0.9.58"

[Conrad Irwin]

This answer still applies on Google Cloud SDK 137.0.0!

[Michael Younkin]

This is still an issue on Google Cloud SDK version 139.0.1 and Mac OS X Sierra. I think it's a different issue from the one noticed by Mark Shust. Commands went from taking about 10 seconds each to taking less than a second.

[bw4sz]

On windows 10, gcloud ssh into docker-optimized image, not as slow as reported about, but still extremely distracting. Disabling reporting helps only marginally.

[Anupam]

Looks like subsequent runs of the above command (without updating the config) reduces the time anyways (maybe some kind of caching). This what I got on two subsequent runs without any change in the config: First run: time gcloud compute -h | tail -1 real 0m3.031s user 0m0.756s sys 0m0.477s Second run: time gcloud compute -h | tail -1 real 0m0.585s user 0m0.426s sys 0m0.147s Also I didn't see any visible difference by disabling reporting on ver SDK 169.0.0

[grgry]

this answer helped me tremendously.. ipv6 was enabled, had to login to the arris router admin and under firewall, disable block fragmented ip packets (enabled by default..)

[Gabe Kopley]

This help me as well on AT&T Small Business fiber. In fact none of the rules were enabled for us, but I still had this problem. Disabling the packet filter completely solved it.

[JamWils]

Thank you. I have their fiber service as well. I had to disable all the packet filters as well then reset my broadband connection.

[GaryO]

Still pretty slow with v310.0.0. 1.8 sec just for gcloud -v or gcloud config get-value product on my machine.

[polvoazul]

I just disabled ipv6 on my router and it worked as well

[Andrew Rembrandt]

In addition to my comment on the OP, it turns out my real issue was actually to do with the fact that IPv6 was mis-configured on a second router in my home network (it was requesting prefix delegation when it shouldn't have). As soon as I corrected the prefix issue, the gcloud sdk delays (when renewing auth tokens I suspect) went away. NB: Running strace -f -e trace=network -s 10000 gcloud <args> shows some interesting things that one would normally not expect - it appears that google has written it to detect if it's running within google's internal networks, which can be problematic