gcloud preview app deploy process takes ~8 minutes, is this normal?

Question

Trying out new flexible app engine runtime. In this case a custom Ruby on Rails runtime based on the google provided ruby runtime.

When firing of gcloud preview app deploy the whole process takes ~8 minutes, most of which is "updating service". Is this normal? And more importantly, how can I speed it up?

Regards,

Ward

Solution 1

Yes, that is totally normal. Most of the deployment steps happen away from your computer and are independent of your codebase size, so there's not a lot you can do to speed up the process.

Various steps that are involved in deploying an app on App Engine can be categorized as follows:

1. Gather info from app.yaml to understand overall deployment
2. Collect code and use the docker image specified in app.yaml to build a docker image with your code
3. Provision Compute Instances, networking/firewall rules, install docker related tools on instance, push docker image to instance and start it
4. Make sure all deployments were successful, start health-checks and if required, transfer/balance out the load.

The only process which takes most of time is the last part where it does all the necessary checks to make sure deployment was successful and start ingesting traffic. Depending upon your code size (uploading code to create container) and requirements for resources (provisioning custom resources), step 2 and 3 might take a bit more time.

If you do an analysis you will find that about 70% of time is consumed in last step, where we have least visibility into, yet the essential process which gives app-engine the ability to do all the heavy lifting.

Solution 2

Deploying to the same version got me from 6 minutes to 3 minutes in subsequent deploys.

Example:

$ gcloud app deploy app.yaml --version=test

Solution 3

Make sure you check what is in the zip it's uploading (it tells you the location of this on deploy), and make sure your yaml skip_files is set to include things like your .git directory if you have one, and node_modules

Solution 4

Note that the subsequent deploys should be much faster than 8 mins. It's usually 1 minute or less in my tests with Node.js on App Engine Flex.

Solution 5

As suggested above by @ludo you could use in the meantime Google App Engine Standard instead of Flex. Which, takes approximately ~30-50 seconds after the first deployment.

You can test GAE Standard by running this tutorial, which doesn't require a billing account:

https://codelabs.developers.google.com/codelabs/cloud-app-engine-springboot/index.html#0

And agreed. this doesn't address GAE Flex but gives some options to accelerate during development.

View more solutions

Comments

[Igor Artamonov]

that's normal, you're building & uploading docker images most the time

[Nan Hui]

I have answer this question at there, maybe you could see it. stackoverflow.com/a/37806746/2746292

[slideshowp2]

I am stuck Updating service [flex-env-get-started] (this may take several minutes)...⠏. It takes me 20 mins.....

[Vikram Tiwari]

Not sure, but that should be the case if you were deploying over the same service and version. If it's a newer version or a different service, it takes similar time durations. That means patches get applied quickly, but a rollout/rollback is slow.

[Ward Bekker]

In my experience deployment times of subsequent versions don't improve.

[Luís Brito]

Mine are always ~10min in the Gitlab CI

[Randy L]

yeah AppEngine is getting to the point where it's not even usable, due to these 10+ minute deployments

[colinwong]

Subsequents take about 5 mins with the same version. Never seen 1 minute or less.

[Ali Khosro]

I think you are using standard env, the question is about flexible env and it takes soooo long

[Darren]

I've been dealing with this and we routinely clock in at 12-15 mins for an aspnet core flex instance. I don't buy the checking/load balancing argument because deploying with no-promote still takes as long, the startup times for the app are sub 2 seconds, the health checks are light (we get 2 per second from early on). Docker image is built in < 1 minute. You can even download the docker image and deploy on your own machine 10x faster than they can. Something is seriously broken in the flex instance deployment ecosystem.

[Vikram Tiwari]

Hey Darren - It might have certainly changed. Recently gVisor (github.com/google/gvisor) was announced and they might be trying to use it and it might have some of these performance implications.

[ludo]

Hi, Flex deployment can be slow (as opposed to App Engine Standard), but the new gVisor product announced in Open Source is in no way in the deployment path...

[poy]

I don't know that I have seen this help. Its about the same.

[Eyal Levin]

It's been a while since I wrote this answer and things might have changed in GCP.

[ndtreviv]

With the recent updates requiring all deployments to go through gcloud the Standard Environment takes just as long. I kicked mine off over 17 minutes ago, and it's still going.

[scipilot]

You can also use a .gcloudignore file to prevent stuff getting uploaded.