how to schedule the shell script using Google Cloud Shell?

Question

I have a .sh file that is stored in GCS. I am trying to schedule the .sh file through google cloud shell.

I can run the same file using gsutil cat gs://miptestauto/baby.sh | sh command but not able to schedule it.

Following is my code for scheduling the file:

16 17 * * * gsutil cat gs://miptestauto/baby.sh | sh

It displays the message as "auto saving..done" but the scheduled job is not get displayed when I use crontab -l

# contents of .sh file 
bin/bash
bq load --source_format=CSV babynames.baby_destination13 gs://testauto/yob2010.txt name:string,gender:string,count:integer

Please can anyone tell me how schedule it using google cloud shell. I am not using compute engine/app engine. Just wanted to schedule it using the cloud shell.

thank you in advance :)

Solution 1

To schedule a script you first have to create a project if you don’t have one. I assume you already have a project so if that’s the case just create the instance that you want for scheduling this script.

To create the new instance:

1. At the Google Cloud Platform Console click on Products & Services which is the icon with the four bars at the top left hand corner.

2. On the menu go to the Compute section and hover on Compute Engine and then click on VM Instances.

3. Go to the menu bar above the instance section and there you will see a Create Instance button. Click it and fill in the configuration values that you want your new instance to have. The values that you select will determine your VM instance features. You can choose, among other values, the name, zone and machine type for your new instance.
4. In the Machine type section click the drop-down menu tab to select an “f1-micro instance”.

5. In the Identity and API access section, give access scope to the Storage API so that you can read and write to your bucket in case you need to do so; the default access scope only allows you to read. Also enable BigQuery API.

6. Once you have the instance created and access to the bucket, just create your cron job inside your new instance: In the user account under which the cron job will execute, run crontab -e and edit this file to run the cron job that will execute your baby.sh script.The following documentation link should help you with this.

Please note, if you want to view output from your script you may need to redirect it to your current terminal.

Solution 2

As per the documentation, Cloud Shell is intended for interactive use only. The Cloud Shell instances are provisioned on a per-user, per-session basis and sessions are terminated after an hour of inactivity.

In order to schedule a daily cron job, the instance needs to be up and running all time but this doesn’t happen with Cloud Shell and I believe your jobs are not running because of this.

When you start Cloud Shell, it provisions a f1-micro instance which is the same machine type you can get for free if you are eligible for “Always Free”. Therefore you can create a f1-micro instance, configure the cron job on it and leave it running so it can execute the daily job.

You can check free usage limits at https://cloud.google.com/compute/pricing#freeusage

Solution 3

You can also use the Cloud Scheduler product https://cloud.google.com/scheduler which is a serverless managed Cron like scheduler.

Comments

[Willian Fuks]

I usually use AppEngine standard environment for this type of task, it works as a serverless solution and you can interact with GCP any way you want for the most part.

[samit]

Thank u Willian Funks :) u edited very nicely.. .. Do you have steps that you used for your environment ..can you share more about the appEngine environment that you used..I'd love to give a try!! thank u

[Willian Fuks]

I have this repository where I'm using GAE cron feature, it might guide you a little bit on how to setup one (but it's still under development). Other than that official docs are quite good as well.

[samit]

thank u for your help...If i wanted to schedule a file then how can i do it? pls help me out :) thank u