spark-submit not found in bash

Question

I have a cronjob to run a script at startup, which i created using crontab -e

@reboot /home/ubuntu/startup.sh > /home/ubuntu/log.log 2>&1

My startup.sh script contains:

#!/bin/bash

...
bash /home/ubuntu/ec2-script.sh

and my ec2-script.sh contains:

spark-submit

But i'm getting spark-submit: command not found. If I try from terminal, it is working perfectly fine. But when the script runs at boot up, it is unable to find spark-submit command. I try to put sleep as well, so the spark starts properly, but it doesn't help. It would be great help if someone points out what is wrong or missing.

Solution

You will have to use full path for spark-submit.

Cron don't set up the environment as you have in terminal / bash. This means, among other things, that PATH (search path for executables) isn't set.

You can set the environment either in the first lines of crontab, like:

PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
OTHERVAR=/some/thing

or in your script. If you set them in the script you will have to export PATH and other variables you set, to get them passed down when you call other scripts or programs.

Comments

[Soren A]

Have you tried to use full path for spark-submit ?

[Waqar Ahmed]

Yeah, It worked with full path. Can I ask why?

[Soren A]

It works with full path, because cron don't set up the environment as you have in terminal / bash. This means, among other things, that PATH isn't set. If it works please accept my answer below.

[David Foerster]

Possible duplicate of Why crontab scripts are not working?

[Waqar Ahmed]

How can we set up the environment using cron?