Can't run AWS CLI from CRON (credentials)
Solution 1
If it works when you run it directly but not from cron there is probably something different in the environment. You can save your environment interactively by doing
set | sort > env.interactive
And do the same thing in your script
set | sort > /tmp/env.cron
And then diff /tmp/env.cron env.interactive
and see what matters. Things like PATH
are the most likely culprits.
Solution 2
When you run a job from crontab, your $HOME
environment variable is /
The Amazon client looks for either
~/.aws/config
or
~/.aws/credentials
If $HOME
= /
, then the client won't find those files
To make it work, update your script so that it exports an actual home directory for $HOME
export HOME=/root
and then put a config or credentials files in
/root/.aws/
Solution 3
I was able to solve this issue through the following:
export AWS_CONFIG_FILE="/root/.aws/config"
export AWS_ACCESS_KEY_ID=XXXX
export AWS_SECRET_ACCESS_KEY=YYYY
Solution 4
Put this code before your command line to be executed into crontab -e
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Solution 5
The aws cli tool's binaries are installed under /usr/local/bin/aws
.
The error I had is that the cron user could not access /usr/local/bin/aws
while running; it can only access /usr/bin/
What I did was to create a link in /usr/bin
for aws with the below command.
root@gateway:~# ln -s /usr/local/bin/aws /usr/bin/aws
I also added some changes in my script; here is a sample function:
starter () {
echo "
==================================================
Starting Instance
==================================================
"
/usr/bin/aws ec2 start-instances --instance-ids $instance --region us-east-1
sleep 30
echo "Assigning IP Address "
/usr/bin/aws ec2 associate-address --instance-id $instance --region us-east-1 --public-ip XX.XX.XX.XX
}
And the cron entry:
30 5 * * * sh /usr/local/cron/magentocron.sh
This method worked for me.
Related videos on Youtube
binaryorganic
I'm a Web Designer, SEO & Standards Consultant from Cleveland, Ohio.
Updated on September 18, 2022Comments
-
binaryorganic almost 2 years
Trying to run a simple AWS CLI backup script. It loops through lines in an include file, backs those paths up to S3, and dumps output to a log file. When I run this command directly, it runs without any error. When I run it through CRON I get an "Unable to locate credentials" error in my output log.
The shell script:
AWS_CONFIG_FILE="~/.aws/config" while read p; do /usr/local/bin/aws s3 cp $p s3://PATH/TO/BUCKET --recursive >> /PATH/TO/LOG 2>&1 done </PATH/TO/INCLUDE/include.txt
I only added the line to the config file after I started seeing the error, thinking this might fix it (even though I'm pretty sure that's where AWS looks by default).
Shell script is running as root. I can see the AWS config file at the specified location. And it all looks good to me (like I said, it runs fine outside of CRON).
-
ceejayoz almost 10 yearsTry an absolute path to
~/.aws/config
. -
binaryorganic almost 10 yearsDefinitely tried that first (was using /root/.aws/config), but jumped back to ~/ after seeing it in some other threads. Same error either way.
-
nico almost 9 yearsNot a direct answer but a comment about using the API keys: It is better practice (and much easier) to assign roles to your instances, and create policies around those roles, and then you are not required to specify the keys at all, or have them lying around in plaintext on the instance. Unfortunately this can only be specified at instance creation time. As an aside, for copying logfiles (and backups etc) have a look at the s3cmd tools, that provides functionality similar to rsync.
-
-
binaryorganic almost 10 yearsThanks! A step toward being able to troubleshoot the problem on my own is basically invaluable. There were definitely several differences in the PATH variable, and I kind of think in this case it was the difference in HOME that was throwing things off. As for my specific issue, I ended up just running this from the user's cron file, instead of /etc/crontab, which solved everything on my end. Thanks again!
-
Aldekein over 7 yearsMansur, your answer formatting is completely broken.
-
Fr0zenFyr over 7 yearsRight. adding a correct
PATH
variable(echo $PATH
will tell what it should be) in script usually solves it. -
Madbreaks over 7 yearsThis should be the accepted answer.
-
borracciaBlu over 6 yearson ubuntu 16.04 i couldn't find the package.
-
borracciaBlu over 6 yearsI tried the first solution with the diff but nothing. the trick for me was the PATH variable.
-
AWippler over 6 yearsDon't store
AWS_ACCESS_KEY_ID
andAWS_SECRET_ACCESS_KEY
values in scripts. The first line should have already provided those values. -
Ramratan Gupta over 6 yearsusing full path
/usr/bin/aws
is key to solution. -
Hussain7 about 5 yearsYour answer worked for me. Scratching my head for an hour. Thanks, buddy
-
the4thv almost 4 yearsAdding the PATH variable was a simple solution to this problem for me.