Easiest way to automatically check EC2 disk space and be alerted if it is running low?

Question

Running the Amazon Linux AMI. It seems that CloudWatch does not check for free disk space. I have a number of servers and ideally don't want to have to configure each one with a mail server, script to check disk space etc.

Is there a simpler way to do this?

Solution 1

Amazon provides scripts for this as of march 2012:

Amazon CloudWatch Monitoring Scripts for Linux: http://aws.amazon.com/code/8720044071969977

Solution 2

There is no way for the EC2 control and monitoring tools to give you this data because the file system of your instances is ONLY accessible by the instance itself. Both the basic architecture of the hardware and the security model demand this limitation. Think about how bad it would be if software outside your computer could poke around at the files on your hard drives!

Here is a low key way to make cron (installed on most systems anyway) check this data for your periodically. Your systems should have the minimum requirements to handle root mail notifications anyway. I recommend having at least a materialistic outgoing mail agent and configuring the root or administrator alias to forward to you on all systems you administer. Many programs including cron expect this configuration.

You could add this to your crontab:

0 0 * * * test $(df / | grep ^/ | awk '{print $4}') -lt 1048576 && echo "Warning: Free disk space is less than 1G on /"

To break that down, this

• Creates a job that runes once a day at 00:00.
• Cron automatically handles emailing the system administrator with the output of jobs. This job only produces output if there is an error or if the disk space is low
• The test command sets up a simple shell comparison using the -lt less than operator and a fixed value equivolent to 1Gb free space.
• The df command tests free space on the / file system
• The grep gets you just the line of output you need instead of the headers df includes.
• The awk get's just the fourth column in the output, the free space number.
• The && says to run the next command only if the first one (the test x -lt y) returns true.

Solution 3

I wrote a script as I needed to check several servers within my EC2 group. It needs a file with a list of each server IP/domain name on a single line.

#! /bin/bash

ADMIN="[email protected]"
ALERT=85

for SERVER in `cat ~/scripts/servers.txt` do
ssh -i ~/.ssh/yourkey.pem $SERVER df -H | grep -vE '^Filesystem|tmpfs|cdrom' | awk '{ print $5 " " $1 }' | while read output;
do
echo $output
usep=$(echo $output | awk '{ print $1}' | cut -d'%' -f1  )
partition=$(echo $output | awk '{ print $2 }' )
if [ $usep -ge $ALERT ]; then
echo "Running out of space \"$partition ($usep%)\" on $SERVER as on $(date)" | 
mail -s "Alert: Almost out of disk space $usep" $ADMIN
fi
done done

Solution 4

Step by step instructions for setting this up on an EC2 instance with CloudWatch:

http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/mon-scripts.html

View more solutions

Comments

[Laurion Burchall]

The "Amazon CloudWatch Monitoring Scripts for Linux" can push the disk usage into CloudWatch as custom metrics. docs.amazonwebservices.com/AmazonCloudWatch/latest/…

[Joe Constant]

@LaurionBurchall please put that as an answer. IMO that is the correct answer as it gives the ability to use CloudWatch alerts.

[sergiopereira]

The scripts look fairly straight forward. The thing that worries me is that it needs a known pair of AWS access key and secret. Does anyone know if it will also look in the EC2 instance role to get a temporary pair or if I'll have to code that myself?

[sergiopereira]

To answer my own question. Yes! The scripts will use the IAM Role of the EC2 instance (if present.) The catch is that the role you chose must have the necessary CloudWatch permissions.