Cloudwatch Log Alert - How to include error / exception / stack trace data in email notification

Question

I just configured Cloudwatch logs on my ec2 instances and am loving it so far. I also set up alerts for certain keywords, like "ERROR". While the email alert seems to be working fine, I was wondering if there's a way to fine-tune the alert email to make it a little concise & informative. Specifically, I'm looking to

1. Get rid of all the boilerplate text in the alert email.

2. Include some information about the Error/Exception that triggered the alert. This could be something as simple as including the log statement that generated the alert.

Right now, the alert email looks like

You are receiving this email because your Amazon CloudWatch Alarm "App-Error-Alarm" in the US East - N. Virginia region has entered the ALARM state, because "Threshold Crossed: 1 datapoint (1.0) was greater than or equal to the threshold (1.0)." at "Tuesday 07 February, 2017 16:39:43 UTC".

View this alarm in the AWS Management Console: https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#s=Alarms&alarm=App-Error-Alarm

Alarm Details: - Name: App-Error-Alarm - Description: Errors in app.log - State Change: INSUFFICIENT_DATA -> ALARM - Reason for State Change: Threshold Crossed: 1 datapoint (1.0) was greater than or equal to the threshold (1.0). - Timestamp: Tuesday 07 February, 2017 16:39:43 UTC - AWS Account: <>

Threshold: - The alarm is in the ALARM state when the metric is GreaterThanOrEqualToThreshold 1.0 for 300 seconds.

Monitored Metric: - MetricNamespace: LogMetrics - MetricName: ERROR - Dimensions: - Period: 300 seconds - Statistic: Sum - Unit: not specified

State Change Actions: - OK: - ALARM: [arn:aws:sns:us-east-1:<>:support] - INSUFFICIENT_DATA:

I'd like it to something like

Alarm: App-Error-Alarm

Keyword: "ERROR"

Reason: ERROR 2017-02-07 07:31:47,375 [SimpleAsyncTaskExecutor-5] com.app.server.rest.Watcher: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure

Its short, sweet and instantly tells me whether its something that needs my immediate attention. Can this be done without writing code as suggested here?

Solution 1

You have this problem because you configured an alarm and is meant for aggregated data, not for specific log record. You configure it for some metric (number of log records with ERROR keyword).

You can use log subscription instead and stream all log records matching a filter to a custom Lambda function. You can use it to send notifications to email or Slack.

To configure log streaming, go to Lambda in AWS console and create a new function from a blueprint named "cloudwatch-logs-process-data". It has a basic structure and is easy to customize to your needs.

Solution 2

You can customize Cloudwatch alarms by using Cloudwatch Logs Customize alarms Package provided by AWS labs.

When you get an alarm, you want enough information to decide whether it needs immediate attention or not. You also want to customize the alarm text to operational needs. The CloudWatch Logs Customize Alarms is a Lambda function that helps in reading the logs from CloudWatch Logs during an alarm and send a customized email through SES.

The packages provides for using a Lambda SNS endpoint, which can then customize the alarm notification according to your operational requirements.

Comments

[Mark B]

As far as I know this can not be done.

[Charlie Schliesser]

What if we have many Lambdas and we want to receive SNS notification for any errors logged in any of them? Do we need a CloudWatch-subscribed Lambda for every log group?

[Rob Schmuecker]

@CharlieSchliesser - You can set the trigger to be invoked on /aws/lambda and then set the filter according to what you want/need to listen for. You can get pretty expressive and easily get a lot of coverage with this method. docs.aws.amazon.com/AmazonCloudWatch/latest/logs/…

[Aspiro]

@RobSchmuecker Can you explain how to trigger on /aws/lambda? I've searched around, but have not found it. Thanks.

[Fábio Paiva]

What about costs? It seems it will trigger the function on every single log message

[Igor Romanov]

@FábioPaiva no, only for log records matching a filter. For example all ERRORs

[Boris Le Méec]

How can you listen for /aws/lambda ? I can't figure it out. AWS always says "that log group does not exists"

[2540625]

I'm not permitted to choose /aws/lambda, I'm required to choose one function.