How to track unused resources in AWS?

Question

I have been using AWS for a while now. I always have the difficulty tracking AWS resources and how they are interconnected. Obviously, I am using Terraform but still, there is always ad-hoc operations that cut down my visibility.

Since I have been charged multiple times for resources/services that are present but not used by me.

Unused services include resources that are not pointing to other services but present in the AWS environment.

Tools suggestions are also welcome.

Solution 1

There are a few ways to approach deleting unused resources, perhaps tagging items that are part of some trusted or known group, as some have suggested here and then manually monitoring for resources that isn’t tagged appropriately.

You could then extend this to a python script that you run every so often to ID and delete these resources.

However, there are other (potentially less intensive) options.

1) AWS already provide a way for us to monitor our resources, identify unused ones and terminate them with CloudWatch alarms. You can start by configuring some alarming based on some set of checks that you might already be doing manually. Once you are comfortable that the alarming is accurate, you can add a termination step to the chain.

2) Alternatively, our friends at Netflix made this chill tool called JanitorMonkey to monitor and delete unused Instances, EBS Volumes, EBS Snapshots and Autoscaling groups. It is part of the original so-called “Simian Army” open source pack from Netflix and written in Java. Even thought it’s a few years old now, this thing is highly configurable and well worth a look.

Both options here allow you to configure warnings and business rules that fit your needs. Additionally, using CloudWatch might also be a good opportunity to learn more about AWS logs and anomaly identification.

Solution 2

If you wish to do it purely with AWS provided tools, try this:

1. figure out a tagging scheme that works for you
2. tag all the resources you provision, make sure nothing is left untagged
3. for stuff you provision outside of the terraform and are not always tagged correctly by your tagging scheme, use AWS Config service to enforce the tagging scheme

After you have all of your resources tagged, use AWS Budgets to monitor and tracked down changes in you spend and get alerts when it crosses the budget threshold you define.

Also, create and periodically review some reports within the AWS Costs Explorer to find daily changes in region spend / service spend / etc.

Using a strict and meaningful tagging scheme helps maintain a healthy account with less money going to waste.

Solution 3

I created such a tool, it is called AWS Inventory and eventually, it will cover all available resources in your account. While it is still a work in progress, should be very easy to add the items you might be missing just by adding the API name to a list.

The tool is just a single HTML file that uses aws-sdk.js to query all the different list... and describe... APIs of CORS-enabled services.

The project is MIT license and open source at https://github.com/devops-israel/aws-inventory.

Solution 4

That would be one way of doing, using just the AWS console.

I have been using TotalCloud for many other things but this is very easily possible on the platform. It seems to be free for now. I have been using it with two large AWS accounts for 3 months now.

It shows everything in a single view with visual topological view which makes it fairly easy to understand the relationships between resources/services. The grouping can definitely help you find the resources as needed.

View more solutions

Comments

[Admin]

It's not clear what you mean by unused resources. Do you mean resources that are stopped (i.e. not being utilized)? Resources that are running, but not performing the operations that you need? An active EBS volume not associated with any EC2 instance or an ELB not pointing to any instances?

[Admin]

@PrestonM I am referring to both. Primarily to ELBs that are not pointing to any instance but also resources that are stopped.

[Veer Abheek Singh Manhas]

The issue with tagging is. You have to do it for everything you need. Obviously, I have my critical tags present. Every now and then something new comes along and I have to change the tagging scheme as the KPIs change. It's not really a scalable solution.

[tolib]

it is if you automate all of your tagging have a look at graffiti-monkey / similar tools that help deploy tags / ensure a tagging scheme

[Anh]

tried it, pretty awful.

[Tim Malone]

@Evgeny Mind adding why? Seems like a glowing recommendation above, and the answer's been given a bounty, so it'd be useful to see why it didn't work for you to help others make an informed decision.

[Anh]

The "why" is that it is very low-quality software. Horribly broken. Much worse than the AWS console. Trying to "see" what you actually have in AWS based on using this software is a futile exercise. Sorry, some software is just no good.

[silverdagger]

Janitor monkey seems to be retired.