Use boto3 to get current price for given EC2 instance type

Question

Now that AWS have a Pricing API, how could one use Boto3 to fetch the current hourly price for a given on-demand EC2 instance type (e.g. t2.micro), region (e.g. eu-west-1) and operating system (e.g. Linux)? I only want the price returned. Based on my understanding, having those four pieces of information should be enough to filter down to a singular result.

However, all the examples I've seen fetch huge lists of data from the API that would have to be post-processed in order to get what I want. I would like to filter the data on the API side, before it's being returned.

Solution 1

Here is the solution I ended up with. Using Boto3's own Pricing API with a filter for the instance type, region and operating system. The API still returns a lot of information, so I needed to do a bit of post-processing.

import boto3
import json
from pkg_resources import resource_filename

# Search product filter. This will reduce the amount of data returned by the
# get_products function of the Pricing API
FLT = '[{{"Field": "tenancy", "Value": "shared", "Type": "TERM_MATCH"}},'\
      '{{"Field": "operatingSystem", "Value": "{o}", "Type": "TERM_MATCH"}},'\
      '{{"Field": "preInstalledSw", "Value": "NA", "Type": "TERM_MATCH"}},'\
      '{{"Field": "instanceType", "Value": "{t}", "Type": "TERM_MATCH"}},'\
      '{{"Field": "location", "Value": "{r}", "Type": "TERM_MATCH"}},'\
      '{{"Field": "capacitystatus", "Value": "Used", "Type": "TERM_MATCH"}}]'


# Get current AWS price for an on-demand instance
def get_price(region, instance, os):
    f = FLT.format(r=region, t=instance, o=os)
    data = client.get_products(ServiceCode='AmazonEC2', Filters=json.loads(f))
    od = json.loads(data['PriceList'][0])['terms']['OnDemand']
    id1 = list(od)[0]
    id2 = list(od[id1]['priceDimensions'])[0]
    return od[id1]['priceDimensions'][id2]['pricePerUnit']['USD']

# Translate region code to region name. Even though the API data contains
# regionCode field, it will not return accurate data. However using the location
# field will, but then we need to translate the region code into a region name.
# You could skip this by using the region names in your code directly, but most
# other APIs are using the region code.
def get_region_name(region_code):
    default_region = 'US East (N. Virginia)'
    endpoint_file = resource_filename('botocore', 'data/endpoints.json')
    try:
        with open(endpoint_file, 'r') as f:
            data = json.load(f)
        # Botocore is using Europe while Pricing API using EU...sigh...
        return data['partitions'][0]['regions'][region_code]['description'].replace('Europe', 'EU')
    except IOError:
        return default_region


# Use AWS Pricing API through Boto3
# API only has us-east-1 and ap-south-1 as valid endpoints.
# It doesn't have any impact on your selected region for your instance.
client = boto3.client('pricing', region_name='us-east-1')

# Get current price for a given instance, region and os
price = get_price(get_region_name('eu-west-1'), 't3.micro', 'Linux')
print(price)

This example outputs 0.0114000000 (hourly price in USD) fairly quickly. (This number was verified to match the current value listed here at the date of this writing)

Solution 2

I have updated toringe's solution a bit to handle different key errors

def price_information(self, instance_type, os, region):
        # Search product filter
        FLT = '[{{"Field": "operatingSystem", "Value": "{o}", "Type": "TERM_MATCH"}},' \
              '{{"Field": "instanceType", "Value": "{t}", "Type": "TERM_MATCH"}}]'
    
        f = FLT.format(t=instance_type, o=os)
        try:
            data = self.pricing_client.get_products(ServiceCode='AmazonEC2', Filters=json.loads(f))
            instance_price = 0
            for price in data['PriceList']:
                try:
                    first_id =  list(eval(price)['terms']['OnDemand'].keys())[0]
                    price_data = eval(price)['terms']['OnDemand'][first_id]
                    second_id = list(price_data['priceDimensions'].keys())[0]
                    instance_price = price_data['priceDimensions'][second_id]['pricePerUnit']['USD']
                    if float(price) > 0:
                        break
                except Exception as e:
                    print(e)
            print(instance_price)
            return instance_price
        except Exception as e:
            print(e)
            return 0

Solution 3

If you don't like the native function, then look at Lyft's awspricing library for Python. Here's an example:

import awspricing

ec2_offer = awspricing.offer('AmazonEC2')

p = ec2_offer.ondemand_hourly(
  't2.micro',
  operating_system='Linux',
  region='eu-west-1'
)

print(p) # 0.0126

I'd recommend enabling caching (see AWSPRICING_USE_CACHE) otherwise it will be slow.

Comments

[toringe]

That is correct for the EC2 API, but Boto3 now has a separate Pricing API that do return on-demand prices.

[toringe]

As a FYI, I also had to create a mapping function to translate a region name (short id) to a full region name, as I usually use the short naming convention. I implemented this by utilizing the endpoints.json file included with the botocore module, as mentioned here.

[Brian]

Did you refine this any further? The solution above doesn't handle pagination, which is required if the initial filtering isn't specific enough. Are there other filters you've been successful with?

[Brian]

For what it is worth, I found this: aws.amazon.com/blogs/aws/…

[toringe]

@Brian can you elaborate on what you mean if the filtering isn't specific enough. I'm using the above code in production and haven't had an issue with pagination. (I now added the mapper function I mentioned in my comment).

[Brian]

Your use case might be specific enough actually since you are requesting an instanceType. If you leave that filter out, you get far more results that forces pagination, unless I'm doing something wrong. In my case, I'm looking for pricing, as well as just getting a list of instanceType.

[toringe]

That is probably true. But in my use-case I always have a specific instance type in addition to the other two facts when querying for a price.

[userqwerty1]

It's probably a recent change in the aws api, but now you should add `'{{"Field": "capacitystatus", "Value": "Used", "Type": "TERM_MATCH"}},'` to the filters to get the price for ondemand instances, otherwise code above returns unreliable results, because each price list contains 3 items in a random order per instance with different capacitystatus and different prices more info stackoverflow.com/a/55131304/1952982

[Caesar]

@toringe Could you update your answer with what userquerty1 said? In the current state, it returns 0 for m5d.8xlarge.

[toringe]

Thanks userqwerty1 and Caesar for your suggestions. Answer has now been updated.

[étale-cohomology]

What are valid values for location? Is it something like Oregon, or something like us-west-2?

[toringe]

@étale-cohomology: The location field in the Pricing API contains region names like US East (N. Virginia) and EU (Ireland) instead of region codes like us-east-1 and eu-west-1. Since most other APIs use region codes instead of names, I found it necessary to do a translation.