List out auto scaling group names with a specific application tag using boto3

Question

I was trying to fetch auto scaling groups with Application tag value as 'CCC'.

The list is as below,

gweb
prd-dcc-eap-w2
gweb
prd-dcc-emc
gweb
prd-dcc-ems
CCC
dev-ccc-wer
CCC
dev-ccc-gbg
CCC
dev-ccc-wer

The script I coded below gives output which includes one ASG without CCC tag.

#!/usr/bin/python
import boto3

client = boto3.client('autoscaling',region_name='us-west-2')

response = client.describe_auto_scaling_groups()

ccc_asg = []

all_asg = response['AutoScalingGroups']
for i in range(len(all_asg)):
    all_tags = all_asg[i]['Tags']
    for j in range(len(all_tags)):
        if all_tags[j]['Key'] == 'Name':
                asg_name = all_tags[j]['Value']
        #        print asg_name
        if all_tags[j]['Key'] == 'Application':
                app = all_tags[j]['Value']
        #        print app
        if all_tags[j]['Value'] == 'CCC':
                ccc_asg.append(asg_name)

print ccc_asg

The output which I am getting is as below,

['prd-dcc-ein-w2', 'dev-ccc-hap', 'dev-ccc-wfd', 'dev-ccc-sdf']

Where as 'prd-dcc-ein-w2' is an asg with a different tag 'gweb'. And the last one (dev-ccc-msp-agt-asg) in the CCC ASG list is missing. I need output as below,

dev-ccc-hap-sdf
dev-ccc-hap-gfh
dev-ccc-hap-tyu
dev-ccc-mso-hjk

Am I missing something ?.

Solution 1

I got it working with below script.

#!/usr/bin/python
import boto3

client = boto3.client('autoscaling',region_name='us-west-2')

response = client.describe_auto_scaling_groups()

ccp_asg = []

all_asg = response['AutoScalingGroups']
for i in range(len(all_asg)):
    all_tags = all_asg[i]['Tags']
    app = False
    asg_name = ''
    for j in range(len(all_tags)):
        if 'Application' in all_tags[j]['Key'] and all_tags[j]['Value'] in ('CCP'):
                app = True
        if app:
                if 'Name' in all_tags[j]['Key']:
                        asg_name = all_tags[j]['Value']
                        ccp_asg.append(asg_name)
print ccp_asg

Feel free to ask if you have any doubts.

Solution 2

In boto3 you can use Paginators with JMESPath filtering to do this very effectively and in more concise way.

From boto3 docs:

JMESPath is a query language for JSON that can be used directly on paginated results. You can filter results client-side using JMESPath expressions that are applied to each page of results through the search method of a PageIterator.

When filtering with JMESPath expressions, each page of results that is yielded by the paginator is mapped through the JMESPath expression. If a JMESPath expression returns a single value that is not an array, that value is yielded directly. If the result of applying the JMESPath expression to a page of results is a list, then each value of the list is yielded individually (essentially implementing a flat map).

Here is how it looks like in Python code with mentioned CCP value for Application tag of Auto Scaling Group:

import boto3

client = boto3.client('autoscaling')
paginator = client.get_paginator('describe_auto_scaling_groups')
page_iterator = paginator.paginate(
    PaginationConfig={'PageSize': 100}
)

filtered_asgs = page_iterator.search(
    'AutoScalingGroups[] | [?contains(Tags[?Key==`{}`].Value, `{}`)]'.format(
        'Application', 'CCP')
)

for asg in filtered_asgs:
    print asg['AutoScalingGroupName']

Solution 3

Elaborating on Michal Gasek's answer, here's an option that filters ASGs based on a dict of tag:value pairs.

def get_asg_name_from_tags(tags):
    asg_name = None
    client = boto3.client('autoscaling')
    while True:

        paginator = client.get_paginator('describe_auto_scaling_groups')
        page_iterator = paginator.paginate(
            PaginationConfig={'PageSize': 100}
        )
        filter = 'AutoScalingGroups[]'
        for tag in tags:
            filter = ('{} | [?contains(Tags[?Key==`{}`].Value, `{}`)]'.format(filter, tag, tags[tag]))
        filtered_asgs = page_iterator.search(filter)
        asg = filtered_asgs.next()
        asg_name = asg['AutoScalingGroupName']
        try:
            asgX = filtered_asgs.next()
            asgX_name = asg['AutoScalingGroupName']
            raise AssertionError('multiple ASG\'s found for {} = {},{}'
                     .format(tags, asg_name, asgX_name))
        except StopIteration:
            break
    return asg_name

eg:

asg_name = get_asg_name_from_tags({'Env':env, 'Application':'app'})

It expects there to be only one result and checks this by trying to use next() to get another. The StopIteration is the "good" case, which then breaks out of the paginator loop.

Solution 4

The right way to do this isn't via describe_auto_scaling_groups at all but via describe_tags, which will allow you to make the filtering happen on the server side.

You can construct a filter that asks for tag application instances with any of a number of values:

Filters=[
        {
            'Name': 'key',
            'Values': [
                'Application',
            ]
        },
        {
            'Name': 'value',
            'Values': [
                'CCC',
            ]
        },
    ],

And then your results (in Tags in the response) are all the times when a matching tag is applied to an autoscaling group. You will have to make the call multiple times, passing back NextToken every time there is one, to go through all the pages of results.

Each result includes an ASG ID that the matching tag is applied to. Once you have all the ASG IDs you are interested in, then you can call describe_auto_scaling_groups to get their names.

Solution 5

yet another solution, in my opinion simple enough to extend:

client = boto3.client('autoscaling')

search_tags = {"environment": "stage"}

filtered_asgs = []

response = client.describe_auto_scaling_groups()

for group in response['AutoScalingGroups']:

    flattened_tags = {
        tag_info['Key']: tag_info['Value']
            for tag_info in group['Tags']
    }

    if search_tags.items() <= flattened_tags.items():
        filtered_asgs.append(group)


print(filtered_asgs)

View more solutions

Comments

[Michal Gasek]

client.describe_auto_scaling_groups() returns max 100 Auto Scaling Groups in a single API call. If you have more ASGs you must use pagination and call describe_auto_scaling_groups again with nextToken parameter, otherwise you will have incorrect results. See my answer that deals with this problem.

[Ashfaq]

Thanks a lot @Michal, This really did the job. My above script was listing only asg names with Application as first tag. But your's listed all asg names with CCP tag. Really helpful :) One question. !! PaginationConfig={'PageSize': 100} . What does this do ? Is this a limit that we should set ?. Regards, Ashfaq

[Ashfaq]

Yes! I saw your comment below. It really helped. Thank you :)

[Yugendran]

The above code works but for recent versions of python you will get the error 'generator' object has no attribute 'next'. To solve the error use the asg = next(filtered_asgs) and asgX = next(filtered_asgs).

[Mikey]

I get this error when I tried your approach: "botocore.exceptions.ClientError: An error occurred (ValidationError) when calling the DescribeTags operation: Filter type MyCusstomTag is not correct. Allowed Filter types are: auto-scaling-group key value propagate-at-launch". Did you actually run your code?

[interfect]

It's been adapted to fit the question, but this way of making filters is exactly what I use in one of my applications. You aren't supposed to replace key and value; those strings have to be literal. MyCusstomTag would have to replace Application in the example I give.