Why can't an AWS lambda function inside a public subnet in a VPC connect to the internet?

Question

I've followed the tutorial here to create a VPC with public and private subnets.

Then I set up an AWS lambda function inside the public subnet to test if it could connect to the outside internet.

Here's my lambda function written in python3

import requests

def lambda_handler(event, context):
    r = requests.get('http://www.google.com')
    print(r)

The function above failed to fetch the content of http://www.google.com when I set it inside the public subnet in a VPC.

Here's the error message:

"errorMessage": "HTTPConnectionPool(host='www.google.com', port=80): Max retries exceeded with url: / (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 110] Connection timed out',))", "errorType": "ConnectionError",

I don't understand why.

The route table of the public subnet looks like this:

The GET request to http://www.google.com should match igw-XXXXXXXXX target. Why can't the internet-gateway(igw) deliver the request to http://www.google.com and get back the website content?

This article says that I must set the lambda function inside the private subnet in order to have internet access.

If your Lambda function needs to access private VPC resources (for example, an Amazon RDS DB instance or Amazon EC2 instance), you must associate the function with a VPC. If your function also requires internet access (for example, to reach a public AWS service endpoint), your function must use a NAT gateway or instance.

But it doesn't explain why I can't set the lambda function inside the public subnet.

Solution

Lambda functions connected to a VPC public subnet cannot typically access the internet.

To access the internet from a public subnet you need a public IP or you need to route via a NAT that itself has a public IP. You also need an Internet Gateway (IGW). However:

1. Lambda functions do not, and cannot, have public IP addresses, and
2. the default route target in a VPC public subnet is the IGW, not a NAT

So, because the Lambda function only has a private IP and its traffic is routed to the IGW rather than to a NAT, all packets to the internet from the Lambda function will be dropped at the IGW.

Should I Configure my Lambda Function for VPC Access?

If your Lambda function does not need to reach private resources inside your VPC (e.g. an RDS database or Elasticsearch cluster) then do not configure the Lambda function to connect to the VPC.

If your Lambda function does need to reach private resources inside your VPC, then configure the Lambda function to connect to private subnets (and only private subnets).

NAT or Not?

If the Lambda function only needs access to resources in the VPC (e.g. an RDS database in a private subnet) then you don't need to route through NAT.

If the Lambda function only needs access to resources in the VPC and access to AWS services that are all available via private VPC Endpoint then you don't need to route through NAT. Use VPC Endpoints.

If your Lambda function needs to reach endpoints on the internet then ensure a default route from the Lambda function's private subnets to a NAT instance or NAT Gateway in a public subnet. And configure an IGW, if needed, without which internet access is not possible.

Be aware that NAT gateway charges per hour and per GB processed so it's worth understanding how to reduce data transfer costs for NAT gateway.

Best Practices

When configuring Lambda functions for VPC access, it is an HA best practice to configure multiple (private) subnets across different Availability Zones (AZs).

Intermittent Connectivity

Be sure that all the subnets you configure for your Lambda function are private subnets. It is a common mistake to configure, for example, 1 private subnet and 1 public subnet. This will result in your Lambda function working OK sometimes and failing at other times without any obvious cause.

For example, the Lambda function may succeed 5 times in a row, and then fail with a timeout (being unable to access some internet resource or AWS service). This happens because the first launch was in a private subnet, launches 2-5 reused the same Lambda function execution environment in the same private subnet (the so-called "warm start"), and then launch 6 was in a public subnet (a "cold start") where the Lambda function has no route to the internet.

Comments

[kichik]

You need to setup a NAT gateway docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.htm‌​l

[LLL]

Is there a reason why you advise to run the lambda in a private subnet? Any drawbacks vs running it on a public one?

[jarmod]

@LLL stepping back, you run Lambda functions in a VPC if you need them to run there, e.g. because they need to access private resources such as a MySQL DB inside the VPC or an S3 bucket restricts access to a specific VPC via private endpoint. The drawback of running in VPC is cold start latency is higher than when not run in VPC (because an ENI must be attached). The difference between public and private subnets is routing (what the 0.0.0.0/0 default route points to, IGW or NAT). If your Lambda needs outbound access then it won't work in a public subnet, because the default route is the IGW.

[LLL]

Thanks! I didn't realize that only private subnets can have a NAT GW, so it makes perfect sense. BTW while going this rabbit hole I also found out that you actually CAN have a lambda access the internet from a public subnet, but an elastic IP needs to be attached to its ENI. I know it's silly but it's nice to know.

[jarmod]

@LLL You're welcome. My understanding is that, while you technically can do that, you probably shouldn't. See stackoverflow.com/questions/55771064/…

[LLL]

Nice! I was wondering how concurrency would be handled using that solution and that explains it - it wouldn't. Thanks again!

[Aardvark]

And it all changes again. AWS have listened regarding the cold start and ENI consumption issues and now support Hyperplain for Lambda, you still have to add the appropriate IAM permissions extra, but it looks like significant improvements : aws.amazon.com/blogs/compute/…

[Ken Colton]

Thank you! This was an extremely clear explanation.

[Ganesh Satpute]

I'm not a networking person, but why doesn't AWS do NATing for instances without public IP? docs.aws.amazon.com/vpc/latest/userguide/…

[jarmod]

@GaneshSatpute AWS offers a managed NAT Gateway option and there are NAT instance options, so you can pro-actively choose NAT if it makes sense for your application (though it would only operate for instances in private subnets because of the earlier default routing constraint). Many customers need to prevent their compute instances from routing to the internet so it’s not the default. Plus NAT has a charge.

[nikhil]

Well explained, Thanks much.

[Santhos Ramalingam]

In case your lambda (inside vpc) is accessing specific resources like SSM, you can create a VPC endpoint instead of a NAT gateway.

[Robert Reiz]

Technically the public/private subnet + NAT solution is great. But the NAT Gateway is quiet expensive. That's why I would like to avoid it!

[jarmod]

@RobertReiz yes, NAT gateway charges per hour and per GB processed so it's worth understanding how to reduce data transfer costs for NAT gateway.

[Aditya Nair]

Thanks mate...It was extremely useful!

[Karthick Selvam]

If the lambda has to connect to dynamo and external api.. then should be assign it to both a private subnet and a public subnet?

[jarmod]

@KarthickSelvam No, don't do that. Such a Lambda function fundamentally needs a network route to both DynamoDB and to the external API. Neither of these intrinsically requires the Lambda to be connected to your VPC at all. If, however, the Lambda function is connected to your VPC (for compliance reasons or because it also needs to access private resources in your VPC e.g. RDS) then the Lambda should be in a private subnet. Your route to DynamoDB is then via VPC Endpoint (if you need that) or NAT, and your route to the external API is via NAT.