How to verify a AWS VPC (S3) endpoint works?

Question

I added a VPC endpoint to my VPC using CloudFormation, and allowed s3 usage. The routes are visible in the AWS console, but not in the local routing tables of the EC2 instances:

$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         172.29.4.129    0.0.0.0         UG    0      0        0 eth0
169.254.169.254 0.0.0.0         255.255.255.255 UH    0      0        0 eth0
172.29.4.128    0.0.0.0         255.255.255.128 U     0      0        0 eth0

How do I verify that the EC2 instances in the VPC actually uses the VPC endpoint for S3, and not the available internet connection?

Solution 1

I have found a method to verify the VPC endpoint usage.

1. Log in to an AWS EC2 instance in the VPC
2. Configure the aws cli client
3. run aws ec2 describe-prefix-lists; for Windows PowerShell, Get-EC2PrefixList

The result should contain the the VPC endpoints prefix list ID in the attribute PrefixListId.

For additional verification, you can apply the following policy to an S3 bucket:

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Principal": "*",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::mybucket"
      ],
      "Condition": {
        "StringNotEquals": {
          "aws:sourceVpc": [
            "vpc-121212"
          ]
        }
      }
    }
  ]
}

with your vpc ID instead of vpc-121212. You should then only be able to access the S3 bucket from the given VPC

Solution 2

I guess the straightfwd way is to actually probe those routes.

You can traceroute to s3 and see if the NAT Gateway's internal IP is anywhere in the output (eg. the first hop).

First, check the NAT Gateway internal IPs in the console.

Example output with the endpoint set - no gateway IP shown. This is what you want to see.

$ traceroute -n -T -p 443 s3.amazonaws.com                                
traceroute to s3.amazonaws.com (52.216.204.93), 30 hops max, 60 byte packets
 1  * * *
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  52.216.204.93  0.662 ms  0.668 ms  0.637 ms

Example output of a different destination, going via NAT (see the first hop)

$ traceroute -n -T -p 443 serverfault.com
traceroute to serverfault.com (151.101.129.69), 30 hops max, 60 byte packets
 1  172.20.10.188  0.206 ms  0.147 ms  0.145 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  100.65.13.49  0.956 ms 100.65.13.113  1.253 ms *
 8  52.93.28.209  1.083 ms 52.93.28.231  1.213 ms 52.93.28.235  1.151 ms
 9  100.100.4.38  1.770 ms 100.100.4.46  2.089 ms 100.100.4.36  1.723 ms
10  103.244.50.242  1.136 ms 100.100.4.44  1.702 ms  2.738 ms
11  151.101.129.69  1.013 ms 103.244.50.244  1.745 ms 151.101.129.69  1.142 ms

Solution 3

You can turn on S3 logging and check if the files are being accessed from your private IP rather than public. If your logging shows private IPs are accessing the buckets you've configured it correctly. Goodluck!

Solution 4

I would recommend to launch ec2 instance (with IAM role allowed to list s3 buckets) in subnet without internet access.

Basically only 2 active rules in route table (your VPC subnet range and s3 endpoint).

Connect to instance and run command:

aws s3 ls /**

It should fail with timeout because boto by default will create request to global s3 url (s3.amazonaws.com).

export AWS_DEFAULT_REGION=us-east-1** ## your region here
aws s3 ls /**

should list your buckets in us-east-1 region (vpc router will route your request to s3.us-east-1.amazonaws.com).

Solution 5

Elaborating on @m-glatki solution, add a policy on the bucket that restricts S3 access to a particular VPC Endpoint:

{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::mybucket",
            "Condition": {
                "StringNotEquals": {
                    "aws:SourceVpce": "vpce-01ab2c3d4"
                }
            }
        }
    ]
}

You will only be able to list bucket contents from a process that uses the VPC endpoint. Otherwise, you will receive a message:

An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

To get the VPC Endpoint ID, use this command: aws ec2 describe-vpc-endpoints

See this link

View more solutions

Comments

[M. Glatki]

I suppose with ACL you mean SecurityGroups? I will try to add a limiting egress rule to the instance.

[Michael - sqlbot]

@M.Glatki by ACL, I believe he means endpoint policy.

[jaminto]

This command is Get-EC2PrefixList in the AWS Windows Tools for Powershell - docs.aws.amazon.com/powershell/latest/reference/items/…

[Noah Sparks]

I am not sure if this is accurate, per the docs here traffic to the endpoints do not traverse nat or internet gateways docs.aws.amazon.com/vpc/latest/userguide/vpc-endpoints.html

[Valer]

@NoahSparks That's right, and it's in line with the answer. The answer is basically: if you don't see a VPC IP as first hop then all good, it's going via endpoint; but if you see one, then you're NAT-ed and the endpoint is probably misconfigured. That's what the docs indicate.

[Noah Sparks]

Ah sorry, I guess I misread this initially and thought you were saying the example with the nat gateway in the route was the desired behavior. It's clear now though, thanks.

[duality_]

How does this prove anything or relate to the AWS documentation to give peace of mind that requests will go through the endpoint?

[Francisco Cardoso]

You might have to write bucket region in the URL like that: traceroute -n -p 443 s3.us-west-2.amazonaws.com

[Stagg]

I have setup with a S3 Gateway and NAT gateway in my route table. I see the NAT gateway IP as the first hop for s3.amazonaws.com. But when I try the bucket policy with "deny accept when from the VPC", as per other answer, I can view the bucket. That seems to suggest that it is using the gateway for S3 access. I've verified by removing the gateway, keeping the bucket policy and the access is then denied. Then I tried traceroute in MY REGION: sudo traceroute -n -T -p 443 s3.eu-west-2.amazonaws.com which didn't show the NAT gateway.You do need to use s3 with the correct region (unless us-east)