Why can't I connect AWS RDS instance from EC2 instance in another VPC after peering

Question

I am running Tableau Server on our EC2 instance in VPC A. Meanwhile, I created a postgres RDS in another VPC B. Now I want to establish the connection between the Tableau Server and RDS. CIDR of RDS VPC is 172.31.0.0/16 and that of EC2 VPC is 10.0.0.0/16.

According to A DB Instance in a VPC Accessed by an EC2 Instance in a Different VPC, I created peering between VPC A and VPC B, pcx-xyz123. Besides, I also created the following route tables for the VPCs.

RDS VPC
Destination      Target
172.31.0.0/16    local
10.0.0.0/16      pcx-xyz123

EC2 VPC
Destination      Target
10.0.0.0/16      local
172.31.0.0/16    pcx-xyz123

Both route tables are main. Each has 0 Subnets though (not sure if this matters).

However I still can't connect RDS from Tableau Server.

The two instances are created by same account. They are both listed under US East(Ohio). So I assume they are in the same region. Plus both have us-east-2 in their hostnames. From my PC, I can connect to RDS with psql command or pgAdmin.

Why can't I connect the two instances?

EDIT: I've created another EC2 Linux instance within the same subnet of the same VPC as the Tableau Server just for debugging purposes. I configured the peering and route table same way and also associate the subnets to the route tables. However, I still can't connect to RDS on the EC2 Linux instance.

Solution 1

VPC Peering works much the same way as how Public Subnets connect to the Internet Gateway -- the Route Tables define how traffic goes in/out of the Subnets.

For VPC Peering to work:

• Invite & Accept the peering connection (Done)
• Create a Route table in each VPC that points to the Peering connection for the other VPC's IP range (Done)
• Associate each subnet that you want able to peer to the Route Table
• Alternatively, edit existing route tables to include the peering entry
• If your RDS database is public, and you are attempting to connect using the public DNS of the database, then you will need to edit the DNS settings of your peering connection to allow DNS resolution.

The routing works as follows:

• When traffic leaves a subnet, the Route Table is consulted to determine where to send the traffic
• The most restrictive (eg /24) is evaluated first, through to the least restrictive (eg /0)
• The traffic is routed according to the appropriate Route Table entry

This means that you can configure some of the subnets to peer, rather than having to include all of them. Traditionally, it is the Private subnets that peer and possibly only specific Private subnets -- but that is totally your choice.

Think of it as directions on a roadmap, telling traffic where it should be directed.

Solution 2

Below are the steps to make private RDS accessible via VPC peering:

Let’s say you have 2 VPCs:

• Production VPC: 10.0.1.0/24
• RDS VPC: 10.0.2.0/24

Step 1: create VPC peering connection between the two VPCs. Then accept the request to establish the connection. You will get a connection ID such as: pcx-e8e8e8e8

Step 2: configure route table in each VPC

• Production VPC: add this route to RDS VPC: 10.0.2.0/24 —> pcx-e8e8e8e8
• RDS VPC: add this route to Production VPC: 10.0.1.0/24 —> pcx-e8e8e8e8

Step 3: configure security group of RDS to accept the IP range of Production VPC, by adding this inbound rule

• Port (MS SQL: 1433, MySQL: 3306, etc) — allow source: 10.0.1.0/24

Should be ready for connection now.

Note: when connecting to RDS, you should use the provided DNS name for better resiliency. AWS VPC DNS will take care of resolving this name to a local IP address of the RDS instance.

Solution 3

VPC peering is all about the details. Here are the items we had to run down to get it to work.

Peer VPC 1 to VPC 2 (obvious, but included for those that did not do this step). From VPC 1, establish peering to VPC 2. Accept request. If different region, switch to VPC 2 region and accept the peer request.

Examples:

VPC 1 CIDR = 10.0.0.0/16
VPC 2 CIDR = 172.16.0.0/16

VPC 1 (VPC With RDS Instance)
1. Route Table Servicing Subnet of RDS Instance - Add route destination to VPC 2 CIDR block (172.16.0.0/16) and target VPC 2 peering connection (select from list - pcx-#####).
2. RDS Security Group - Add inbound rule for DB port with source IP being the VPC 2 CIDR block (172.16.0.0/16). So, you will have two inbound rules for the DB port. One for the VPC 1 (10.0.0.0/16) CIDR Block and one for VPC 2 (172.16.0.0/16).
3. Network Access Control List for Private Route Table - if you are only allowing certain ports, add a rule for the DB Port, source = VPC 2 CIDR block (172.16.0.0/16) and Allow.

VPC 2
1. Route Table Servicing Subnet of EC2 Instance - Add route destination to VPC 1 CIDR block (10.0.0.0/16) and target VPC 1 peering connection (select from list - pcx-#####).
2. Instance Security Group - Add inbound rule for DB port with source IP being the VPC 1 CIDR block (10.0.0.0/16).
3. Network Access Control List for Route Table - if you are only allowing certain ports, add a rule for the DB Port, source = VPC 1 CIDR block (10.0.0.0/16) and Allow.

I think that was it - but if I find another setting, I will update this message.

Just some history, we were doing this for disaster recovery. Our production instances and RDS MS SQL DB are in us-east-1 (VPC 1) and our disaster recovery warm standby instances are in us-west-2 (VPC 2). We mostly get traffic from the US, but we may consider making the standby site a true production copy (scaling group) and then changing the Route 5 records to latency based routing.

Solution 4

The question has already been answered but I wanted to add if you are connecting to public DNS of RDS(eg. prod.upd9999upd.us-east-1.rds.amazonaws.com) then you must enable DNS resolution to private IP. This is done through AllowDnsResolutionFromRemoteVpc.

Example: To connect Vpc EC2_PROD(172.0.0.0/16) to Vpc RDS_PROD(30.0.0.0/16).

1) Create Peering connection from EC2 VPC(Requester) to RDS VPC (Accepter). Make sure to enable AllowDnsResolutionFromRemoteVpc with UI by right clicking on already created peering connection and "Edit DNS Settings". Or with following command

aws ec2 modify-vpc-peering-connection-options --vpc-peering-connection-id "pcx-04a511409bb08ef16" --requester-peering-connection-options '{"AllowDnsResolutionFromRemoteVpc":true}' --accepter-peering-connection-options '{"AllowDnsResolutionFromRemoteVpc":true}' --region us-east-1

Your final peering connection will look like:

aws ec2 describe-vpc-peering-connections --profile aws-work --region us-east-1
{
    "VpcPeeringConnections": [
        {
            "Status": {
                "Message": "Active", 
                "Code": "active"
            }, 
            "Tags": [
                {
                    "Value": "ec2-to-rds-peering-connection", 
                    "Key": "Name"
                }
            ], 
            "AccepterVpcInfo": {
                "PeeringOptions": {
                    "AllowEgressFromLocalVpcToRemoteClassicLink": false, 
                    "AllowDnsResolutionFromRemoteVpc": true, 
                    "AllowEgressFromLocalClassicLinkToRemoteVpc": false
                }, 
                "VpcId": "vpc-RDS", 
                "Region": "us-east-1", 
                "OwnerId": "?", 
                "CidrBlockSet": [
                    {
                        "CidrBlock": "30.0.0.0/16"
                    }
                ], 
                "CidrBlock": "30.0.0.0/16"
            }, 
            "VpcPeeringConnectionId": "pcx-04a511409bb08ef16", 
            "RequesterVpcInfo": {
                "PeeringOptions": {
                    "AllowEgressFromLocalVpcToRemoteClassicLink": false, 
                    "AllowDnsResolutionFromRemoteVpc": true, 
                    "AllowEgressFromLocalClassicLinkToRemoteVpc": false
                }, 
                "VpcId": "vpc-ec2", 
                "Region": "us-east-1", 
                "OwnerId": "?", 
                "CidrBlockSet": [
                    {
                        "CidrBlock": "172.0.0.0/16"
                    }
                ], 
                "CidrBlock": "172.0.0.0/16"
            }
        }
    ]
}

2) Your Requester VPC (Ec2 VPC) route table must have Accepter IP cider(eg 30.0.0.0/16) added. (see Routes tab below)

aws ec2 describe-route-tables --filters Name=tag:Name,Values=EC2_PROD --profile aws-work --region us-east-1
{
    "RouteTables": [
        {
            "Associations": [
                {
                    "RouteTableAssociationId": "rtbassoc-?", 
                    "Main": true, 
                    "RouteTableId": "rtb-?"
                }
            ], 
            "RouteTableId": "rtb-?", 
            "VpcId": "vpc-EC2_PROD", 
            "PropagatingVgws": [], 
            "Tags": [
                {
                    "Value": "EC2_PROD", 
                    "Key": "Name"
                }
            ], 
            "Routes": [
                {
                    "GatewayId": "local", 
                    "DestinationCidrBlock": "172.0.0.0/16", 
                    "State": "active", 
                    "Origin": "CreateRouteTable"
                }, 
                {
                    "Origin": "CreateRoute", 
                    "DestinationCidrBlock": "30.0.0.0/16",    // Accepter IP cider block
                    "State": "active", 
                    "VpcPeeringConnectionId": "pcx-04a511409bb08ef16"
                }, 
                {
                    "GatewayId": "igw-???", 
                    "DestinationCidrBlock": "0.0.0.0/0", 
                    "State": "active", 
                    "Origin": "CreateRoute"
                }
            ]
        }
    ]
}

3) Similarly Acceptor VPC (RDS VPC) route table must have Requester IP cider(eg. 172.0.0.0/16) added. (see Routes tab below)

aws ec2 describe-route-tables --filters Name=tag:Name,Values=RDS_PROD --profile aws-work --region us-east-1
{
    "RouteTables": [
        {
            "Associations": [
                {
                    "SubnetId": "subnet-?", 
                    "RouteTableAssociationId": "rtbassoc-?", 
                    "Main": false, 
                    "RouteTableId": "rtb-?"
                }
            ], 
            "RouteTableId": "rtb-?", 
            "VpcId": "vpc-RDS", 
            "PropagatingVgws": [], 
            "Tags": [
                {
                    "Value": "RDS_PROD", 
                    "Key": "Name"
                }
            ], 
            "Routes": [
                {
                    "Origin": "CreateRoute", 
                    "DestinationCidrBlock": "172.0.0.0/16",    // Requester IP cider block
                    "State": "active", 
                    "VpcPeeringConnectionId": "pcx-04a511409bb08ef16"
                }, 
                {
                    "GatewayId": "local", 
                    "DestinationCidrBlock": "30.0.0.0/16", 
                    "State": "active", 
                    "Origin": "CreateRouteTable"
                }, 
                {
                    "GatewayId": "igw-???", 
                    "DestinationCidrBlock": "0.0.0.0/0", 
                    "State": "active", 
                    "Origin": "CreateRoute"
                }
            ]
        }
    ]
}

4) Finally also update firewall/ security group on Accepter VPC(RDS) to allow connection from Ec2 VPC on port 3306 if its mysql.

aws ec2 describe-security-groups --filters Name=tag:Name,Values=RDS_FIREWALL --profile aws-work --region us-east-1
{
    "SecurityGroups": [
        {
            "IpPermissionsEgress": [
                {
                    "IpProtocol": "-1", 
                    "PrefixListIds": [], 
                    "IpRanges": [
                        {
                            "CidrIp": "0.0.0.0/0"
                        }
                    ], 
                    "UserIdGroupPairs": [], 
                    "Ipv6Ranges": []
                }
            ], 
            "Description": "Dev", 
            "Tags": [
                {
                    "Value": "RDS_FIREWALL", 
                    "Key": "Name"
                }
            ], 
            "IpPermissions": [
                {
                    "PrefixListIds": [], 
                    "FromPort": 3306, 
                    "IpRanges": [
                        {
                            "Description": "EC2_VPC_IP_CIDER", 
                            "CidrIp": "172.0.0.0/16"
                        }
                    ], 
                    "ToPort": 3306, 
                    "IpProtocol": "tcp", 
                    "UserIdGroupPairs": [], 
                    "Ipv6Ranges": []
                }
            ], 
            "GroupName": "RDS_FIREWALL", 
            "VpcId": "vpc-???", 
            "OwnerId": "???", 
            "GroupId": "sg-???"
        }
    ]
}

Solution 5

After following all the above, I was still having issues connecting to my RDS instance. For the first time, well, ever, I actually found the answer on Reddit not SO (here: https://www.reddit.com/r/aws/comments/8hx28w/rds_access_from_a_different_vpc/dyn616i/).

Tl;Dr If you've already peered VPC's, modified route tables, and opened up the DB security group to allow connections from the source VPC CIDR (basically, @John Rotenstein's suggestion https://stackoverflow.com/a/46331624/1830623), then make sure your RDS instance is not marked as Public.

View more solutions

Comments

[John Rotenstein]

It does not matter which one initiated the peering, as long as it was accepted. Have you updated the Route Tables in each VPC to point to the Peering connection? The Security Groups? Can you please Edit your question to show the configuration? Can you try connecting between two EC2 instances across the peering connection? Or an EC2 instance and RDS? What debugging have you done?

[hiropon]

Do you tie with vpc-peering and route-table ?

[helloV]

Is the peering accepted? Can you post the routing table of VPC A? Or what is the rule in the routing table of subnet of EC2 instance for the VPC B target CIDR?

[ddd]

@JohnRotenstein Configuration of route tables added in original post

[ddd]

@helloV I updated post with route tables. And yes, peering is accepted. Somehow I don't have any subnets associated with the route tables. Should I?

[ddd]

I just associated all the subnets to the route tables of the two VPCs. However still can't connect to RDS.

[John Rotenstein]

Then you should progressively debug. Start by creating an EC2 instance in each VPC. Log into one, and from there try logging into the other. Make sure security groups permit the connection. This way, you're checking the network rather than RDS. If it works, then your peering is okay. Next would be to connect from the EC2 instance into RDS in the other VPC. Only after that should you try from Tableau, since that's harder to debug.

[ddd]

I tried to access from EC2 in one VPC to EC2 in another and it works both ways. How do I access RDS from EC2 instance rather than using psql command? Can I ssh? I can't seem to find the public IP for RDS instance though.

[John Rotenstein]

If it is a PostgreSQL database, then psql is an excellent way to test the connectivity. Also check the Security Group associated with your RDS instance. It should permit access from the private address range of both VPCs (or at least from where you want to connect).

[ddd]

Still can't get it to work between VPCs. Ended setting EC2 and RDS in the same VPC. After adding the sg of EC2 instance to the inbound rules of Security Group of RDS, it is working now.

[Naveen]

This should be -> RDS VPC: add this route to Production VPC: 10.0.1.0/24 —> pcx-e8e8e8e8

[Snowball]

Your RDS endpoint xxxxxx-db.xxxxxxxxx.us-east-2.rds.amazonaws.com likely needs DNS resolution. Go into your peering connection and edit DNS settings to true. In your EC2 instance, then nslookup and the RDS endpoint should change to an ip in this range 172.31.0.0/16

[Hector Magana]

If you forget Step 3 you get a timeout, don´t forget! Thanks for indicate every step.

[claytond]

P.S. If it didn't previously, that link now mentions the DNS resolution across VPC. Marking the RDS a private fixes the problem "brute force" by eliminating the public IP address from resolution. This may be good security practice anyway, but the "elegant" solution is to get the DNS for the RDS server resolving to the private IP address without removing the public IP address.

[Matthew Poer]

If your EC2 instance's VPC has more than one subnet, be sure to modify the route table that is referenced on the EC2 instance

[Espoir Murhabazi]

I know, we should not say thank you in SO comment, but Thank you man for these clear explanations.