Google Compute Engine: Trouble accessing internet from an instance without external IP

Question

I am trying to install some libraries from some instances that don't have external IP addresses on a VPC network. There is one instance on that network that does have an external IP address and therefore internet access which I'm trying to use as a IP forwarding gateway.

I am currently trying to verify that I have set this up correctly:

• (Firewall rule) UDP, TCP, ICMP has been allowed on the VPC network for all instances. Have verified this by confirming that each instance can ping each other, and can do the nc -l <port> and nc -v <ip> <port> shenanigans to check they can open up a tunnel over tcp.
• (VM creation, OS) The gateway instance has been set up on creation to allow IP forwarding. It has also been enabled at an OS level with sudo sysctl -w net.ipv4.ip_forward=1.
• (Route) A new route has been created to route destination IPs for an 'allow-internet-access' tag to specify that the next hop is the gateway instance. The non-gateway instances on the VPC network have had that tag applied.

However, I still can't access internet IPs, e.g. the command sudo apt-get install default-jdk fails with:

E: Failed to fetch http://europe-west1.gce.archive.ubuntu.com/ubuntu/pool/main/libx/libxt/libxt-dev_1.1.5-0ubuntu1_amd64.deb  Unable to connect to europe-west1.gce.archive.ubuntu.com:http: [IP: 35.205.79.146 80]

Even though

1. the gateway instance can access it
2. the internal instance can access the gateway

I think it potentially may still have something to do with the routing at an OS level, as when I do an ip route get to an internet address, it gives me the default gateway (10.0.0.1), and I'm not sure if adding that new route should have changed this? (The gateway instance's IP is 10.0.0.5, the internal instance is 10.0.0.3).

ip route get 35.205.79.146
35.195.141.26 via 10.0.0.1 dev ens4  src 10.0.0.3
    cache

Any pointers on how to debug further would be much appreciated!

Solution 1

For this particular problem, I ended up following the instructions here to set up an instance-based NAT: https://cloud.google.com/vpc/docs/special-configurations - configuring an instance as a NAT Gateway. The step I was missing was the IPTables configuration; after adding that in, it fixed my problem.

As others have suggested though, using a Cloud NAT may be the best/idiomatic way to do this now on GCP for the typical use-case.

Solution 2

As mentioned in the other comments, you will need to use Cloud NAT so that your instances without IP addresses can connect to the internet. You can setup Cloud NAT by following the step by step directions here.

Comments

[Ron Trunk]

The non-public instance needs a NAT gateway, or your gateway instance needs to perform NAT for the other instance.

[kasperd]

Any reason you aren't using NAT?

[xdl]

Thanks for the links, I didn't actually know about Cloud NATs then, although will stick with the instance-based approach for now as I may need to do some ssh based port forwarding to that gateway.