Configuring correct firewall settings for GCP HTTP load balancer

Question

I have two VMs running HTTP servers on port 8545 as shown in the figure and placed in an instance group.

I created a HTTP load balancer in GCP to balance HTTP requests from end users. However, with the setup shown in the figure and the firewall rules, the health checks fail and I also can't access http://35.190.34.26:80/

Error: Server Error
The server encountered a temporary error and could not complete your request.
Please try again in 30 seconds.

However, if I add a firewall rule, which allows everything (0.0.0.0/0 on 8545), the health checks pass and I can also access http://35.190.34.26:80/

But, I don't want to write this firewall rule which allows everyone on 8545. What is the correct way to configure the firewall so that only the load balancer can access the instance group on this port?

Solution

As the global forwarding rule is configured on port 80 but the backend instances are serving traffic on port 8545, two separate firewall rules need to be created to allow traffic from 130.211.0.0/22 and 35.191.0.0/16 on those ports. These are IP address ranges that the load balancer uses to connect to backend instances. This rule allows traffic from both the load balancer and the health checker.

I would suggest to check on the backend instances to make sure it is listening on all addresses (0.0.0.0/0) and is not bind to any particular IP. Additionally, running tcpdump on the backend instances would be helpful to know if the traffic is reaching the instances from the above IP ranges and to continue further debugging.

More information about the firewall rules regarding Google HTTP(S) Load Balancing can be found in this help center article.