Azure Load Balancer Resource Limits Hit, Ports Exhausted

Question

Found several hundred events in Azure Diagnostics from the past week of the following type:

Category: LoadBalancerAlertEvent
OperationName: LoadBalancerAlertEvent
eventName_s: Resource Limits Hit
eventDescription_s: Ports exhausted
ResourceProvider: MICROSOFT.NETWORK
ResourceType: LOADBALANCERS

I can't find any guidance in the Microsoft docs regarding any resource limits the load balancers might have. We're not operating a huge site by any stretch, so I'm surprised to find I'm hitting limits of any kind.

This is for a simple layer-4 Azure Load Balancer with 3 backend virtual machines, a public-facing frontend IP pool with 3 rules. 2 health probes.

Solution

find any guidance in the Microsoft docs regarding any resource limits the load balancers might have.

In Azure, the load balancer configuration supports full cone NAT for UDP. Full cone NAT is a type of NAT where the port allows inbound connections from any external host (in response to an outbound request).

For each new outbound connection that a virtual machine initiates, an outbound port is also allocated by the load balancer. The external host sees traffic with a virtual IP (VIP)-allocated port. For scenarios that require a large number of outbound connections, it is recommended to use instance-level public IP addresses so that the VMs have a dedicated outbound IP address for SNAT. This reduces the risk of port exhaustion.

The maximum number of ports that can be used by the VIP or an instance-level public IP (PIP) is 64,000. This is a TCP standard limitation.

More information about Load balancer Source NAT, please refer to the link.

Comments

[designarti]

I think this is it! Our VMs were configured without a public IP address. I just checked our Azure SQL sys.dm_exec_connections and it appears every connection to SQL is going back through the load balancer as well. The web servers also do make outbound calls to the Internet. I'm adding dynamic public IP addresses to all VMs now.