Multiple Nics, multiple default gateways on a Server 2012R2 cluster

Question

This is really just a question about best practices as there seems to be a little confusion. The physical hosts of our Hyper-V cluster have 4 Nic interfaces relating to the 4 subnets.

Now, when running the best practices analyzer, it loves to complain that most of these interfaces can't talk to each other. This is because all but one of the Nics have no gateway defined in the static settings. I was under the impression that best practice was to define only the one gateway, but I could be wrong.

So, the question:

Is it best practice to define one gateway per server, per subnet or per nic (in a cluster environment)? And, What are the effects of defining multiple gateways on a cluster environment?

Solution 1

For every network that the cluster members have in common (having a NIC in the same subnet), the cluster will create a cluster network and enable cluster communication for that cluster network (cluster heartbeat traffic, CSV metadata traffic and redirected CSV traffic). The cluster validation wizard will attempt to communicate from all cluster networks to all other cluster networks across all cluster members (such that if any single cluster network is unavailable on any cluster member, the cluster can still communicate with that cluster member across the other cluster networks). The fact that it can't communicate from a routed cluster network to a non-routed cluster network is expected. What you should do is determine which cluster networks you want the cluster to communicate on and disable cluster communication on all the others. You can do this in the properties of each cluster network. For instance, your iSCSI network doesn't need to be enabled for cluster communication (and in fact, you don't want cluster communication occurring on your iSCSI network).

If you assigned an ip address to each physical NIC (presumably in different subnets and presumably in different VLAN's) then that's why you have multiple cluster networks and that's why the cluster is trying to communicate across all of them.

Are you using a dedicated NIC for iSCSI traffic? Do you have a dedicated NIC for LiveMigration traffic? How about for CSV traffic?

Can you tell us how you have your Hyper-V virtual switches configured? Are you allowing the management operating system to share all of your virtual switches? How many virtual switches have you created? Have you created one for each physical NIC?

Solution 2

Having a single default gateway is best. It then becomes the interface where traffic is sent if it doesn't match the other three interfaces or if there are no explicit routes in the routing table.

If you have traffic for a specific subnet that shouldn't go out the "default gateway" use the route command to update the routing table. Specifically route add.

Specifying multiple default gateways will impact routing. Use the route print command to verify the system's routing table doesn't have any conflicting entries.

Solution 3

Let me see if I can explain this, while all other answers given are valid as well.

so you have 4 NIC's, and therefore you have 4 subnets configured, as the mask of the subnets will determine the size of the network the server can send packets to (directly).

If a situation occurs that the server wants to send a packet to an address which is not directly attached to the configured subnets, it needs a "gateway" to send it to.

Every NIC properties has a field called "Default Gateway" this is a little bit misleading, and comes from a time where a server would usually only have a single NIC.

One or more of the subnets which are configured may have a router (which is a gateway)

So you need to determine what router to use if a packet need to be send to an address outside of the configured subnets.

If you only have one router, on one subnet, it would be simple: on the NIC connecting to that subnet you put in the IP address of the router in the "Default Gateway" box. And you leave this field blank for all the other NIC's

Once you have done that, all packets which are destined outside the 4 subnets will be sent to that router, as this router is now the "Default Gateway" for the host.

More advanced: It will get a bit more complex if you have more than one router, on more than one subnet. If this is your case, then you need to add multiple gateways to the routing table using the ROUTE ADD command. And you do need to put a metric in to determine which one is the "Default Gateway" (which is the one with the lowest metric).

Summary: So there is only one "Default Gateway" for a given host, it is the "Gateway" with the "Lowest Metric" in the "route table".

Easy config: only fill in the gateway in one of the NIC's and leave the other NIC's blank.

Advanced config: add routers as gateways specifying metric with the ROUTE ADD command.

To Check: you can use the ROUTE PRINT command to check the route table to see how this is configured.

HTH, Edwin.

Comments

[Naryna]

The subnets in question are 192.168.1.x, 2.x, 3.x, and 4.x. I was under the impression that all the nics but one should leave the gateway blank, but best practices analyzer doesn't really like this because it tries to get every nic to communicate with every nic on the other cluster node. Obviously any nic without a gateway has no way to route to a different subnet.

[joeqwerty]

This is a great answer as it relates to networking in general but it isn't relevant to the Failover Cluster or the errors in the cluster validation wizard. The issue is exactly as I describe it in my answer. The cluster expects to be able to communicate across all cluster networks across all cluster members. The solution is to disable cluster communication on all of the cluster networks except the networks that you want cluster communication to occur on.

[joeqwerty]

This is a great answer as it relates to networking in general but it isn't relevant to the Failover Cluster or the errors in the cluster validation wizard. The issue is exactly as I describe it in my answer. The cluster expects to be able to communicate across all cluster networks across all cluster members. The solution is to disable cluster communication on all of the cluster networks except the networks that you want cluster communication to occur on.

[Edwin van Mierlo]

@joeqwerty ; I agree with your answer. I read the OP question very carefully and there is no mention of the Cluster Validation wizard. It clearly states the BPA. Hence the generic answer I have given. However if there are actual errors in the cluster validation, I like to know what they are. The Cluster Validation should not (and in my opinion would not) error on a subnet without gateway, so it would be interesting to know if it is BPA or Cluster Validation which comes up with these results (calling them results for now; errors only when there are actually errors).

[Edwin van Mierlo]

@Brandyn Baryski : Could you please clarify if you see these errors as output from the Cluster Validation Wizard or from the Best Practices Analyzer ?

[user2320464]

@joeqwerty, the OP specifically asks for best practices regarding multiple default gateways and it's effect on cluster communication. Proper host routing needs to be established otherwise networking will fail which will impact cluster communications too. Best practice is to define a single default gateway and use the route command to fill in the gaps.