Load Balancing DNS Servers: UDP / TCP

Question

I have been asked to rebuild our load balancing infrastructure in the data center.

The original request was to load balance FTP servers. I tried doing that using the current load balancer (Piranha / LVS), but didn't get it up and running. Not just because there's little to none documentation for this software. Since Piranha is considered deprecated, I went over to HAProxy after a couple of days trying, which did the job in a fraction of time spent on Piranha.

So I've got FTP load balancing (passive mode) in place. Now, I was asked to replace the whole Piranha Load Balancer in the data center. In the current Piranha configuration, we have several web servers, IIS servers....aaaand DNS.

No here's the thing:
HAProxy seems to be a commonly used LB, but it is not capable of handling UDP load balancing. This is a bummer, since I like how HAProxy works. So I googled a lot and came across several things. Most people seem to use LVS as a LB for DNS (TCP/UDP). Some use dlbDNS, some use lbnamed, and some use netfilter / iptables.

Since I would want to stick with HAProxy for FTP, HTTP, IIS servers, I got confused on using it side by side with LVS.

Requirements:
2 LB instances with failover
2 DNS servers (already existent) with failover
Multiple backend servers (http, application, etc...)

Questions:
Is this possible? Is UDP load balancing on DNS servers even necessary? Is there any kind of resource that might show me how to get started with that? Or is there a LB solution that is capable to not only handle TCP/HTTP, but also UDP load balancing?

PS: The LB solution should be non-hardware and open source / GPL license / free of costs.

Any help or lead to respective resources is much appreciated!

Solution 1

Don't load balance your DNS.

It's an incredibly light protocol - you'd need an enormous amount of traffic to need more than one box (in which case you'll just be bottlenecking on your load balancer anyway), and there's resilience built in because you can use multiple NS records in your delegation (other servers will be used if one's down).

Solution 2

I'm uncomfortable with this Q&A because it hasn't really been established what type of DNS server you're talking about. There are some significant misconceptions when it comes to the resiliency of recursive DNS and it's important that people cruising in via search engines don't walk away from this discussion with a false sense of security.

• Authoritative DNS: For authoritative DNS servers, the common knowledge regarding the resilience of DNS is pretty spot on. So long as you have multiple authoritative DNS servers that are geo-redundant, you're fine. The main reason for adding high availability for individual IPs is if you're hosting many authoritative zones. This allows you to grow your number of servers without having to change the registrar settings for every domain that is hosted.

• Recursive DNS: Always use some form of high availability solution. (BGP, appliance, etc.) This is where you can get into some serious trouble. All resolver libraries are not created equal: Windows DNS clients will round robin the initial server used between queries, but the majority of Unix-based systems will always cycle through the list sequentially. What is even less known is that these Unix libraries will have to time out on every search domain combination before moving on to the next server. If you have multiple search domains configured and the first server in the resolver lookup order is dead, this can create significant delays in DNS resolution for every single request: more than enough to cause problems within your critical applications.

When it comes to recursive DNS, remember that your server infrastructure is only as resilient as the most braindead client configuration. As your company grows, this is something you never have control over. Do not make any design assumptions based on a homogonous server OS environment, as things rarely stay the same in a growing company. This will definitely bite someone if you don't plan ahead for it.

Solution 3

These days you can use dnsdist by PowerDNS

From the README

dnsdist is a highly DNS-, DoS- and abuse-aware loadbalancer. Its goal in life is to route traffic to the best server, delivering top performance to legitimate users while shunting or blocking abusive traffic.

dnsdist is dynamic, in the sense that its configuration can be changed at runtime, and that its statistics can be queried from a console-like interface.

https://github.com/PowerDNS/pdns/tree/master/pdns/dnsdistdist

They provide repositories for common OSes: https://repo.powerdns.com/

Comments

[user433519]

Check Nginx nginx.com/blog/announcing-udp-load-balancing this seems to address your dns question

[cpt_fink]

Definitely use the multiple NS records, as TCP becomes more common for DNS just let it load-balance itself. Re-inventing the wheel is painful for a reason.

[Icebreaker]

Multiple DNS servers do offer redundancy and prevent total failure, but a downed DNS server would still cause name resolution delays.

[Mosh Pit]

We are using bind / named as a DNS service so far.

[Mosh Pit]

Okay, makes sense for me. The thing is that I would still need a virtual IP for failover purposes if I'm not mistaking, because we will have 2 machines that will contain the HAProxy and LVS for routing. How would I approach this?

[ravi yarlagadda]

@MoshPit vrrpd or keepalived are good options.

[Mosh Pit]

I assume you are talking about an Active - Active Hot Standby setup, correct?

[Mosh Pit]

And which of those would you recommend? I've heard about keepalived before, didn't come across vrrpd yet.

[ravi yarlagadda]

Good point - I was assuming authoritative in my answer but you're right, it may be a recursor.

[ravi yarlagadda]

@MoshPit Depends what you want to achieve; vrrpd just does simple VRRP, while keepalived adds stuff like service health checking to determine whether or not the services being load balanced are alive. Use whichever best suits your needs.

[Andrew B]

@MoshPit Your "DNS service" comment didn't clarify whether this was recursive or authoritative. Authoritative is when you are hosting domains. Recursive is something you would use to get the IP address of domains you don't host. Servers which do both are "mixed", which is against security best practices.

[Mosh Pit]

I apologize for that, noticed that when you posted that comment. We are in control of our own DNS servers, and those are authoritative.

[Mosh Pit]

As mentioned before, I would love to have HAProxy do all the load balancing stuff for HTTP, FTP, etc. and just use keepalived / vrrpd for failover of DNS servers.

[womble]

Don't use a load balancer for high availability.

[Andrew B]

@womble I've edited to generalize the need for HA, but I still can't agree with your statement as worded. Handing out the (non-BGP) IPs of your recursive servers is a terrible idea unless you're 100% certain that you're a Windows shop and will never, ever, ever become a mixed shop. Anything is better than nothing in this use case.

[womble]

What does BGP have to do with deploying a HA service? It's one (heavyweight) way to do HA, but there's far more appropriate solutions available.