Dynamic subdomain routing

Question

I asked this question over at stackoverflow, but got very few views: https://stackoverflow.com/questions/2284917/route-web-requests-to-different-servers-based-on-subdomain

Perhaps it's more applicable to this crowd. Here it is again for convenience:

I have a platform where a user can create a new website using a subdomain. There will be thousands of these, eg abc.mydomain.com, def.mydomain.com . Hopefully if we are successful hundreds of thousands.

I need to be able to route these domains to a different IPs to point at a particular app server. I have this mapping in a database right now.

What are the best practices and recommended technologies here?

I see a couple options:

1. Have DNS setup with a wildcard CNAME entry so that all requests go to a single IP where perhaps two machines using heartbeat (for failover) know how to look up the IP in the database and then do an http redirect to the appropriate app server. This seems clunky and slow to me.
2. Run my own DNS server that can be programatically managed such that when a new site is created a DNS entry is added. We also move sites around to different app servers, so I would need to be able to update DNS entries in close to real time. Thoughts anyone?

Thanks.

Update2: I've setup external wildcard DNS pointing at an HAProxy web server whose job it is to route requests to backend servers. The mapping is stored in our internal PowerDNS server. Question now is how to get the HAProxy server (or another) to use the value of the internal DNS and not some config file or access list? –

Update: Based on some suggestions below, it seems like reverse-proxy server(s) is the way to go. As I'll be rebalancing the domain->server mapping, these need to work instantly and the TTL on a DNS solution could be a problem. Any recommendations on software to use considering this domain->IP data is stored in a DB, and I'll need this to be performant?

Solution

Given your requirement for real-time failover between servers I couldn't recommend a DNS-based solution - too many clients will simple not spot the changes.

Are all these servers going to be on one site? A resilient front-end "reverse proxy" with access to the customer -> sever mapping table would seem to be easiest.

Comments

[Nader]

Thanks mat. I've heard of PowerDNS, but haven't checked it out in detail yet. I am a little hesitant to run my own DNS, because I would need to set it up for failover, worry about performance, etc. (We're running all our hardware on EC2). I believe PowerDNS uses a MySQL backend, so I could also just write to that right?

[Nader]

How would I dynamically assign new subdomains/reassign existing subdomains to different servers with this setup?

[Nader]

Thank for the suggestion Alnitak. All the application servers are in the EC2 cloud. I'm not very familiar with reverse proxies. I've used Apache, but that's about it. My app servers are running drupal, and folks have recommended Squid and Varnish as reverse proxy servers. What would you recommend? And how would those RP servers access the customer->server mapping table? This data doesn't change very often, so I'd probably want to cache it for performance

[jeffatrackaid]

You would control that via your internal DNS or re-writing your host files. I assume you are not changing these on the fly frequently?

[Nader]

Jeff, I will be adding entries frequently, and periodically changing entries (moving sites to different servers to rebalance). So two questions. 1) What's the benefit of using internal DNS? Seems like I would have the same problems as I described above as I would with external DNS. 2) Regardless of whether I use internal or external DNS how would I dynamically update it in real time? Seems like I would still need something like PowerDNS or some DNS that has an API. Proxy servers have other benefits like caching, etc. but that's not the problem I'm trying to solve right now.

[jeffatrackaid]

By frequently do you mean every minute, every hour? Using the proxy allows you to use a single wildcard DNS record on the public side, which I think is a must if you want to load balance. I suspect you will be mapping to your backend servers over a private network. So you will need to to map the domain to the backend server's IP. You can do this by running your own DNS or even via the /etc/hosts file. Also just throwing out some ideas but an httpd server with configs stored in a database could be useful. redmine.lighttpd.net/projects/lighttpd/wiki/Docs:ModMySQLVho‌​st

[Nader]

Hi Jeff. So I'm going down this route. I've setup external wildcard DNS to point at my reverse proxy server and have PowerDNS setup with a db backend with the mappings. I'm currently trying HAProxy as the proxy server. Do you know a way to get it to use the DNS server for the IP to route to? Seems like most proxies rely on an access list. And I don't really want to have to do something like run cron to update the access list every x seconds. I checked out lighttpd, but it looks like it just allows for dynamic docroots, not IPs

[raja]

I don't think you have much choice here execept to either run your own internal DNS with a public wildcard and proxy or a customized copy of power DNS- neither of which I'd want to run on EC2