How can I deploy a scalable, reliable haproxy cluster on Amazon EC2?

Question

We need some more advanced functionality than ELB provides (mostly L7 inspection), but it's not obvious how to handle things like heartbeat and high availability with something like haproxy using EC2. There's a high likelihood we'd need 3 or more haproxy nodes in the cluster, so simple heartbeat between two nodes isn't going to work.

Seems like having a heartbeat'd layer in front of the haproxy nodes would be the way to go, possibly using IPVS, but handling the configuration changes as the EC2 cluster changes (either via intentional changes, like expansion, or unintentional, like losing an EC2 node) seems non-trivial.

Preferably the solution would span at least two Availability Zones.

In answer to Qs: No, sessions aren't sticky. And yes, we'll need SSL, but that could in theory be handled by another setup entirely - we're able to direct SSL traffic to a different location than non-SSL traffic.

Solution 1

OK, I've never built an AWS load balancing solution with traffic on the levels of SmugMug myself, but just thinking of theory and AWS's services, a couple of ideas come to mind.

The original question is missing a few things that tend to impact the load balancing design:

1. Sticky sessions or not? It is very preferable to not use sticky session, and just let all load balancers (LB's) use round robin (RR) or random backend selection. RR or random backend selections are simple, scalable, and provide even load distribution in all circumstances.
2. SSL or not? Whether SSL is in use or not, and over which percentage of requests, generally has a impact on the load balancing design. It is often preferable to terminate SSL as early as possible, to simplify certificate handling and keep the SSL CPU load away from web application servers.

I'm answering from the perspective of how to keep the load balancing layer itself highly available. Keeping the application servers HA is just done with the health checks built into your L7 load balancers.

OK, a couple of ideas that should work:

1) "The AWS way":

• First layer, at the very front, use ELB in L4 (TCP/IP) mode.
• Second layer, use EC2 instances with your L7 load balancer of choice (nginx, HAProxy, Apache etc).

Benefits/idea: The L7 load balancers can be fairly simple EC2 AMI's, all cloned from the same AMI and using the same configuration. Thus Amazon's tools can handle all HA needs: ELB monitors the L7 load balancers. If a L7 LB dies or becomes unresponsive, ELB & Cloudwatch together spawn a new instance automatically and bring it into the ELB pool.

2) "The DNS round robin with monitoring way:"

• Use basic DNS round robin to get a coarse-grained load distribution out over a couple of IP addresses. Let's just say you publish 3 IP addresses for your site.
• Each of these 3 IP's is an AWS Elastic IP Address (EIA), bound to a EC2 instance, with a L7 load balancer of your choice.
• If a EC2 L7 LB dies, a compliant user agent (browser) should just use one of the other IPs instead.
• Set up an external monitoring server. Monitor each of the 3 EIPs. If one becomes unresponsive, use AWS's command line tools and some scripting to move the EIP over to another EC2 instance.

Benefits/idea: Compliant user agents should automatically switch over to another IP address if one becomes unresponsive. Thus, in the case of a failure, only 1/3 of your users should be impacted, and most of these shouldn't notice anything since their UA silently fails over to another IP. And your external monitoring box will notice that an EIP is unresponsive, and rectify the situation within a couple of minutes.

3) DNS RR to pairs of HA servers:

Basically this is Don's own suggestion of simple heartbeat between a pair of servers, but simplified for multiple IP addresses.

• Using DNS RR, publish a number of IP addresses for the service. Following the example above, let's just say you publish 3 IPs.
• Each of these IP's goes to a pair of EC2 servers, so 6 EC2 instances in total.
• Each of these pairs uses Heartbeat or another HA solution together with AWS tools to keep 1 IP address live, in a active/passive configuration.
• Each EC2 instance has your L7 load balancer of choice installed.

Benefits/idea: In AWS' completely virtualized environment it's actually not that easy to reason about L4 services and failover modes. By simplifying to one pair of identical servers keeping just 1 IP address alive, it gets simpler to reason about and test.

Conclusion: Again, I haven't actually tried any of this in production. Just from my gut feeling, option one with ELB in L4 mode, and self-managed EC2 instances as L7 LBs seems most aligned with the spirit of the AWS platform, and where Amazon is most likely to invest and expand later on. This would probably be my first choice.

Solution 2

If you're not doing sticky sessions, or if you're using tomcat/apache style (append node ID to sessionid, as opposed to storing state in the LB), then I'd use ELB in front of a group of haproxies. ELB has a healthcheck built in, so you can have it monitor the haproxies and take any down ones out of the pool. Lots less to set up than heartbeat failover.

As far as propagating changes, I don't have a great answer. Puppet is great for initial configuration and implementing changes, but for adding/removing nodes you tend to want faster response than its 30 minute polling interval.

Solution 3

I haven't used it myself but I've seen a lot of people mention using puppet to handle these sort of problems on EC2

Comments

[Iain]

I'm researching how to do canary deploys with a slowly-increasing percentage of traffic going to the new version of the software, and I'm super-curious about where you ended up with this. Did you end up trying any of Jesper's suggestions?

[Tom O'Connor]

Yeah, Puppet on EC2 makes managing a cluster quite straightforward. Just create a micro instance and use that as your puppetmaster.

[Don MacAskill]

We use puppet in our datacenters, but haven't tried on EC2 yet. Is puppet EC2-aware somehow, such that it can find nodes using ec2-describe-instances or something, and automagically configure / reconfigure based on that output? And how would you handle the puppetmaster going away suddenly?

[Tom O'Connor]

Why would it go away suddenly?

[Ben Jencks]

It's not EC2-aware, but you can set it up so new nodes will be marked for signing when you start them, and use an external nodes script to describe them. I wrote some python to do this with SimpleDB (external nodes) and SQS (queue of signing requests for new nodes); an ubuntu dev wrote scripts using S3: ubuntumathiaz.wordpress.com/2010/04/07/…

[Ben Jencks]

If the puppetmaster goes away suddenly, it just doesn't run the manifest, i.e. it leaves the nodes in whatever state they're in.

[Rafiq Maniar]

That's a good solution (and a good question!) You can use Amazon SNS to propagate configuration changes in a push fashion. You need a notification system for adding/removing nodes from the haproxy configuration.

[Don MacAskill]

So I love approach #1, that's the direction I've been leaning, but there are still some interesting gotchas - not the least of which is that ELB doesn't handle an entire AZ failing very well (something we've had happen already). The easy, but yucky, 'solution' there is to have the haproxies behind ELB configured to cross AZs (maybe with a backup cluster in another AZ) so if at least one haproxy is up in each AZ, we should be fine. But that only mimimizes, not eliminates the problem. Any ideas around this problem?

[Frank]

@Don MacAskill: I know AWS has had a couple of large-scale service downtimes, but doing better than AZ reliability on AWS is hard. Moving to multi-AZ operation of the frontend could easily be the first step towards multi-AZ operation of the entire stack, and that's a whole kettle of snakes...

[Frank]

@Don MacAskill: One option would be geo-aware DNS resolution like DynDNS Dynect --> ELB + L7 LBs inside one AZ, with another ELB+L7's on hot standby in another AZ. (Besides being geo-aware, Dynect also has some health checks.) DynDNS has a great track record for uptime, but even so, adding geo-aware DNS is another SPOF. Whether Dynect + load balancing in 2 AZ's has better long-term uptime than just one AWS AZ isn't clear to me. See this for an overview of what I mean, sans the multi-AZ databases: dev.bizo.com/2010/05/improving-global-application.html

[Frank]

@Don MacAskill: Just one last thing -- keep in mind that a single ELB instance can span multiple AZ's. It cannot span across EC2 regions. But if just using ELB to L7 LB's in two AZ's within the same region is acceptable, this would be by far the simplest... You wrote "ELB doesn't handle an entire AZ failing very well", perhaps you already know more than I do.

[Don MacAskill]

Yeah, if an ELB spans multiple AZs and has some sort of failure where it cannot get to any of the backend nodes in an AZ (they're overloaded, down, returning 503s, whatever), end users see those errors - it doesn't re-route to the other AZ(s). I'm hoping that's planned, but it's bitten us once already.

[Raj]

How do you keep HAProxy/Nginx etc from becoming a single point of failure? Is there a way to run a high availability config for HAProxy/Pound/ etc in EC2 (VPC) infrastructure?