How to set up ELB health checks with multiple applications running on each EC2 instance?

Question

At AWS we'd like to to use ELBs to load balance EC2 instances which host multiple applications. Ideally we'd like to have a health check for application.

However, AWS Elastic Load Balancers currently only allow you to ping one location for a health check.

What would be the best way of implementing a health checks with ELB that takes into account the state of multiple applications deployed on each EC2 instance?

Solution 1

Here are two ways to solve this;

First option is to add another health check on the host that validates the health and returns HTTP 200s to the ELB if the logic says that you want to keep the host online. The logic there is, of course, up to you. The disadvantage here would be that if App 2 deployed successfully on some hosts all hosts would still be 'healthy' and receiving traffic.

Another option is to use an additional ELB for each application. You can point several ELBs to the same backend EC2 instances and the cost is pretty minor to do so. That way you can health check per application and drop hosts with issues at a per-application level rather than an all-or-nothing approach.

Edit: Please note this is an older answer and is specific to ELB not ALB. ALB supports separate targets on one host natively.

Solution 2

Using one ELB per app is the way to go here.

First, you may need them anyway if each application is on it's own domain and you need to support SSL. Amazon ELBs currently only allow one SSL certificate for each domain, requiring separate ELBs for each SSL-enabled domain. (Wildcard SSL certifications being an exception).

The challenge here is that ELB health checks cannot currently be directed to a particular virtual domain hosted on an EC2 instance. (No "Host:" header is sent). ELB health pings always go to the default domain, as if the you had loaded the IP address for the EC2 instance in your browser. So some glue is required to recieve the health checks on the default domain and then reply with health status of a particular application.

Here is a working example configuration that could be added to an Nginx server directive. It would be installed on each of the EC2 instances being load balanced.

    # This goes in the `server` block noted by 'default_server', often /etc/nginx/sites-enabled/default

    # All AWS Health Checks from the ELBs arrive at the default server.
    # Forward these requests on the appropriate configuration on this host.
    location /health-check/ {
      rewrite ^/health-check/(?<domain>[a-zA-Z0-9\.]+) /api/v1/status break;
      # Lie about incoming protocol, to avoid the backend issuing a 301 redirect from insecure->secure,
      #  which would not be considered successful.
      proxy_set_header X-Forwarded-Proto 'https';
      proxy_set_header "Host" $domain;
      proxy_pass http://127.0.0.1;
    }

In the In the "Health Check" setting of of the ELB for "first-application.com", you would select "HTTP" and Port 80 and enter a path like:

/health-check/first-application.com

With the above Nginx configuration running on the host, the request would be received on the default domain and proxy the response from the Nginx configuration on the same host for https://first-application.com/api/v1/status

With this approach there is no per-app configuration in Nginx. As long as each app has a unique domain name, you just need to make sure you set up an ELB for each app appropriately.

Solution 3

On 11 August 2016, Amazon introduced Application Load Balancers. These let you specify multiple Target Groups, each with its own type of health check. So this is now possible using a single load balancer!

Comments

[Nathan C]

One way would be to implement your own health check by making a script return a status based on its checks of both applications.

[Guy]

ELB is a managed service. Create another ELB for the second application with its own health check. Most of the cost is per request, therefore it will cost you almost the same to operate 2 ELBs.

[Tom Harrison Jr]

I think @NathanC's answer is the best solution; I have similar case where if either of two conditions fails, the health check should fail. Adding another ELB will allow you to have another health check, but AFAIK only one ELB can be used to route traffic (or not)

[Josip Rodin]

@Guy actually, given that the ELB-hours are charged regardless of requests, that's ~20 USD a month (depending on region), so most of the cost is per request only if you already serve well over 2.5 TB of data per month (at $0.008/GB)

[tourdownunder]

thank you. This seemed to do the trick though. I was hoping to avoid needing to have multiple load balances with this.