How to automatically update nginx upstream server list when aws ec2 hostname changes or increases?

Question

I want to setup autoscaling in AWS. I don't want to use Elastic Load Balancer.

Autoscalling in Amazon creates EC2 instances seamlessly during demand spikes to maintain performance, and decreases automatically during demand lulls to minimize costs.

Since this EC2 instances are created automatically, their host names are unknown to NGINX.

I know and already have upstream setup in nginx to 10 EC2 instances.

I want to be able to add/update/delete automatically server names to my upstream nginx configuration, when autoscaling adds/updates/deletes EC2 instances.

Solution 1

This can be achieved by using Amazon SDK ( I am almost done with it, will put it on github ), utilizing the SNS, EC2 and Autoscaling service.

I have followed the below steps to achieve this:

1. Enable HTTP notification and subscribed my webserver.
2. Added a lifecycle-hook with heartbeat of 1 min (to wait for 1 min before terminating) to my autoscaling group for terminating server
3. Created an index file to parse the message to detect what kind of message it is ( i.e Launch or Terminate)
4. Once type of event is decided i queried EC2 to get the private ip of the instance
5. In case of Launch wait till header 200 is recieved and then add the ip to nginx config and reload
6. In case of Terminate remove the IP from config and reload nginx

Please find the script here https://github.com/singhupendra/aws-autoscale

Solution 2

Thank you @talonx, I've done some research, Amazon Autoscale has an api to query current autoscaling group status, and enumerates its members. It returns instance id (http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/api_requests.html#query-example), then you can use the describe tools to get the server name (http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-DescribeInstances.html) and finally recreate the upstream include file. I could sense the Autoscaling notifications to launch a process that performs these tasks.

I still didn't implement it but its a way to go.

One can also use Autocaling with SNS http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/ASGettingNotifications.html

Solution 3

I haven't implemented this yet myself, but I'm looking into using the On-the-fly Reconfiguration of NGiNX Plus. I'm thinking that either the AMI, or the configuration management (Puppet, Salt, or such) that sets up an Auto Scaling Group instance, could reach the NGiNX reconfiguraiton API (perhaps, via an internal Route53 domain name so no fixed IP would need to be used), and add itself to the upstream cluster for the reverse proxy. After that NGiNX's built-in health check would then take over for that [added] instance, and drop it in case it becomes unavailable. This seems the cleanest solution and there is no delay in adding the instance, and hardly any delay in dropping it since NGiNX Plus features out-of-band health check.

This approach avoids needing to set up an auto-discovery system (Consul, Serf, or such) which for smaller setups often seems like much overhead both in the terms of setup/administration as well as required EC2 instances. Consul, for example, requires minimum of three instances to be stable. Serf could perhaps run on the ASG instances themselves, but there's still the overhead of maintaining it, and if the ASG scales down to one or two instances, you'd lose the quorum.

Finally, this could be combined with automatic notification of Auto Scaling Group changes, perhaps on the NGiNX server(s) that is/are used for load balancing. A listener triggered by such notification (this may be what Upendra also referred to) could then instantly add the new instance to NGiNX via the On-the-fly modification API. Besides the cost of NGiNX Plus, this makes one wonder why would anyone use Elastic Load Balancer with its numerous issues in the first place.

Edit 2015-12-07: ngx_openresty's balancer-by-lua (see this GitHub thread) offers a another possible open source solution for hot-adding/removing servers from NGiNX upstream group. I have not yet experimented with this myself, but wanted to add a mention here for anyone stumbling across this post.

Comments

[Gagan Pal]

You need to remove "autoscaling" from your question. Autoscaling is an AWS term. I think what you mean is you want to automatically scale (horizontally), by adding more upstream nodes to your nginx acting as an LB, and you are asking how to automatically modify your nginx configuration when upstream nodes are added/deleted/modified. If this is so, please edit your question accordingly.

[Luis Lobo Borobia]

well, actually, I know what autoscalling is, and I ment to say that. I want to mix both. I'll update the question.

[Gagan Pal]

The question is clearer now, in its intent. I wanted to vote to re-open, but I don't see an option - guess I don't have enough rep yet.

[Luis Lobo Borobia]

Thank you @talonx I hope others can upvote to find my answer

[Gagan Pal]

I think you can combine AWS autoscaling notifications (delivered using SNS) - assuming it returns the host name of the newly created/terminated instance - and one of the third party nginx APIs to update and reload your nginx configuration. Sorry for being vague - I'm not very familiar with the autoscaling API.

[MastaJeet]

This is basically what I did. I wrote a ruby script that runs every N minutes. Using the AWS SDK it queries for the members of the the ASG and using an ERB template it generates a new config. If the new config is different than the current config it copies it into place and tells the daemon (haproxy in my case) to reload its config. Note that instances remain in the ASG for a bit after they are terminated so make sure the instance.status == :running. Also note that if it takes N minutes after the instance is launched for it to serve requests don't use it unil now > instance.launch_time + N.

[Luis Lobo Borobia]

Thank you @MarkWagner. Is there any possibility you can share that script somewhere? Gist, github? Thanks!

[Aaron]

Any chance you posted this on github? I'm trying to do the same thing and any assistance would be appreciated.

[Aaron]

Did you have any luck with this script? Is there an example on github or elsewhere?

[Luis Lobo Borobia]

No but right now nginx-plus (the paid version) allows this a more.

[Upendra]

please use - github.com/singhupendra/aws-autoscale