ZooKeeper alternatives? (cluster coordination service)

Question

ZooKeeper is a highly available coordination service for data centers. It originated in the Hadoop project. One can implement locking, fail over, leader election, group membership and other coordination issues on top of it. Are there any alternatives to ZooKeeper? (free software of course)

Solution 1

I've looked extensively at Zookeeper/Curator, Eureka, etcd, and consul. Zookeeper/Curator and Eureka are in many ways the most polished and easiest to integrate if you are in the Java world. Etcd is pretty cool and very flexible, but It is really just a HA key store so you would have to write a lot of code to turn it into an opinionated service discovery system.

Consul is (to me) the best of both worlds. It is an opinionated service discovery system written on top of serf, using raft for cluster consensus and gossip for communication. It exposes the discovery / registration endpoints with a well documented REST api, and also allows you to discover services with DNS SRV records, and register services with configuration (i.e. so you can register a database or application you can't integrate a client with, or if you just want to keep your service discovery decoupled from your app)

I've written a blog post about consul where you can learn more and walk through my "try it out" demo

I've also discussed service discovery with etcd & docker if you want to see more about what that custom code might look like.

One last thing! etcd & consul are written in go, so maintaining them is much easier then java solutions like zookeeper. All you need is the consul / etcd binary. no dependencies, no linked libraries, no jvm.

Solution 2

There's a very promising alternative to ZooKeeper called etcd (github.com/coreos/etcd), written by CoreOS team. Unlike Doozerd, etcd is being actively developed.

Solution 3

Just discovered Accord (C) and OpenReplica/ConCoord (Python) which may be interesting solutions

[EDIT] The Hashicorp crew, of Vagrant and Packer fame, are cooking "a decentralized solution for service discovery and orchestration" called Serf.

[EDIT2] Hashicorp strikes again ! They just released Consul, built on top of Serf. The pitch: "a solution for service discovery and configuration, completely distributed, highly available, scalable to thousands of nodes and services across multiple datacenters".

Solution 4

Yes, there is also Doozerd (https://github.com/ha/doozerd). Take a good look at it, it's a nice, single binary distributed coordination service developed by Heroku. With bindings/libraries for java/python/ruby/node. Very easy to get started with and play around.

Solution 5

Take a look at Serf. There is a comparison vs Zookeeper here.

View more solutions

Comments

[Admin]

Doozerd is nice but it's unmaintained and doesn't have a production track-record.

[Admin]

1) ZooKeeper is object-oriented, see zookeeper.apache.org/doc/trunk/zookeeperProgrammers.html 2) ? ZK is basically a femto filesystem for storing infrastructure metadata, which also supports presence detection (ephemeral nodes) and observers.

[Admin]

Accord is an up-and-coming project for write-intensive loads. It uses CoroSync, which is also used by Qpid and can be used with Linux HA.

[Admin]

Noah might be useful for applications that do not require high availability or where job security is optional.

[wizzard0]

I've pulled the git repo and it seems the last commit is from May. Hope you still hanging around :)

[user1404662]

Hi Barry: ZooKeeper is not object-oriented. The main abstractions (znodes, paths, watches) are all there to support the maintenance of a small object and the API is that of a filesystem. If you take a closer look at OpenReplica, you can see that there is a big difference between the filesystem API of ZK and OO API of OpenReplica. wizzard0: We're alive and kicking! New release slated for early April.

[Trade-Ideas Philip]

Accord appears to be dead: "This site can’t be reached / www.osrg.net’s server IP address could not be found."