Distributed Concurrency Control

Question

I've been working on this for a few days now, and I've found several solutions but none of them incredibly simple or lightweight. The problem is basically this: We have a cluster of 10 machines, each of which is running the same software on a multithreaded ESB platform. I can deal with concurrency issues between threads on the same machine fairly easily, but what about concurrency on the same data on different machines?

Essentially the software receives requests to feed a customer's data from one business to another via web services. However, the customer may or may not exist yet on the other system. If it does not, we create it via a web service method. So it requires a sort of test-and-set, but I need a semaphore of some sort to lock out the other machines from causing race conditions. I've had situations before where a remote customer was created twice for a single local customer, which isn't really desirable.

Solutions I've toyed with conceptually are:

1. Using our fault-tolerant shared file system to create "lock" files which will be checked for by each machine depending on the customer

2. Using a special table in our database, and locking the whole table in order to do a "test-and-set" for a lock record.

3. Using Terracotta, an open source server software which assists in scaling, but uses a hub-and-spoke model.

4. Using EHCache for synchronous replication of my in-memory "locks."

I can't imagine that I'm the only person who's ever had this kind of problem. How did you solve it? Did you cook something up in-house or do you have a favorite 3rd-party product?

Solution 1

you might want to consider using Hazelcast distributed locks. Super lite and easy.

java.util.concurrent.locks.Lock lock = Hazelcast.getLock ("mymonitor");
lock.lock ();
try {
// do your stuff
}finally {
   lock.unlock();
}

Hazelcast - Distributed Queue, Map, Set, List, Lock

Solution 2

We use Terracotta, so I would like to vote for that.

I've been following Hazelcast and it looks like another promising technology, but can't vote for it since I've not used it, and knowing that it uses a P2P based system at its heard, I really would not trust it for large scaling needs.

But I have also heard of Zookeeper, which came out of Yahoo, and is moving under the Hadoop umbrella. If you're adventurous trying out some new technology this really has lots of promise since it's very lean and mean, focusing on just coordination. I like the vision and promise, though it might be too green still.

• http://www.terracotta.org
• http://wiki.apache.org/hadoop/ZooKeeper
• http://www.hazelcast.com

Solution 3

Terracotta is closer to a "tiered" model - all client applications talk to a Terracotta Server Array (and more importantly for scale they don't talk to one another). The Terracotta Server Array is capable of being clustered for both scale and availability (mirrored, for availability, and striped, for scale).

In any case as you probably know Terracotta gives you the ability to express concurrency across the cluster the same way you do in a single JVM by using POJO synchronized/wait/notify or by using any of the java.util.concurrent primitives such as ReentrantReadWriteLock, CyclicBarrier, AtomicLong, FutureTask and so on.

There are a lot of simple recipes demonstrating the use of these primitives in the Terracotta Cookbook.

As an example, I will post the ReentrantReadWriteLock example (note there is no "Terracotta" version of the lock - you just use normal Java ReentrantReadWriteLock)

import java.util.concurrent.locks.*;

public class Main
{
    public static final Main instance = new Main();
    private int counter = 0;
    private ReentrantReadWriteLock rwl = new ReentrantReadWriteLock(true);

    public void read()
    {
        while (true) {
            rwl.readLock().lock();
                try {
                System.out.println("Counter is " + counter);
            } finally {
                rwl.readLock().unlock();
            }
            try { Thread.currentThread().sleep(1000); } catch (InterruptedException ie) {  }
        }
    }

    public void write()
    {
        while (true) {
            rwl.writeLock().lock();
            try {
               counter++;
               System.out.println("Incrementing counter.  Counter is " + counter);
            } finally {
                 rwl.writeLock().unlock();
            }
            try { Thread.currentThread().sleep(3000); } catch (InterruptedException ie) {  }
        }
    }

    public static void main(String[] args)
    {
        if (args.length > 0)  {
            // args --> Writer
            instance.write();
        } else {
            // no args --> Reader
            instance.read();
        }
    }
}

Solution 4

I recommend to use Redisson. It implements over 30 distributed data structures and services including java.util.Lock. Usage example:

Config config = new Config();
config.addAddress("some.server.com:8291");
Redisson redisson = Redisson.create(config);

Lock lock = redisson.getLock("anyLock");
lock.lock();
try {
    ...
} finally {
   lock.unlock();
}

redisson.shutdown();

Solution 5

I was going to advice on using memcached as a very fast, distributed RAM storage for keeping logs; but it seems that EHCache is a similar project but more java-centric.

Either one is the way to go, as long as you're sure to use atomic updates (memcached supports them, don't know about EHCache). It's by far the most scalable solution.

As a related datapoint, Google uses 'Chubby', a fast, RAM-based distributed lock storage as the root of several systems, among them BigTable.

View more solutions

Comments

[John Channing]

Rather than test-and-set, could the responsibility for ensuring that there are not any duplicate be moved into the service that creates new customers?

[Bob Gettys]

I apologies, I know my "question" wasn't the clearest in the world - , the ESB is interacting via services, so we have no control over constraints. I have to ask via services if their Customer exists, and then make a separate request to create one if it does not.

[Bob Gettys]

That is EXACTLY the kind of semantics I was attempting to use with my (failed) home-grown solution. I'm curious why I wasn't able to find it on my own, but thanks for the tip!

[newacct]

You should use Thread.sleep(...) instead of Thread.currentThread().sleep(...) because it's a static method; your code might tempt you to do someOtherThread.sleep(...) which does not sleep someOtherThread.

[Mathieu]

Your client does not have permission to get URL /p/vitrit/ from this server

[Dunaril]

I fail to see why the fact that Hazelcase uses P2P would make it untrustable for large scale use.

[npgall]

+1 for mentioning a concern about the scalability of P2P locking. @Dunaril Think about how many nodes need to be notified about a lock being issued. In the naive case it's an N squared problem (cluster_size^2): every node needs to be notified about locks issued on every other node. In practice sharding or quorums can mitigate in these problems, but still it's a hard problem and a library advertising "plug and play" usability is unlikely to solve it! P2P locking has inverted scalability: performance degrades as additional nodes are added.

[Dungeon Hunter]

seems like the API has changed now. Check it out hazelcast.org/docs/3.2/javadoc/com/hazelcast/core/class-use/‌​…

[Marko Bonaci]

I wouldn't use Redis as a locking service, no way: aphyr.com/posts/283-call-me-maybe-redis

[Nikita Koksharov]

@mbonaci Lock object has concurrent tests and passes it each time. Here is an antirez answer to this antirez.com/news/55 UPDATE from antirez: in the meantime the Redis Sentinel architecture and implementation changed significantly, moreover a feature of Redis to stop accepting writes if there are not enough connected slaves was added, so basically it is possible to reach a better kind of consistency guarantee using Sentinel.

[Marko Bonaci]

I know, and here's aphyr's revisiting redis and those (and more recent) antirez's comments/whimps: aphyr.com/posts/307-call-me-maybe-redis-redux. Redis, when paired with Sentinel, is only good for caching. Session storage at best (in non-critical environments). Until its model is peer reviewed, that is.