Distributed Development – Looking for a Distributed Locking Pattern

cconcurrencydistributed-developmentmultithreading

I need to come up with a custom recursive object locking mechanism\pattern for a distributed system in C#. Essentially, I have a multi-node system. Each node has exclusive write permissions over n-number pieces of state. The same state is also available in read-only form on at least one other node. Some writes/updates must be atomic across all nodes, while other updates will eventually become consistent thru background replication processes, queues, etc…

For the atomic updates I'm looking for a pattern or samples that efficiently allow me to mark an object as locked for writes which I can then distribute, commit, rollback, etc… Since the system has high-levels of concurrency, I'm assuming I'll need to be able to stack-up locks which will either timeout or be unrolled once the locks are released.

The transaction or messaging pieces are not the focus of this question, but I've provided them for some extra context. With that said, feel free to articulate what messages you think would be needed if you like.

Here is a vague sample of what I was envisioning although I'm open to any new ideas aside from implementing whole new products

thing.AquireLock(LockLevel.Write);

//Do work

thing.ReleaseLock();

I was thinking of using extension methods, which might look something like this

public static void AquireLock(this IThing instance, TupleLockLevel lockLevel)
{ 
    //TODO: Add aquisition wait, retry, recursion count, timeout support, etc...  
    //TODO: Disallow read lock requests if the 'thing' is already write locked
    //TODO: Throw exception when aquisition fails
    instance.Lock = lockLevel;
}

public static void ReleaseLock(this IThing instance)
{
    instance.Lock = TupleLockLevel.None;
}

To clarify a couple details…

  • All communications are TCP/IP using a binary request/response protocol
  • There are no intermediary technologies such as queues or databases
  • There is no central master node. In this case, the locking arrangement is defined by the initiator of the lock and the partner which will honour the request with some form of timeout to govern its behaviour

Anyone have any suggestions?

Best Answer

Thanks for the clarifications.

In that case, what I'd recommend is using a publish / subscribe model. Google's Chubby distributed locking protocol ( an implementation of Paxos)

I've never used Paxos (or Chubby), but there seems to be an open source implementation here.

If that doesn't work, you could implement your own version of Paxos using, for example, one of the usual suspects in terms of messaging libraries : the zero message queue library, RabbitMQ, or ActiveMQ.

Previous Answer:

Most of the suggestions on SO ([A], [B]) go for use of a message queue to achieve cross-machine locking.

Your AcquireLock method would push something identifying the lock object into the queue, checking for previous instances of locks before success. Your ReleaseLock method would remove the lock object from the queue.

SO user atlantis suggests, in this post, Jeff Key's post for some of the detail.

Related Topic