Architecture – Choice of Technology to Develop Multi-Platform Applications

Since you are moving away from a single master node (which is appropriate) you will have to change a few things. You will need to setup a Quorum. Since you already have 9 nodes, you are in good shape. For a Quorum to work you need 2n+1 nodes where (n) is the number of nodes that can go down and the system will still work. Within the Quorum a vote will take place on who the leader is, and what transactions are successful. This can be used to pass around configuration information and ensure everyone is synchronized without a database.

There are existing technologies out there that can help you with this. One of thos is ZooKeeper. It is an open source Apache v2 product for Distributed Coordination. You will need something along these lines. Whether it is using ZooKeeper or rolling your own their white papers will be invaluable. It can also be used to maintain your configuration information about each node.

ZooKeeper is written in Java but I have created a project (ZooKeeperNet that will allow it to be embeded within .NET application using IKVM. If this isn't acceptable then you'll want to read about Leader Elections when determining who will be the current Master node. I suggest reading all their Wiki pages and Recipes to get an idea of what you need to account for in a proper distributed system.

Just so you have a good understanding. ZooKeeper is the backing coordination system of Hadoop and HBase. Hadoop is a distributed Map/Reduce framework.

If you already aren't, you can use WCF adhoc or registry discovery information when attempting to find the current master node in your system. If only a single Master node is alive it will be the only registered to support IMaster features. Then your slave nodes will listen on each other's znodes for each other to go away, picking up being the Master almost immediately.

Keep in mind that in order to be high efficient, the data each node needs to work with has to be close (i.e. on the node itself) to the node. If one node acts as a data intermediary you won't be as efficient as you could if the nodes could pull data in a distributed fashion.

I'd go with Option 3, with the following notes:

1. Try and reduce the amount of domain logic that your clients need to know to get the job done. Create services that expose that data in meaningful (to your clients) ways, so that you can request collections of domain objects which fulfil certain criteria, rather than doing that crunching on your client.
2. Validation should be considered optional on the client-side as, you can never guarantee that future client implementations are going to be done properly. Therefore, always validate on the server side as if it hadn't been done elsewhere. Of course, clients should be validating on the client-side too.
3. Rather than mapping the WCF data back to full domain objects on the client side, consider mapping them to simpler ViewModel-type objects - a slimmed down version of your full domain objects only containing properties appropriate to the client - makes client programming simpler.

The problem you're still faced with is lots of mapping. I guess this price is worth paying (and made easier with a tool such as AutoMapper) because removing the client dependency on your domain model gives you breathing room to change your domain, tweak the mapping, without breaking any client code.