Apache Mesos – Persistent Storage Solutions

Since you are moving away from a single master node (which is appropriate) you will have to change a few things. You will need to setup a Quorum. Since you already have 9 nodes, you are in good shape. For a Quorum to work you need 2n+1 nodes where (n) is the number of nodes that can go down and the system will still work. Within the Quorum a vote will take place on who the leader is, and what transactions are successful. This can be used to pass around configuration information and ensure everyone is synchronized without a database.

There are existing technologies out there that can help you with this. One of thos is ZooKeeper. It is an open source Apache v2 product for Distributed Coordination. You will need something along these lines. Whether it is using ZooKeeper or rolling your own their white papers will be invaluable. It can also be used to maintain your configuration information about each node.

ZooKeeper is written in Java but I have created a project (ZooKeeperNet that will allow it to be embeded within .NET application using IKVM. If this isn't acceptable then you'll want to read about Leader Elections when determining who will be the current Master node. I suggest reading all their Wiki pages and Recipes to get an idea of what you need to account for in a proper distributed system.

Just so you have a good understanding. ZooKeeper is the backing coordination system of Hadoop and HBase. Hadoop is a distributed Map/Reduce framework.

If you already aren't, you can use WCF adhoc or registry discovery information when attempting to find the current master node in your system. If only a single Master node is alive it will be the only registered to support IMaster features. Then your slave nodes will listen on each other's znodes for each other to go away, picking up being the Master almost immediately.

Keep in mind that in order to be high efficient, the data each node needs to work with has to be close (i.e. on the node itself) to the node. If one node acts as a data intermediary you won't be as efficient as you could if the nodes could pull data in a distributed fashion.

It seems like your "storage system" whatever that is has a very simple abstraction interface. It essentially boils down to "here's an id, gimme data".

So you can easily define this interface and build your entire app on top of it. Behind the scenes you can continue using PostgreSQL like you do today. And if you want to experiment with flat file storage, it should take you more than 1 or 2 days to implement something very simple that reads/writes files straight on disk (my recommendation is to have 1-3 tiers of directories based on first portion of the ID, so you don't have too many files in one flat directory).

If you do that you can compare the performance straight up and see if it's good enough for you.

However, as Euphoric pointed out, most NOSQL stores where introduced and became popular for the very purpose that you are trying to accomplish. I'm not going to recommend a specific store as that's something for you to decide but some advantages they provide are:

1. managing storage of huge amounts of small entities by buffering and doing writes in larger chunks. In my experience most file systems can work with very large numbers of files, but not in the most efficient way. For example, if you were to try to delete that many files from disk, unless you just reformat the whole disk, it could take multiple hours just for "rm -rf *" command.
2. If and when you outgrow the limitations of a single physical box, a lot of NoSQL solutions allow you to scale horizontally which will give you a) more storage, b) redundancy of data so if one host goes down, your storage server is still online and c) faster query times since your clients can load balance where they get the info.

Another option to consider is that potentially your storage and your indexing does not have to be implemented in the same system. You could use a separate indexing product like Solr or Elasticsearch and store the actual data in a NoSQL DB (or a straight-up file system)