Architecture – Designing a social network with CQRS, graph databases and relational databases in mind

architectural-patternsArchitecturecqrsgraph-databasessocial-networks

I have done quite an amount of research on the topic so far, but i couldn't come up with a conclusion to make up my mind.

I am designing a social network and during my research i stumbled upon graph databases, i found neo4j pretty interesting for user relations and traversing through nodes. I also thought of using a relational database such as MS-SQL or MySQL to store entity data only and depending on neo4j for connections between entities. Of course this means more work in my application to store and pull data in and out of 2 different sources.

My first question : Is using this approach (graph + relational) a good approach for designing my social network keeping in mind that users on social networks don't have to in synch with real data by split second ? What are the positives and negatives of this approach ?

My Second question : I've been doing some reading on CQRS and as i understood it is mostly useful for collaborative environments, and environments where users see a lot of "stale" data. social networks has shared comments, events, etc .. and many users query or update the same data. Could CQRS be a helpful approach ? Would it give any performance/scalability benefits or non-useful complexity ? Is it fairly applicable with my possible choice of (graph + relational) databases approach mentioned in the question above ?

My purpose is to know if the approaches i have mentioned above seem good enough for the business context.

Best Answer

In my opinion, you are over-engineering the project. I think you do this because you believe you have to rely on cutting-edge techniques to handle business scale, but in many cases you will be better off relying on proven techniques and innovating only in a very focused scope.

A word of caution on graph databases: in my experience, they promise more than they can deliver. My experience is now some years ago, so I can't tell you if they have matured as products; but do you want to find out if something scales by using it as your main workhorse?

Let me remark there are some alternatives for such graph algorithms, some of them with provable scalability because they build on Hadoop's HDFS: see this SO thread or this Spark library.

On the topic of CQRS, it seems to deal with the kind of problems that large websites traditionally handle with a cache layer on top of their database replica sets. Write a wrapper around your queries that first looks into the cache layer, and if that misses its mark, then pulls the data from the database and also writes the result set into the cache. Here is a simple example in Python.

Moreover, splitting your queries into Commands and Queries on top of two database engines means that you have to decide, for each user request, whether it is graph-related or not, and whether it writes or just reads something; usually you will have a mix of all four possibilities. If you get your decisions right, you will get a faster, more responsive social network; but be aware that you will get the same performance boost by making the right decisions in almost any language and runtime. And even so, response times will almost certainly be dominated by network latency.

In your place, I would concentrate on one of both topics, and I would also concentrate much more on the question: what does this technique enable that is better than existing social networking sites?

Related Solutions

Database Design – Architecture for Social Graph Data with Time Frame

You should check out Neo4j-open source graph database. It is a mature and well supported project. It has a well written-updated documentation. It is java based but has several client libraries for ruby, jruby, php, python, c#... It works as embedded(a jar file) or as a server(you may connect and operate over HTTP with its REST based structure). And finally, it is disk based transactional.

In Neo4j, you could have properties assigned to nodes and also to relations. That means, in your case, you may connect two nodes with a relation that has properties called "StartDate" and "EndDate" or any other property you wish.

For example if you have users as nodes you may have a relationship called "interested_in" and you may connect UserA to UserB with the relation "interested_in" and assign StartDate as the date you created relation and later on you may assign an EndDate property as the date "interested_in" relation come to an end.

It may look like:

UserA -[interested]-> UserB
      StartDate:20121102
       EndDate:20121107

And users(I mean your nodes) could be connected to your existing database via giving "id" property to Neo4j nodes coming from your existing database. Or you may copy all or several properties(i.e name, surname, birth date, etc...) to the nodes in Neo4j, but this time you may need to synchronize your users in Neo4j and your database every time an update occurs in your data stores.

There are several example data models within the documentation of Neo4j

There is also InfiniteGraph which is similar to Neo4j and stated as "Distributed Graph Database", but I do not have any experience with it. For your case(50,000 - 70,000 for each type of node) Neo4j would be a perfect match with its support up to billions of nodes and relationships.

Database – Why would I use ElasticSearch if I already use a graph database

I hesitate to call ElasticSearch a database. It is not a replacement for a database, but it makes a good addition to add functionality, specifically advanced text searching, along side your existing database.

I see where you can get them confused. They can actually fit the same need, but not always. ElasticSearch does exactly what it sounds like, searches. A graph database doesn't specify relations or indexes, where as ElasticSearch does. So fundamentally they work quite differently. ElasticSearch analyzes documents with, for example, English analyzer. What this does it will take words and analyze different variations of that word or even synonyms. For example, dig, would be anaylzed as dig,digs,dug,digging,digger .... When you run a query on elasticsearch your queries can also be analyzed, then those words are queried for and can be scored by relevance.

ElasticSearch is a great tool, because it's really flexible. You can find a wide range of relative content, or you can find a needle in the hay stack, and its relatively easy.

Graph Databases have their advantage too. Finding relevance/relations between things like hash tags for example, or things with many mutable relations. They're great and interesting pieces of technology, however I'd have to say that its not as powerful as ElasticSearch. Mostly because ElasticSearch is geared towards this sort of thing, and it handles analysis for you so you can do full-text search. However if you're looking to use a system more so like twitter's search that's based on predefined tagging/keywords, then you'd be better off using the Graph Database your already using.

The question is how robust do you want your searching to be? If you have a need to do really fine grain(full text) searches I'd use elasticsearch. Otherwise you can always implement a search relatively easily on a graph database. Once you have search implemented its not impossible to migrate to elasticsearch if you find yourself later needing a more robust search engine, just implement your search with that in mind.

Best Answer

Related Solutions

Database Design – Architecture for Social Graph Data with Time Frame

Database – Why would I use ElasticSearch if I already use a graph database

Related Topic