We are developing two related systems. One of them (A) will be installed on our customers' machines. The remaining (B) will be used by my organization.
Each system has its own database (relational) and their schemas differ. However, both systems have to be synchronized. In addition, some changes in B has to be exported to all class A systems and other only to a specific one.
Some customers don't have an Internet connection so the synchronization, in some cases, has to be done via exchanging files.
So, we are planning to resolve this problem as following:
- Each system maintains a changelog of its database. We are planning to implement it with MongoDB.
- When a system initializes a synchronization process, it retrieves all made changes from a log. If the system is B, the changes retrieved depend on the destination. Then, the system serializes them in XML format and, finally, sends them (via a file or a network).
- When the other endpoint receives the changeset, it unserializes them. Then, the system makes some transformations over the data, which can be necessary, and finally, records the changes. In this step, if it's necessary, the system has to resolve the conflicts which might exist.
- Last, the receiver system sends its changes (and other products of conflict resolution).
Is this approach feasible, scalable and elegant? What changes or additions would you make?
Best Answer
If you have not already done so you may find it interesting to read up on event-driven systems, event sourcing and eventual consistency. The system you are describing has many parallels with these patterns, which is a good thing.
Your approach sounds good, in particular:
Without knowing more about the domain model my guess is that resolving conflicts is the part that will cause you the most trouble. I would spend some time thinking through how each sort of conflict would be resolved. In particular: