Web-development – Two way data synchronization between web application and REST API server

designperformancerestsynchronizationweb-development

I have a web application and a REST API server (Microsoft Dynamics CRM 2016) that I don't have access to it's code, but I have access to the available resources. I need to create a data sync between both servers, some of the tables are one way and the others two way data sync has to be done, I haven't found any good solution to solve the problem.

I can handle the one way data sync by creating a log table of each SQL statement executed in my web application, and after a period of time I can use the REST API endpoints to sync data from my web application into the REST API server, but how to do the inverse (i.e sync data from REST API application into my web application)?

The naive solution are to get all table records from REST API server and then start syncing by comparing records from both sources, given that I currently don't have timestamp on the records, and I have no way to get the resources that are modified/inserted from the REST Server, because of that I should get all the resources from REST server and not the changed records which will increase the network traffic during every sync operation.

Insertion are not a problem, the problem is with update and delete operations.

Best Answer

Given the scenario where you can't control/access REST server, you have to make calls for every table until REST service expose some API to enable this feature. But off-course you can optimize data flowing through network:

  1. Create list of tables which have updates more recent than your service (by querying REST service).
  2. Now only sync tables from above list.

or,

Try to fetch updated/new data in one call (in step 1).

In my opinion above solutions are not fundamentally different from your current solution but they do try to control amount of traffic in network. You have to implement both and run load test on them to see what fits best.

Related Topic