I'm currently assigned a task, where I have to program a bidirectional synchronisation between 2 ERP-related software, one providing webservices (Visual Studio Web Reference) and the other one providing a REST API.
I managed to match the fields of the APIs together, so I could potentially start creating objects in both APIs, using the same data. However, I reached the point of the actual sync, and it's harder than I thought.
In my head, everything goes fine.
Grabbing the data -> determine, what change it is on which side -> sync the data from the source system to the target system
But in actual coding, I'm encountering some serious problems:
- Both APIs don't have any way to tell, when data was modified the last time, making it hard to determine, what kind of change it is (I thought about having a list of the previous sync to compare, but that leads to a different problem)
- There's a priority on the system with the REST API, so if there are 2 changes at the same time, the change in the REST API would count
- In which order should they be sync'ed? If there's a focus on the REST API, then the REST API would need to apply their changes after the webservice one, right?
- The sync software would run on a server, at the moment in a permanent loop of getting the data, comparing and applying the changes. I'd love to only sync the changed data, but that wouldn't be possible, regarding the problem, that there's no way for me to determine, what change occured (something was created, something was changed, something was deleted)
Can you give me advice on how to accomplish this, in a reliable and resource-friendly way?
The webservice API is a lot slower than the REST API, meaning there's an interval, where nothing gets sync'ed (it may sound stupid, but I thought I could solve this problem single-threaded). Could I potentially need multi-threading here? The webservice API is rate limited to 10000/hr, dunno if this could get a problem.
Best Answer
Welcome to the world of integration. The hardest thing about it is how poorly most vendors understand what you need and the resulting gaps in their interfaces.
In this case you seem to have a really intractable problem:
No amount of design in your integration will fix this flaw in the API. You either have to get the vendors to address this issue (last update time, among other things should be included, there's no justification for not providing it) or you need to come up with a different approach to meet your requirements.
I think what you might be trying to do is on an interval, look at the two systems and apply any changes on one to the other. If both systems have changed in the interval you want to prefer on over the other. This is possible but you will probably lose changes made on the non-preferred system if your interval is fairly long. The way you can do this based on what you have provided is to look at all the records and compare them to some sort of record that you keep to determine if they have changed. For example you can keep a table of the record ids and a hash of the record. When the hash changes, you know there was a change made since the last time you checked. Then you can apply the update. This is not efficient and you can have a dirty read issue.
The problem you are trying to solve is essentially that of eventual consistency. Bitcoin, for example has a pretty interesting way of addressing it. I think if you read up on how it is accomplished, you'll find that you lack the requisite data elements to make it work.