I'm currently working on a service that is supposed to aggregate data from a number of APIs, unify the data, and offer it through another API to the users. I had a couple of ideas for solving this, but I'm not confident my best solution is the best possible solution.
Solutions I considered:
- Let the browser poll our API through javascript, which in turn polls the third party APIs. This introduces quite a bit of lag between the request and the user seeing the data, but doesn't waste resources on users that rarely log in.
- Let the browser poll the third party APIs and afterwards send the data to our API. This is potentially insecure, and requires me to provide extra API endpoints and a lot more data cleanup.
- The option I currently consider the most optimal is letting a service on the server (an actual executable) periodically poll the third party APIs. I would, for example, poll some endpoints every 30 minutes, and others maybe once a day. This is the safest option, and introduces little lag to user requests, and prevents users from reaching the request limit of the API. But data will only be up-to-date right after polling.
Are there any alternatives? If not, is my current optimal solution good enough?
Best Answer
Another option is not to poll at all, but to implement a caching solution where, when a request comes in to your API, you use a cached value if you have one that's recent enough, otherwise you update your cache with a call to the external API.