Handling Large Amounts of Web Service Requests and Data Storage

When building distributed systems, the difference between a 'synchronous' system and an 'asynchronous' system is this: A synchronous system has known upper bounds on computation and message delivery times. So: you have an asynchronous system where certain events do not have these known upper bounds. How do you handle it?

1. If these asynchronous processes have probabilistic upper bounds then you can use timeouts to make your system act like a partially synchronous system. If the payment gateway's 98th percentile response time is 5 seconds then a 5 second timout will make 98% of your requests succeed and the other 2% will just fail. This means that you now have a known upper bound on how long this process will take to either succeed or fail. This probabilistic failure detection is a critical tool for turning asynchronous systems into synchronous systems.

2. Keep a durable record of these events so that you can recover your system state in the event of system failure. If your payment gateway handler is keeping these events in volatile memory and it crashes then you're screwed.

3. Each complex transaction is essentially a series of state transformations based on the sending and receiving of messages (events) within the system. It sounds like you are informally modeling this using your "record of pending transactions" but I would suggest that you go further: For each transaction you need to manage, create a formal state machine that describes it and keep a durable record of its current state. You will find that these state machines are easy to understand, easy to test, and give much-needed visibility into these processes both for you and your users.

The more asynchronous your system, the more formal and explicit you need to be when managing these complex evented state transformations. Timeouts, durable event logging, and state machines are the best practice here. This is why Erlang OTP bases much of its application behavior on the state machine model, for instance.

For reference, I haven't found anything better than Introduction to Reliable and Secure Distributed Programming. It will give you a strong algorithmic basis for understanding both synchronous and asynchronous systems from first principles.

I worked on a similar application for the California Energy Commission (UT3, or Universal Translator 3). A couple points:

• The "filling in" algorithm may not be correct mathematically. The data in question is typically average values. The correct way to resample in time, average value data, is to use a time weighted average. It is a type of integration.
• On the other hand, you may have some instantaneous type data, that needs a different type of numerical integration. I suggest you study the mathematical properties needed more closely.
• The best performance will come from using a custom database that stores only the original, non-resampled data, in a binary form. A "custom database" can consist of a set of read-only files, each of which contains a data stream. A time-series specific database is probably fast enough.
• The re-sampling algorithms are fast enough that they can be performed in real-time on the client.
• If the end user sampling frequency is much lower than the original data, of course it makes sense to cache the resampled streams on the server. Access to the original data makes client based designs (like Universal Translator) more appealing for some uses.