Design periodic batch notification system

Architecturedesignperformancescalability

Problem:

  • I am trying to design a system which takes requests from customers in
    real time and log them in a database. For example – request to
    purchase an item.
  • The customer then gets a unique request-ids for his request and also
    is asked to wait for 'X' days because the request is being processed.
  • Then the system notifies the item providers by batching all these
    requests in one single email periodically (lets say 4 hours).
  • Item providers update items in their repository and publish them to
    customers. And then customer gets a notification that item providers
    have responded for his request.

Design:

I'm thinking of using a workflow based system for this use case. With
following components (high level) –

Synchronous API:
 - Receive request from customer. 
 - Record the request in the database.
 - Also start an asynchronous workflow, return request id as a response.

Workflow does the following :

Workflow Step 1:
 - Get the list of item providers
 - Notify them

Workflow Step 2: (This workflow steps gets executed after "X" days)
 - Send a response email back to the customer

But I'm having hard time to support the batching use-case. Is there
any way I can efficiently batch the customer requests and send
notification every 4 hours without having to write a cron job
separately?

I'd appreciate any suggestions/help on this.

Best Answer

This is very simple: requests for process a task are made as normal - all that happens is an entry gets written to a DB with a timestamp on it. Trivial to implement.

Elsewhere in your program, you have a thread that goes to sleep for the required time, wakes up after 4 hours, reads all outstanding tasks and sends them off to suppliers.

The same applies for responses - the supplier sends a response into the system which writes it to a DB. Periodically a task wakes up, reads all outstanding responses and sends an email off to the customer.

Now you can optimise this, as you should see the steps are the same for both customer and supplier. You can set up one system that receives requests and writes data to the DB, and something else that is responsible for reading and sending responses. If you then optimise further, you can use an external system like cron to handle the wakeup processing - all it has to do is send a request to the system that triggers processing.

So you only have to code up a small amount of architecture that gets reused for the customer-request and supplier-response; and the wake-up and process entries. This is mainly the point of batch-processing, you keep it very simple and reuse the same process to process different data.

Related Topic