Creating a queue for web services

queueweb services

I have a program running that collects metadata from the internets.

Every so often a summary of this data is written up (every 30 min or so) and the summary is posted to a database.

The summary is also sent out to many web service APIs.

I'm trying to avoid the problem of this program locking up when webservice APIs timeout. So I'm thinking of creating a queue to process these webservice requests.

How should I create this queue? With a database of some sort? It could be a program storing queued items in memory, I suppose. Is there some standard process for creating a queue for webservice calls?

Best Answer

This sounds like a perfect use-case for Apache Kafka:

Each partition is an ordered, immutable sequence of messages that is continually appended to—a commit log. The messages in the partitions are each assigned a sequential id number called the offset that uniquely identifies each message within the partition. The Kafka cluster retains all published messages—whether or not they have been consumed—for a configurable period of time. For example if the log retention is set to two days, then for the two days after a message is published it is available for consumption, after which it will be discarded to free up space. Kafka's performance is effectively constant with respect to data size so retaining lots of data is not a problem.

Link to the docs: http://kafka.apache.org/documentation.html#introduction

Related Topic