I really like using queues for interprocess communication. It's great to be able to spin up a ton of workers and be guaranteed that a job in the queue is only run once, is treated fairly, can be scaled horizontally, and can optionally expire if it is not handled quickly enough.
However, I can't seem to find a job queue implementation that also supports putting jobs into the queue at a specified time.
What's the best way to attack this problem? Does a queue like this exist?
So far my alternatives to such a queue are:
- beanstalkd – supports delayed jobs, I could calculate the delta between now and the specified time.
- imprecise: what happens on timezone changes, or really large deltas (months)
- "at" command – create an at job that would spin up and put the schedule job in a queue at the specified time.
- tied to a single machine: What happens if the system restarts? How do you port jobs to a new server? Single point of failure
- manually handle expiration if you have a long downtime.
- delayed_job/custom polling – create a daemon (something like delayed_job) that continuously polls a database of scheduled jobs and inserts into queue at the specified time
- seems too complicated
Best Answer
Something like the Open Source Job Scheduler may be appropriate...