Java – Converting cron schedule to time intervals

Do you have a shared database? I've done this using a database as the arbiter in the past.

Basically, each "job" is represented as a row in the database. You schedule a job by adding a row to the database with the time you want it to run then each server does:

SELECT TOP 1 *
FROM jobs
WHERE state = 'NotRun'
ORDER BY run_time ASC

That way, they'll all pick the job that is scheduled to run next. They all sleep so that they wake up when the job is actually supposed to run. Then, they all do this:

UPDATE jobs
SET state = 'Running'
WHERE job_id = :id
  AND state = 'NotRun'

Where :id is the identifier of the job you got in the step above. Because the update is atomic, only one of the servers will actually update the row, you can check the database's "number of rows updates" status code to determine whether you were the server that actually updated the row, and therefore whether you are the server that gets to run the job.

If you didn't "win" and you're not running the job, just go back to step 1 immediately. If you did "win", schedule the job to execute in another thread, then wait a couple of seconds before going back to step 1. That way, servers that didn't get the job this time are more likely to pick up a job that's scheduled to run immediately.

Part of what you are looking for is a "priority queue". At previous employers, we did a very primitive version of this, but my heuristic was to only allow some processors to handle short running jobs (short jobs could take minutes), while others were handling longer running jobs (the quarterly report could take almost 2 days to process). This guaranteed that short jobs always had processing time available. I also used a scoreboard that listed jobs ready to be run, and the first processor able to handle the task would pick it up and run it single threaded (they were cheap computers that hadn't been depreciated and so could not be discarded). Many folks use the opposite: a scheduler that tells processors which work unit to do next. My advice would be to have each instance running a single task - this drastically simplifies the scheduling.

Scheduling of arbitrary jobs of arbitrary lengths is a hard problem in distributed processing. Almost every decision is going to involve simulating lots of runs. Which is one of the quirks of queuing theory, which this stuff is going to be based on.

One of the other devs suggested we leave the "job processors" alone to just pull whatever is in the queue next or "round robin". I say that this could lead to a potential issue where a single instance has pulled down too many large jobs and is struggling to get them done while the other instances are idle.

This needs simulation to answer. My earlier scheme used something very similar. If you have stats on previous job runs, you can model it in Excel. I've picked up this book from another post recommending it and am looking to learn some techniques to better be able to answer problems like what you're describing. Actual numbers trump everything, so gather data and do simulations based on them.