You should have a look at the thread pool pattern in Wikipedia, I think this is what you are looking for. The trick is to create n threads once when your service starts, avoiding the overhead of thread creation (see this question on SO) during runtime.
As for your workflow, you may want to consider switching the first two steps:
First check if there are records to process, and if there are, see if your thread pool has a free thread. This is an optimisation problem, you want to reach the decision no processing (because there is no data, or no resources for it) with as little effort as possible. If you reach this decision more often because there is no data to process, check for this first.
Some will argue that this is premature optimisation, but I personally prefer to err a little more on the premature side.
This is not an easy question to answer succinctly.
First, the approach you choose depends on your very specific goals and constraints. To what degree will you log object messages or activities? And is logging an intrinsic activity (something that is a key, integral responsibility of the objects, or is it a secondary, really nice to have, book-keeping activity)?
There are situations where logging is intrinsic, and pervasive, but often it is a secondary concern (a book-keeping activity). This suggests that you do not want to strongly couple the logger to each object that you are tracking.
One approach to consider is something along the lines of an application global message queue, or enterprise service bus. Something that is closer to the first alternative rather than the second alternative you suggested.
Each object that you are interested in tracking with a logger would produce object or activity specific messages that each implement a common message interface that supports the common properties you are interested in each object recording in the log for identification. In each object or activity specific message, define and add the specific details to that message structure. Finally, send the message to the application's message listener queue. If acknowledgement of receipt is required, give it to the sender, so it may continue. Generally in a closed, single-host system, explicit acknowledgement of message receipt is not required.
In the application scoped message processor (globally available within your application), you would 'register' message handlers. This could be done 'lazily', as needed, or pre-emptively at start up, as part of 'bootstrapping'. Perhaps it would include a generic handler that works on any message that it receives. It may have a handler for that common logger message. It may also have a handler for several or all specific logger messages, if other activities need to occur in conjunction with that message. In any regard, the message processor would successively dequeue messages, handle messages with the registered message handlers, appropriately discard them, and then grab the next message.
Part of the implication of an application scoped message processor in a multi-threaded application is that the only guarantee to message order is that the order the message enters the message queue will play a large role in the order that it appears in a particular log.
There are strategies for making message order more deterministic, but none will be perfect in all cases. One way is to add a global message sequencer, and 'get_next' from the sequencer when crafting each and every message, then insert into an indexed queue, and wait to dequeue until the next message in the sequence arrives. This too has implications, most notably lost or discarded messages blocking the message queue.
Best Answer
1000 threads (some of them active once in 2 hours) is a big no-no. So is starting a new thread for each job which may finish few seconds later. Make one "scheduler" thread that selects URLs for retrieval, and a number of worker threads that report their state to the scheduler. Scheduler sequentially:
Then sleep and repeat the loop. Essentially, you have a semi-realtime parent thread that does all "fast" jobs and worker threads that have busy-wait states.
Of course the URL distribution can be done through Observer pattern, modified to "consume" the message if a "client" accepts it (hand out URL to retrieve). The list of threads can be a linked list to be traversed recursively.