I have a question understanding the queue in the multiprocessing
module in python 3
This is what they say in the programming guidelines:
Bear in mind that a process that has put items in a queue will wait before
terminating until all the buffered items are fed by the “feeder” thread to
the underlying pipe. (The child process can call the
Queue.cancel_join_thread
method of the queue to avoid this behaviour.)This means that whenever you use a queue you need to make sure that all
items which have been put on the queue will eventually be removed before the
process is joined. Otherwise you cannot be sure that processes which have
put items on the queue will terminate. Remember also that non-daemonic
processes will be joined automatically.An example which will deadlock is the following:
from multiprocessing import Process, Queue def f(q): q.put('X' * 1000000) if __name__ == '__main__': queue = Queue() p = Process(target=f, args=(queue,)) p.start() p.join() # this deadlocks obj = queue.get()A fix here would be to swap the last two lines (or simply remove the
p.join() line).
So apparently, queue.get()
should not be called after a join()
.
However there are examples of using queues where get
is called after a join
like:
import multiprocessing as mp
import random
import string
# define a example function
def rand_string(length, output):
""" Generates a random string of numbers, lower- and uppercase chars. """
rand_str = ''.join(random.choice(
string.ascii_lowercase
+ string.ascii_uppercase
+ string.digits)
for i in range(length))
output.put(rand_str)
if __name__ == "__main__":
# Define an output queue
output = mp.Queue()
# Setup a list of processes that we want to run
processes = [mp.Process(target=rand_string, args=(5, output))
for x in range(2)]
# Run processes
for p in processes:
p.start()
# Exit the completed processes
for p in processes:
p.join()
# Get process results from the output queue
results = [output.get() for p in processes]
print(results)
I've run this program and it works (also posted as a solution to the StackOverFlow question Python 3 – Multiprocessing – Queue.get() does not respond).
Could someone help me understand what the rule for the deadlock is here?
Best Answer
The queue implementation in multiprocessing that allows data to be transferred between processes relies on standard OS pipes.
OS pipes are not infinitely long, so the process which queues data could be blocked in the OS during the
put()
operation until some other process usesget()
to retrieve data from the queue.For small amounts of data, such as the one in your example, the main process can
join()
all the spawned subprocesses and then pick up the data. This often works well, but does not scale, and it is not clear when it will break.But it will certainly break with large amounts of data. The subprocess will be blocked in
put()
waiting for the main process to remove some data from the queue withget()
, but the main process is blocked injoin()
waiting for the subprocess to finish. This results in a deadlock.Here is an example where a user had this exact issue. I posted some code in an answer there that helped him solve his problem.