Is there a way to assign each worker in a python multiprocessing pool a unique ID in a way that a job being run by a particular worker in the pool could know which worker is running it? According to the docs, a Process
has a name
but
The name is a string used for identification purposes only. It has no
semantics. Multiple processes may be given the same name.
For my particular use-case, I want to run a bunch of jobs on a group of four GPUs, and need to set the device number for the GPU that the job should run on. Because the jobs are of non-uniform length, I want to be sure that I don't have a collision on a GPU of a job trying to run on it before the previous one completes (so this precludes pre-assigning an ID to the unit of work ahead of time).
Best Answer
It seems like what you want is simple:
multiprocessing.current_process()
. For example:Output:
This returns the process object itself, so the process can be its own identity. You could also call
id
on it for a unique numerical id -- in cpython, this is the memory address of the process object, so I don't think there's any possibility of overlap. Finally, you can use theident
or thepid
property of the process -- but that's only set once the process is started.Furthermore, looking over the source, it seems to me very likely that autogenerated names (as exemplified by the first value in the
Process
repr strings above) are unique.multiprocessing
maintains anitertools.counter
object for every process, which is used to generate an_identity
tuple for any child processes it spawns. So the top-level process produces child process with single-value ids, and they spawn process with two-value ids, and so on. Then, if no name is passed to theProcess
constructor, it simply autogenerates the name based on the _identity, using':'.join(...)
. ThenPool
alters the name of the process usingreplace
, leaving the autogenerated id the same.The upshot of all this is that although two
Process
es may have the same name, because you may assign the same name to them when you create them, they are unique if you don't touch the name parameter. Also, you could theoretically use_identity
as a unique identifier; but I gather they made that variable private for a reason!An example of the above in action:
Output: