Xargs –max-proc split output per proc

unixxargs

I recently discovered the xargs --max-procs feature.

How can split the output of the command by proc? Should I just create a mycommand --logfile $LOGFILE, or can I do it from xargs itself?

An example (for womble):

Suppose I have script myprocessor.sh, and a list of files. They can go in any order, but i want to keep the logging for each separate, then:

find $MY_FILE_TREE --print0 | xargs --null --max-procs 3 --max-args 1 --no-run-if-empty myprocess.sh

might be the parallel job I want to run. If myprocessor.sh is mouthy, then I'd like to be able to have each invocation print to a different log. Otherwise the stdout for each is the same, and the logs get jumbled.

Best Answer

You could do this by running your xargs command through a shell - this will let you redirect the output - something like this:

find blah -type f | xargs -I{} -P 4 -n 1 sh -c 'yourcommand --input {} > {}.output'

...you'll probably have to tweak it a bit - xargs replaces {} with the item/file it's working on

Related Topic