Linux – Redirecting stdout with find -exec and without creating new shell

linuxunix

I have one script that only writes data to stdout. I need to run it for multiple files and generate a different output file for each input file and I was wondering how to use find -exec for that. So I basically tried several variants of this (I replaced the script by cat just for testability purposes):

find * -type f -exec cat "{}" > "{}.stdout" \;

but could not make it work since all the data was being written to a file literally named{}.stdout.

Eventually, I could make it work with :

find * -type f -exec sh -c "cat {} > {}.stdout" \;

But while this latest form works well with cat, my script requires environment variables loaded through several initialization scripts, thus I end up with:

find * -type f -exec sh -c "initscript1; initscript2; ...;  myscript {} > {}.stdout" \;

Which seems a waste because I have everything already initialized in my current shell.

Is there a better way of doing this with find? Other one-liners are welcome.

Best Answer

You can do it with eval. It may be ugly, but so is having to make a shell script for this. Plus, it's all on one line. For example

find -type f -exec bash -c "eval md5sum {}  > {}.sum " \;

Related Solutions

Python – Why is printing to stdout so slow? Can it be sped up

How can it be that writing to physical disk is WAY faster than writing to the "screen" (presumably an all-RAM op), and is effectively as fast as simply dumping to the garbage with /dev/null?

Congratulations, you have just discovered the importance of I/O buffering. :-)

The disk appears to be faster, because it is highly buffered: all Python's write() calls are returning before anything is actually written to physical disk. (The OS does this later, combining many thousands of individual writes into a big, efficient chunks.)

The terminal, on the other hand, does little or no buffering: each individual print / write(line) waits for the full write (i.e. display to output device) to complete.

To make the comparison fair, you must make the file test use the same output buffering as the terminal, which you can do by modifying your example to:

fp = file("out.txt", "w", 1)   # line-buffered, like stdout
[...]
for x in range(lineCount):
    fp.write(line)
    os.fsync(fp.fileno())      # wait for the write to actually complete

I ran your file writing test on my machine, and with buffering, it also 0.05s here for 100,000 lines.

However, with the above modifications to write unbuffered, it takes 40 seconds to write only 1,000 lines to disk. I gave up waiting for 100,000 lines to write, but extrapolating from the previous, it would take over an hour.

That puts the terminal's 11 seconds into perspective, doesn't it?

So to answer your original question, writing to a terminal is actually blazingly fast, all things considered, and there's not a lot of room to make it much faster (but individual terminals do vary in how much work they do; see Russ's comment to this answer).

(You could add more write buffering, like with disk I/O, but then you wouldn't see what was written to your terminal until after the buffer gets flushed. It's a trade-off: interactivity versus bulk efficiency.)

Linux – ‘find -exec’ a shell function in Linux

Since only the shell knows how to run shell functions, you have to run a shell to run a function. You also need to mark your function for export with export -f, otherwise the subshell won't inherit them:

export -f dosomething
find . -exec bash -c 'dosomething "$0"' {} \;

Best Answer

Related Solutions

Python – Why is printing to stdout so slow? Can it be sped up

Linux – ‘find -exec’ a shell function in Linux

Related Topic