How to implement a multi-threaded program in which each thread needs to write/output some data (in any order of course) but without the locking overhead that is involved with files or a particular data structure.
Is each thread writing to a separate file a feasible option? (wrt to locking overhead)
Can we use System.out.print or BufferWriter instead of files? Or is there any other elegant solution?
Best Answer
If you have multiple threads generating data independently from each other, writing to separate files may be indeed a good option. There will still be some locking under the hood since the Operating System needs to keep the filesystem consistent, but it should be better than writing to a single file with manual locking or writing to standard output (which has internal synchronization). Similarily if you have access to some online storage which you can access via the network, you could send the data independently from each thread.
But there is a big "but". If you write out a lot of data (more than several MB/s, depending on machine specs), your file system may not be able to keep up. In such a situation, the bottleneck will be your file system and not locks in the app. If this happens, you might actually be better off with a single file and locking since concurrent writes to multiple locations may put more load on the filesystem and disk than writing at a single spot. So it all depends and any number of concurrent writes between one and many may be optimal. Your best bet would be to make the number of concurrent writes parametrizable and then to compare the final performance of the application for several different values.