Multithreading – Multithreaded File Copy in .NET

designfile handlingmultithreadingnet

There is an utility we have which is used to upload files (and perform other operations on the file) to a network shared location.
The file size tends to vary from a few mb to 500 mb.
A suggestion has come up that we should maybe support multi-threading when uploading the files to the shared location – not required to do it in byte chunks – each thread should pick one file and try to upload.

I am not so sure if multithreading can speed up IO operations like this. Is my hunch valid?

If indeed we are required to build this functionality I was wondering what would be a good design approach for the copy file engine?
Would it make sense to use a tool like robocopy (I read the newer versions support multithreading)?

Edit: Apologies for the delay and missing some vital information.
This utility is built using C# (.Net 2.0) and any future update also has to be using .Net (framework version is not a constraint). The utility is installed on the machines of the users (around 20 all on WinXP). The target share is on Win2k3 server.

Edit 2: have decided to run some tests with a simple application implementing the file upload through TPL. Post this analysis we will decide whether to go ahead or not. Thanks everyone for the help extended.

Best Answer

This depends on what the limiting factor is, doesnt it? If the bottleneck is the utility program, then sure, running more than one copy or using more threads will speed things up. If the network is the limiting factor, then adding multiple instances of the utility isnt going to help since you still will be stuck moving at most X bytes per second. In fact it might hurt because you have the additional overhead of a second copy of the app. Same with disk-IO. You can only copy as fast as either machine can read from and write to disk. If thats already max'd out, adding copies isnt going to help.

What you need to do is test to see what the bottleneck is, and go from there.

Related Topic