C# – Quickest way in C# to find a file in a directory with over 20,000 files

cfile-ionet

I have a job that runs every night to pull xml files from a directory that has over 20,000 subfolders under the root. Here is what the structure looks like:

rootFolder/someFolder/someSubFolder/xml/myFile.xml
rootFolder/someFolder/someSubFolder1/xml/myFile1.xml
rootFolder/someFolder/someSubFolderN/xml/myFile2.xml
rootFolder/someFolder1
rootFolder/someFolderN

So looking at the above, the structure is always the same – a root folder, then two subfolders, then an xml directory, and then the xml file.
Only the name of the rootFolder and the xml directory are known to me.

The code below traverses through all the directories and is extremely slow. Any recommendations on how I can optimize the search especially if the directory structure is known?

string[] files = Directory.GetFiles(@"\\somenetworkpath\rootFolder", "*.xml", SearchOption.AllDirectories);

Best Answer

Rather than doing GetFiles and doing a brute force search you could most likely use GetDirectories, first to get a list of the "First sub folder", loop through those directories, then repeat the process for the sub folder, looping through them, lastly look for the xml folder, and finally searching for .xml files.

Now, as for performance the speed of this will vary, but searching for directories first, THEN getting to files should help a lot!

Update

Ok, I did a quick bit of testing and you can actually optimize it much further than I thought.

The following code snippet will search a directory structure and find ALL "xml" folders inside the entire directory tree.

string startPath = @"C:\Testing\Testing\bin\Debug";
string[] oDirectories = Directory.GetDirectories(startPath, "xml", SearchOption.AllDirectories);
Console.WriteLine(oDirectories.Length.ToString());
foreach (string oCurrent in oDirectories)
    Console.WriteLine(oCurrent);
Console.ReadLine();

If you drop that into a test console app you will see it output the results.

Now, once you have this, just look in each of the found directories for you .xml files.

Related Solutions

Windows – Batch file to delete files older than N days

Enjoy:

forfiles -p "C:\what\ever" -s -m *.* -d <number of days> -c "cmd /c del @path"

See forfiles documentation for more details.

For more goodies, refer to An A-Z Index of the Windows XP command line.

If you don't have forfiles installed on your machine, copy it from any Windows Server 2003 to your Windows XP machine at %WinDir%\system32\. This is possible since the EXE is fully compatible between Windows Server 2003 and Windows XP.

Later versions of Windows and Windows Server have it installed by default.

For Windows 7 and newer (including Windows 10):

The syntax has changed a little. Therefore the updated command is:

forfiles /p "C:\what\ever" /s /m *.* /D -<number of days> /C "cmd /c del @path"

C# – way to check if a file is in use

Updated NOTE on this solution: Checking with FileAccess.ReadWrite will fail for Read-Only files so the solution has been modified to check with FileAccess.Read.

ORIGINAL: I've used this code for the past several years, and I haven't had any issues with it.

Understand your hesitation about using exceptions, but you can't avoid them all of the time:

protected virtual bool IsFileLocked(FileInfo file)
{
    try
    {
        using(FileStream stream = file.Open(FileMode.Open, FileAccess.Read, FileShare.None))
        {
            stream.Close();
        }
    }
    catch (IOException)
    {
        //the file is unavailable because it is:
        //still being written to
        //or being processed by another thread
        //or does not exist (has already been processed)
        return true;
    }

    //file is not locked
    return false;
}

Best Answer

Related Solutions

Windows – Batch file to delete files older than N days

C# – way to check if a file is in use

Related Topic