File Handling – Optimize Reading of Multiple Files in C#

According to Wikipedia:

Fields with embedded commas must be enclosed within double-quote characters.

And furthermore:

Fields with embedded double-quote characters must be enclosed within double-quote characters, and each of the embedded double-quote characters must be represented by a pair of double-quote characters.

I don't know who invented that, but it effectively shows that eventually you have to escape. It's the only solid solution. Everything else is just duct tape on top of duct tape: maybe works for now, but eventually you'll bump on a case where you need an exception to the exception of an exception, and it doesn't take long before your mudball of rules is way more complex than a simple escape character solution had been.

It seems that CSV creators first tried to avoid escaping commas by coming up with double-quoted special syntax, which allowed saving commas, but then someone wanted to save double-quote characters too, so they had to escape at that point - funnily using the double-quote as an escape character. Had they decided to escape properly in the first place, the syntax would be simpler now.

you can use a random RandomAccessFile

private void join(RandomAccessFile left, RandomAccessFile right, BufferedWriter output) throws Throwable {

    long rightIndex = right.getFilePointer();

    Record leftRecord = read(left);
    Record rightRecord;

    while((leftRecord )!=null){
        right.seek(rightIndex);

        while((rightRecord = read(right))!=null && leftRecord.getKey() > rightRecord.getKey()){
            //skip right until it can match left
            // store pointer because we don't need to return to earlier
            rightIndex = right.getFilePointer();
        }

        while(rightRecord !=null && leftRecord.getKey()==rightRecord.getKey()){
            write(_output, leftRecord, rightRecord );
            rightRecord = read(right))
        }
        leftRecord = read(left);
        //you can test the need to skip this left record if you keep the previous key and the current rightRecord.getKey()
    }
}

this will be a O(n+m) time and O(1) space algorithm