I have FILE_A which has over 300,000 lines and FILE_B which has over 30 million lines. I created a Bash script that greps each line in FILE_A over in FILE_B and writes the result of the grep to a new file.
This whole process is taking over 5 hours.
How can I improve the performance of my script?
I'm using grep -F -m 1 as the grep command. FILE_A looks like this:
123456789
123455321
and FILE_B is like this:
123456789,123456789,730025400149993,
123455321,123455321,730025400126097,
So with Bash I have a while loop that picks the next line in FILE_A and greps it over in FILE_B. When the pattern is found in FILE_B, I write it to file result.txt.
while read -r line; do
grep -F -m1 $line 30MFile
done < 300KFile
Best Answer
Try using
grep --file==FILE_A. It almost certainly loads the patterns into memory, meaning it will only scan FILE_B once.