Diff and patch for big binary files

diff()patch-management

As part of my job I regularly need to upload big files (a 512 MB to 1 GB ISO, for example). In some cases, very little changes from ISO to ISO (say, a 10 MB file in a 1 GB ISO). Unfortunately, I can't just send the changed file – I need to send the whole ISO. A faster connection is not a valid solution.

Is there any tool (preferably Linux, preferably free) that can make and apply diffs of such big binary files?

Best Answer

I use xdelta for this purpose, others use rdiff. Both work nicely.

Related Solutions

Get rsync to generate a patch file instead of copying across files

There might be a better way, but this might work, albeit not that efficiently:

 rsync -vrn / dest:/ > ~/file_list

Then edit test to remove the stats, then:

while read file; do
    diff $file <(ssh dest "cat $file")
done < ~/edited_file_list

Another Option:
You might also consider mounting the file system with something like sshfs/fuse, and then just using diff.

How to get diff to show only added and deleted lines? If diff can’t do it, what tool can

Try comm

Another way to look at it:

Show lines that only exist in file a: (i.e. what was deleted from a)
```
  comm -23 a b
```
Show lines that only exist in file b: (i.e. what was added to b)
```
  comm -13 a b
```
Show lines that only exist in one file or the other: (but not both)
```
  comm -3 a b | sed 's/^\t//'
```

(Warning: If file a has lines that start with TAB, it (the first TAB) will be removed from the output.)

Sorted files only

NOTE: Both files need to be sorted for comm to work properly. If they aren't already sorted, you should sort them:

sort <a >a.sorted
sort <b >b.sorted
comm -12 a.sorted b.sorted

If the files are extremely long, this may be quite a burden as it requires an extra copy and therefore twice as much disk space.