Syncronize Files

[johnweidauer]

I have written a stupid backup utility that copies new or modified files. That's fantastic until I come across files that are 11 GB in size that have been modified. I have a feeling I can compare the file to be backed up and the file that will be over-written (old version of file) and "synchronize" the file instead of copying the entire 11 GB. I am not sure how to start researching this, could anyone point me in the right direction?

[weirddemon]

Hm... if the one file is 11GBs in size, I don't think you can copy bits of it like I think you're suggesting. The only thing you can do, as far as I know, is compare the modified dates and back it up like that.

It seems odd that any one file is 11GBs though. What kind of file is it? I'm intrigued.

[CVMichael]

It seems odd that any one file is 11GBs though. What kind of file is it? I'm intrigued.

A database file for example...

johnweidauer,
One way that I can think of is to keep a hash of each block of the file.
What I mean is:
First time you copy the file, then read every 128KBytes (for example) and hash it, then read the next 128KB and hash it again, and so on.
Store the hash data in a database (or just another file). The hash (if MD5, then it's 128 Bit, or 16 Bytes), so basically you will have 16 bytes from every 128KBytes.

Now... when you want to compare the new file with the old file to see what has changed, then compare the hash data instead.

So, read 128KBytes from the new file, and hash it, then compare the 16 hash result with the previous 16 bytes of hash from the old file. If it's the same, it means no data has changed in the first 128KBytes of the new file. Then repeat the process until you compare the whole file with the old one.
When you get different hash, then copy and override ONLY that 128KByte block of data, into the same place in the old file. If you repeat this process the old file will contain the same data as the new file, but you did not have to copy the entire file.

This is especially useful if you have to transfer the data over the network, or internet.

Awesome isn't it ?

[weirddemon]

A database file for example...

Oh, of course. Silly me.

I too was thinking of hashing, but it didn't occur to me that he could hash in blocks like that.

[johnweidauer]

hash it, that's what I was thinking of, thanks.

[Grimfort]

This may seem like a daft idea, but M$ have created a command line util that does all this called RoboCopy and has great fault tolerance built into it. You could just write a front end for this (even though there is one available) to incorp into your own system.

[johnweidauer]

that is ingenious, I hate reinventing the wheel, thank you.