Printable View
I'm find a way of compressing and encrypted large files without using too much memory.
To do this I'm planning on reading in the file data 4MB at a time, compressing it, and encrypting it before writing to another file.
The problem is the lack of a suitable compression algorithm. Is there one that will compress these 4MB chunks, but still consider them to be sections of a larger file?
Hmmmm... Hope you know what I mean.
RAR can do that, but I don't think it's open.
ZIP can do it too, you it's up to you to decide what to do with the compressed files.
I could write an RLE algorithm that supports file splitting...
Basically, if you write your own compression/decompression tool you can use every compression algorithm for it.
It's not really what I meant. If I've got a 1 GB file (for example) and wanted to compress it then I wouldn't read in all 1 GB into memory at once. I want to read it 4 MB at a time, compress the 4MB chunk then output them sequentially to another file.
I'm not really familiar with data compression, but I think it involved created 'dictionarys'? I need a method whereby the 4MB chunks would all share the same dictionary.
Some use dictionaries like that. With zlib, you can compress your own sized blocks, normally about 32Ks worth in one function call. You can get prewritten stream interfaces for it (look for ozstream).
u can use rar see in their page for the dll www.rarsoft.com
You can't get sources for that though.