[RESOLVED] loading large files

[killo]

i am using:

status.Caption = "loading file..."
filenum = FreeFile
Open Text1.Text For Binary As #filenum
strtext1 = Input(LOF(1), 1)

Close #filenum

to open a file for encryption, but the problem is, if i try and load large files and the computer does not have enough ram to do this then i get a few errors. i was thinking maybe i could split it up and load one chunk then encrypt that chunk and save it and then do the next chunk and save it to the end of the file so when it has done them all the file should be all together but i am not sure how to go about it. i am having trouble because i do not know how to tell visual basic to load, say, the first 10000000 characters of a file. does anyone know how i can do this? thanks.

[Code Doc]

Quick response. Go ahead and use binary mode, but keep using Get to get the next chunk in pieces that are a fraction of the file size (say a twentieth). The pointer will move automatically with each subsequent Get.

MyString = SPACE(LOF(1)/20)
Get 1, , MyString
'Encrypt 
Get 1, , MyString
' Encrypt
'etc.

[killo]

i don't know about this piece of code but are you sure that it isn't
Get 2, , MyString
for the second part? because it is
Get 1, , MyString
for the first part so shouldn't it be
Get 2, , MyString
for the second part?

also is there a quick function to tell how big the file is so i know haw many chunks i have to split it into?

[Merri]

If you're loading for encryption, you should be using different datatype. When you load data into strings, you're actually using double the memory: VB does an automatical ANSI -> UTF-16 conversion, as every character in VB is two bytes. When saving files it does the opposite conversion, so Unicode characters in the strings are lost.

I guess you're probably better off with a byte array, although integer and long arrays might work for you as well. However, I'm all too tired to put up an example (but you'll find plenty if you search: you could search for "chunks").

You can use LOF(1) and FileLen(Filename) to determine the size of a file. Most often chunks are done in a fixed size for all files, and then the last chunk that might not be of the full size is loaded as a smaller chunk.

You can also use LenB(Dir$(Filename, vbHidden)) <> 0 to check if a file exists at all.

[DigiRev]

If you encrypt the file chunk by chunk then that's how you will need to decrypt the file or the decryption will fail.

If the encryption algorithm changes the size/length of the data then it will be hard to know where one chunk starts and ends in the encrypted file unless you use a delimiter (which can also be unreliable).

Either way, for many reasons, loading an entire file into a string (unless it's an ASCII file) is not a very good idea to begin with and is very slow.

[killo]

digi rev: yes i have thought about that, but my encryption algorithm outputs the same string length as the input string length and it should all be ok as long as i join the chunks together in the right order which i will make sure that my program does also it wont matter if i encrypt it in 5 chunks then decrypt in say, 30 chunks because of the kind of encryption algorithm i am using it won't affect it

merri: thanks for the piece of code: FileLen(Filename) i will use that
also does anyone know which piece of code is right in my second post?

[Logophobic]

The number is the file number used to open the file, not any particular part of the file:

Open filename For Binary As #filenum
  Get #filenum, , Chunk ' read the first chunk
  ...
  Get #filenum, , Chunk ' read the next chunk
  ...
Close #filenum

[killo]

i have implemented this into my code and i am having problems can anyone tell me whats wrong? heres how i'm using it:

'get the length of the file
lengthfile = FileLen(load)
'determine how many 1mb parts there is going to be
parts = Round((lengthfile / 1000000), 0)
'open the file
filenum = FreeFile
Open load For Binary As #filenum
strtext1 = Space(LOF(1) / parts)
length1 = 1
For d = 1 To parts

Get #filenum, , strtext1
Call encrypt

'save the part, when this loops around again it is supposed to save the second part onto the end of the file and it knows where the end of the file is by the variable length1
filenum = FreeFile
Open save For Binary As #filenum
Put #filenum, length1, strtext2
Close #filenum

length1 = length1 + Len(strtext1)

Next d

Close #filenum

it encrypts the first chunk ok but then when it gets to the second chunk it gives me an error saying:
"bad file name or number"
and highlights:
Get #filenum, , strtext1

[Merri]

You must use another variable, like filenum2 instead of filenum. Now you're trying to close two times one file, but not close the first opened file.

Also, use LOF(filenum) instead of LOF(1) so you can be sure you get the size of the correct file.

[killo]

ah right i understand now, thanks. i'll see if it works

[killo]

no i'm still having problems, here is my new code:

'get the length of the file and then use this to calculate how many roughly 1mb parts it needs to be split into
lengthfile = FileLen(load)
parts = Round((lengthfile / 1000000), 0)
'load the file
filenum = FreeFile
Open load For Binary As #filenum
strtext1 = Space(LOF(filenum) / parts)
length1 = 1
For d = 1 To parts

'  load the first part into strtext1
  Get #filenum, , strtext1
'  encrypt strtext1
  Call encrypt

'  the output of the encryption is strtext2 so save this onto the end of the file by using length1 which stores how many characters there is until the end of the file
  filenum2 = FreeFile
  Open save For Binary As #filenum2
  Put #filenum2, length1, strtext2
  Close #filenum2



'  update the length of the file
  length1 = length1 + (Len(strtext1))
Next d

Close #filenum

[killo]

it loads and engrypts the first chunk ok but when it starts the second chunk it plays up...

[killo]

doesn't anyone know why it isn't working?

[Merri]

One of the things you're doing "wrong" is that you're splitting file into million pieces, but if there is less than million bytes in the file, the chunk size is smaller than one byte -> code just doesn't work.

    Const CHUNKSIZE = 1048576
    Dim FileRead As String, FileSave As String

    Dim intFF As Integer, intFF2 As Integer, lngLen As Long
    Dim strBuffer As String
    ' get file length
    lngLen = FileLen(FileRead)
    ' open files for reading and writing
    intFF = FreeFile
    Open FileRead For Binary Access Read As #intFF
    intFF2 = FreeFile
    Open FileSave For Binary Access Write As #intFF2
    ' initialize buffer
    strBuffer = Space$(CHUNKSIZE)
    ' loop while still more or as much data left to read
    Do While lngLen >= CHUNKSIZE
        ' read
        Get #intFF, , strBuffer
        ' Encrypt returns encrypted string (assuming a Public Function in a module)
        Put #intFF2, , Encrypt(strBuffer)
        ' decrease bytes left
        lngLen = lngLen - CHUNKSIZE
    Loop
    ' if still something left
    If lngLen > 0 Then
        ' decrease buffer size
        strBuffer = Left$(strBuffer, lngLen)
        ' read
        Get #intFF, , strBuffer
        ' encrypt and write
        Put #intFF2, , Encrypt(strBuffer)
    End If
    ' close files
    Close #intFF2
    Close #intFF
    ' we are done!

As you can see, it is an entire rewrite and it also assumes different kind of coding for Encrypt. Instead of modifying a public variable (which I assume is what you've done) it returns the encrypted string as a return value, being a function. This is the better way to do it, because public variables can be very confusing and hard to debug in the long run and especially as application grows.

[killo]

thanks alot that works perfectly. i tried encrypting a big file which was 68.1mb and it still worked ok, and didn't use more than 45,000k of memory. wheras the old program would try and load the whole 68.1mb into memory and the prgram would lock up aswell as the system. so thanks for helping me with this everyone

[Code Doc]

Merri's code is exactly what I had in mind when I posted Friday. I used to write code like this back in the DOS days for installation programs. As you know, string memory was almost a joke back then so we had to break almost all the files into chunks and then copy them one chunk at a time.

So, your encryption requirement for monster files is quite similar, and Merri's approach will work very well. You have to check for file size usning FileLen()and if the file is smaller than your pre-defined chunk then you encrypt the whole file in one pass. But, if the chunk is smaller than the file, then you process one chunk at a time and finish it off with the residual chunk.

[Merri]

As final notes, the biggest file you can easily handle in VB6 is 2 GB. To bring that limit up to 4 GB, you have to convert FileLen's return value into a Currency (in a way that signed Long values are handled unsigned). However, NTSF can hold files that are bigger than 4 GB: adding support for files that big requires API or external libraries.