Hopeless: memory not being released

[TupacShakur]

This is a hopeless case. I might as well finish writing an O.S before solving it, but here goes:

This problem is related to the topic of the other thread i posted few days ago, and titled: Huge Files/Memory Problem (Link).
That thread prompted me to change the whole structure of my program to adapt to the use of small (1 or 5 MB) buffers in reading from huge files (a 500 MB file for example), instead of the 100 MB buffer i was using at some point. Now the new way naturally has a longer loop (100 times x 5MB) instead of (5 times x 100MB).

In the old way, i was using a string buffer that i was freeing after each iteration by doing: StringVariable = "", thus freeing the RAM. At the end of the procedure, RAM (that went down by 200 MB's cause each byte is represented by 2 bytes in a String type variable) would return to normal.

In the new way (which should be much better), it seems that the long loop is at some point (1/3 of the way) freezing the memory freeing. Meaning that the StringVariable = "" is doing **** from some point on, thus getting the Ram down to 5 MB, and staying like that after the end of execution (although i am using the same variable as the buffer).

So what can i say??? HEEEEEEEEEEEEEEEEEELLLPPP!!!!!!!!!


P.S: I searched this forum, and it seems the the only way to free a string var (and what Microsoft has suggested doing) is the StringVar = "" statement.

[David.Poundall]

I think you are going to have to post some code. On the face of it everything sounds OK so it may well be a slipup that you are skiming over that someone else may be able to spot.

[sishe]

Did you try vbNullString ??

StringVariable = vbNullString

StringVariable = ""

only sets the variables value to zero length string, but the pointer remains. vbNullString sets the adress of the variable to zero, as a result frees the memory.

[TupacShakur]

Did you try vbNullString ??

Yea, did that, although setting the String to "" is the same thing. Anyway, did that before and now again, and same results

[sishe]

Can you post some code?

[TupacShakur]

hFile = FreeFile
    Open fileName For Binary Access Read As #hFile
        ' bla bla bla
        For i = 1 To chunkNum
            DoEvents
            If (i = chunkNum) Then
                tempChunkSize = lastChunkSize
            Else
                tempChunkSize = realChunkSize
            End If
            sliceFileName = desPath & slicePrefix & Format(i, formatStr) & "." & fileExt
            CreateBinaryFile sliceFileName, "", True
            aFile = FreeFile
            Open sliceFileName For Binary Access Write As #aFile
            Do While (tempChunkSize >= BUFFER)
                DoEvents
                dataChunk = Space$(BUFFER)
                Get #hFile, startByte, dataChunk
                startByte = startByte + BUFFER
                Put #aFile, LOF(aFile) + 1, dataChunk
                dataChunk = vbNullString
                tempChunkSize = tempChunkSize - BUFFER
                PB1.Value = PB1.Value + 1
            Loop
            If (tempChunkSize > 0) Then
                dataChunk = Space$(tempChunkSize)
                Get #hFile, startByte, dataChunk
                startByte = startByte + tempChunkSize
                Put #aFile, LOF(aFile) + 1, dataChunk
                dataChunk = vbNullString
                PB1.Value = PB1.Value + 1
            End If
            Close #aFile
        Next i
    Close #hFile

[MajorPaul]

I would suggest you start using a byte array instead.

• Using a string takes twice as much memory.
• Using a String to read binary data can be unreliable on unicode systems.

And perhaps a byte array will not have this problem of freeing up memory.

[Comintern]

If you are married to the idea of strings, you could try making dataChunk a fixed length string, and then copy it to a normal string and Trim$ if the length is less than BUFFER (should only occur on the last dataChunk of a file). My guess is that this would force VB's compiler to allocate memory simularly to how it would allocate for a byte array. Judging from your code, I would classify this as a bug in the compiler or runtime.

[CVMichael]

I looked at your code where you copy the data, and it looks too messy...

Try something like this:

Option Explicit
 
Private Sub CopyFile(ByVal ReadFile As String, ByVal WriteFile As String)
    Dim FL1 As Integer, FL2 As Integer, Buff As String
    Const BuffSize As Long = 65536 ' 64 KBytes
    
    FL1 = FreeFile
    Open ReadFile For Binary Access Read As FL1 ' open read file
    
    FL2 = FreeFile
    Open WriteFile For Binary Access Write As FL2 ' open write file
    
    Do Until Loc(FL1) >= LOF(FL1) ' do until end of file (EOF never actually worked for me)
        If Loc(FL1) + BuffSize >= LOF(FL1) Then
            Buff = String(LOF(FL1) - Loc(FL1), 0) ' read whatever is left
        Else
            Buff = String(BuffSize, 0) ' read buffer size
        End If
        
        Get FL1, , Buff ' read data
        Put FL2, , Buff ' write data
        
        ' show progress
        If LOF(FL1) > 0 Then Me.Caption = Format(Loc(FL1) / LOF(FL1) * 100#, "00.00") & "% Done"
        DoEvents ' if you don't put DoEvents, it won't update the form to see the progress
    Loop
    
    Close FL1, FL2 ' close files
End Sub
 
Private Sub Form_Load()
    Show
    CopyFile "read file.dat", "write file.dat"
End Sub

P.S. - I did not actually test this code...

[TupacShakur]

CVMichael:
Thanks for ur effort, but my function is not actually JUST copying the file. It's a bit complicated to explain it or to include the full code, but anyway, thanks again.


Comintern:
No, i'm still single ... Anyway, in order to change to byte arrays, i had to change a lot all through my program, but anyway, since it is better, i did it.
I can say that it's almost the same results, except that the memory freeing process is dying at some point deeper through the loop, hence i also deduce that this is a VB language problem.. Unless anyone else can suggest something new??


MajorPaul:
I did it, and it's slightly better, but almost the same ****. Thanks for ur suggestion.

[Merri]

Okay, I read you're combining a lot of files into one file. I also checked the code, according to that you're splitting the file here. One problem in your code is that you're reseting the used memory all the time, which is VERY inefficient and slow. But anyways, byte array is certainly the way to go. So, here goes my suggestion, since you haven't posted your updated code on which I could base it:

Dim FullBuffer() As Byte, SmallBuffer() As Byte
    'prepare buffer
    ReDim FullBuffer(BUFFER - 1)
    hFile = FreeFile
    Open FileName For Binary Access Read As #hFile
        For i = 1 To chunkNum
            'apparently tells us how much is still to be read...
            If (i = chunkNum) Then
                tempChunkSize = lastChunkSize
            Else
                tempChunkSize = realChunkSize
            End If
            'get new filename
            sliceFileName = desPath & slicePrefix & Format$(i, formatStr) & "." & fileExt
            aFile = FreeFile
            'open file for write
            Open sliceFileName For Binary Access Write As #aFile
                Do While (tempChunkSize >= BUFFER)
                    'read and write
                    Get #hFile, , FullBuffer
                    Put #aFile, , FullBuffer
                    'progress forward
                    tempChunkSize = tempChunkSize - BUFFER
                    PB1.Value = PB1.Value + 1
                Loop
                If (tempChunkSize > 0) Then
                    'preserving is faster and we are going to replace all data in it anyway
                    ReDim Preserve SmallBuffer(tempChunkSize - 1)
                    'read and write
                    Get #hFile, , SmallBuffer
                    Put #aFile, , SmallBuffer
                    'progress forward
                    PB1.Value = PB1.Value + 1
                End If
            Close #aFile
            'let windows refresh etc.
            DoEvents
        Next i
    Close #hFile
    'free memory
    Erase FullBuffer
    Erase SmallBuffer

Also make sure you use Option Explicit

[TupacShakur]

Merri:
Yes, you're right: in this code, i am splitting the file.
This is very similar to my updated code, although i am using only 1 buffer for both small and large buffers (redim preserve to the small buffer size after the while loop). I also tried ur suggested approach of erasing the byte array just once after the for loop, as i also thought that doing it a lot would be inefficient. My problem got worse then!!!! So in my updated code, i keep erasing and rediming, which, like i already said, seems to work fine up to some point in the loop, where all breaks down.

[Bill Crawley]

You should drop the use of using DOS based file I/O and make a reference to the microsoft Scripting Runtime. This is much more efficient and feature rich.

[Comintern]

One other thing to try would be to use a fixed size array. I'd bet that VB is leaking memory from the array allocations. Dynamic arrays have additional overhead in terms of memory allocations, and the redimensions may be the problem. BTW, the preserve keyword actually uses double the amount of memory because it actually copies the entire array into a new array in memory and then (in theory ) deallocates the previous one. If you think about it, it has to work this way to ensure that the new array gets a contiguous region in memory. There is no way of knowing if the additional addresses above the original allocation are free, so it plays it safe and just makes it bigger. You should be OK if you just avoid making memory operations inside your loop.

See if you can get something like this to work. I collapsed your loop a little bit.

Dim FullBuffer(BUFFER - 1) As Byte, j As Integer        'additional dims. Assumes BUFFER is a const.
    
    hFile = FreeFile
    
    Open Filename For Binary Access Read As #hFile
        For i = 1 To chunkNum
            'tells us how much is still to be read...
            If (i = chunkNum) Then
                tempChunkSize = lastChunkSize
            Else
                tempChunkSize = realChunkSize
            End If
            'get new filename
            sliceFileName = desPath & slicePrefix & Format$(i, formatStr) & "." & fileExt
            aFile = FreeFile
            'open file for write
            Open sliceFileName For Binary Access Write As #aFile
                Do While (tempChunkSize > 0)
                    'read and write
                    Get #hFile, , FullBuffer
                    If tempChunkSize >= BUFFER Then
                        Put #aFile, , FullBuffer        'write the whole thing.
                    Else
                        For j = 1 To tempChunkSize      'write each byte individually - only doing
                            Put #aFile, , FullBuffer(j) 'this once per file, so the overhead isn't
                        Next j                          'that much.
                    End If
                    'progress forward
                    tempChunkSize = tempChunkSize - BUFFER
                    PB1.value = PB1.value + 1
                    'Erase FullBuffer                   'try uncommenting this as a next step.
                Loop
            Close #aFile
            'let windows refresh etc.
            DoEvents
        Next i
    Close #hFile

[TupacShakur]

One other thing to try would be to use a fixed size array.
[...]
You should be OK if you just avoid making memory operations inside your loop.

People, forgive me for using this statement too much: ALREADY TRIED THAT ... It seems this is truely a hopeless case.

I'd bet that VB is leaking memory from the array allocations.

I'd bet VB is leaking memory from all over .

I collapsed your loop a little bit.

I tried this approach, but, as i expected, same problem is occuring. Thanks for ur help guys.

[TupacShakur]

You should drop the use of using DOS based file I/O and make a reference to the microsoft Scripting Runtime. This is much more efficient and feature rich.

How is that, i am not sure what are you talking about. Maybe u can help me a bit further. Thanks .

[CVMichael]

TupacShakur, Are you sure you have the latest VB6.0 Service Pack (SP6) ?

[Merri]

Can you post your complete code, this is beginning to sound like the code has other problems than the part of the code you're talking about. I guess like this because I've done a splitting program myself in the past that processed a bigger file. It was simple and efficient enough.

Comintern: but Preserve is still faster than without in many many cases, because without Preserve the ReDim must clear all bytes to zero. I've tested the speed difference of ReDim and have seen there really is difference in using Preserve.

[TupacShakur]

CVMichael: I'm not sure, how can i know which SP i got, and how much does it have to do with my problem?

Merri: Actually, i am pretty much sure that my code has no problems. The rest of my code is pretty much irrelevant, but i went through it all so many times, that i'm sure it's all fine. The problem seems to be in how VB allocates and deallocates memory, specially in a long loop (in my case). What environment did u test/use ur program in?

[dglienna]

I have faith in merri. I gave him the link to this thread. He can fix it.

[CVMichael]

CVMichael: I'm not sure, how can i know which SP i got, and how much does it have to do with my problem?

Memory leaks, sounds like a bug in VB, I remember I had the same problem in the past with de-allocating memory, and I fixed the problem when I installed the new Service Pack.

To check what Service Pack you have:
First, it shows in the spash screen when you start VB6.0
Second, if you click on "Help" then "About Microsoft Visual Basic...", you will see something like "Microsoft Visual Basic 6.0 (SP6)" where (SP#) is the number of your service pack.

Service Pack 6 is the latest...

If you don't have the latest, you can get it from microsoft.com searching for "visual basic service pack"
Actually, I looked it up, and here is the link:
http://www.microsoft.com/downloads/d...DisplayLang=en

[Merri]

i am pretty much sure that my code has no problems.

This made me laugh, just the other day I read "programmer's famous last words" and this line here is word-to-word perfect on the list

Anyways, here is the code I used. I seem to have no VB6 service pack installed according to the about box, though I do have the newest runtime files and maybe some other. About the code, all it has is a form with a single command button which can be clicked. The program then automatically splits a huge SQL file into pieces which can be used rebuild an SQL database a piece at a time to let server have some breath.

Option Explicit
 
Const Filename = "E:\tempfile"
Const vbMEGABYTE As Long = 2097152
 
Dim Quit As Boolean
Private Sub Command1_Click()
    Dim Filesize As Long, Counter As Long, A As Long
    Dim ReadBuffer() As Byte, TempBuffer() As Byte, EndPos As Long
 
    ReDim ReadBuffer(vbMEGABYTE - 1)
 
    Filesize = FileLen(Filename)
    
    Open Filename For Binary Access Read As #1
        Do While (Filesize - Seek(1)) \ vbMEGABYTE
            Counter = Counter + 1
            Open Filename & "_" & Format$(Counter, "00") & ".sql" For Binary Access Write As #2
                If Not ((Not TempBuffer) = True) Then Put #2, , TempBuffer
                Get #1, , ReadBuffer
                EndPos = InBArrRev(ReadBuffer, "INSERT INTO ")
                ReDim Preserve TempBuffer(vbMEGABYTE - EndPos - 1)
                For A = EndPos To vbMEGABYTE - 1
                    TempBuffer(A - EndPos) = ReadBuffer(A)
                Next A
                ReDim Preserve ReadBuffer(EndPos - 1)
                Put #2, , ReadBuffer
                ReDim Preserve ReadBuffer(vbMEGABYTE - 1)
            Close #2
            Command1.Caption = Format$((Seek(1) / Filesize * 100), "0.00") & " %"
            DoEvents
            If Quit = True Then Close #1: Unload Me
        Loop
        Counter = Counter + 1
        ReDim Preserve ReadBuffer(Filesize - Seek(1) - 1)
        Get #1, , ReadBuffer
        Open Filename & "_" & Format$(Counter, "00") & ".sql" For Binary Access Write As #2
            Put #2, , TempBuffer
            Put #2, , ReadBuffer
        Close #2
    Close #1
    Command1.Caption = "Done!"
End Sub
 
Private Sub Form_QueryUnload(Cancel As Integer, UnloadMode As Integer)
    Quit = True
End Sub

Then the helper function module:

Option Explicit

Public Function InBArrRev(ByRef ByteArray() As Byte, ByRef KeyWord As String, Optional StartPos As Long = -1, Optional Compare As Byte = vbBinaryCompare) As Long
    Static KeyBuffer() As Byte, KeyBufferU() As Byte, KeyPtr As Long, OldCompare As Byte
    Dim A As Long, B As Long, C As Long, KeyLen As Long, KeyUpper As Long
    Dim FirstKeyByte As Byte, LastKeyByte As Byte, TempByte As Byte
    Dim FirstKeyByteU As Byte, LastKeyByteU As Byte
    If KeyWord = vbNullString Then InBArrRev = -1: Exit Function
    KeyLen = StrPtr(KeyWord)
    If Not (KeyPtr = KeyLen And Compare = OldCompare) Then
        KeyPtr = KeyLen
        OldCompare = Compare
        If Compare = vbBinaryCompare Then
            KeyBuffer = KeyWord
        Else
            KeyBufferU = UCase$(KeyWord)
            KeyBuffer = LCase$(KeyWord)
        End If
    End If
    KeyLen = UBound(KeyBuffer) - 1
    KeyUpper = KeyLen \ 2
    If KeyUpper > UBound(ByteArray) Then InBArrRev = -1: Exit Function
    If StartPos < 0 Or StartPos > UBound(ByteArray) - KeyUpper Then StartPos = UBound(ByteArray) - KeyUpper
    FirstKeyByte = KeyBuffer(0)
    LastKeyByte = KeyBuffer(UBound(KeyBuffer) - 1)
    If Compare = vbBinaryCompare Then
        'loop through the array
        For A = StartPos To 0 Step -1
            If ByteArray(A) = FirstKeyByte Then
                If ByteArray(A + KeyUpper) = LastKeyByte Then
                    If KeyLen > 4 Then
                        'check if keyword is found from the array
                        C = A + 1
                        For B = 2 To KeyLen Step 2
                            If Not (ByteArray(C) = KeyBuffer(B)) Then Exit For
                            C = C + 1
                        Next B
                        'keyword is found!
                        If B > KeyLen Then
                            InBArrRev = A
                            Exit Function
                        End If
                    Else
                        InBArrRev = A
                        Exit Function
                    End If
                End If
            End If
        Next A
    Else 'vbTextCompare
        FirstKeyByteU = KeyBufferU(0)
        LastKeyByteU = KeyBufferU(UBound(KeyBuffer) - 1)
        'loop through the array
        For A = StartPos To 0 Step -1
            TempByte = ByteArray(A)
            If TempByte = FirstKeyByte Or TempByte = FirstKeyByteU Then
                TempByte = ByteArray(A + KeyUpper)
                If TempByte = LastKeyByte Or TempByte = LastKeyByteU Then
                    If KeyLen > 4 Then
                        'check if keyword is found from the array
                        C = A + 1
                        For B = 2 To KeyLen Step 2
                            TempByte = ByteArray(C)
                            If Not (TempByte = KeyBuffer(B) Or TempByte = KeyBufferU(B)) Then Exit For
                            C = C + 1
                        Next B
                        'keyword is found!
                        If B > KeyLen Then
                            InBArrRev = A
                            Exit Function
                        End If
                    Else
                        InBArrRev = A
                        Exit Function
                    End If
                End If
            End If
        Next A
    End If
    InBArrRev = -1
End Function

I never meant this code to be seen by others so it is uncommented. The file I split was over 600 MB in size.

[TupacShakur]

To check what Service Pack you have:
First, it shows in the spash screen when you start VB6.0
Second, if you click on "Help" then "About Microsoft Visual Basic...", you will see something like "Microsoft Visual Basic 6.0 (SP6)" where (SP#) is the number of your service pack.

Actually, i checked the splash screen and the about dialog before asking where to find the SP version, but they doent say anything in my case. Anyway, thanks for the suggestion and the link, i will download the SP6 later, although i am now sure what the problem is, and it's not anyhow SP related.

This made me laugh, just the other day I read "programmer's famous last words" and this line here is word-to-word perfect on the list

I totally agree with you. But not in this case. I mean this is a rather (up to some level) simple, or lets say not so complicated, code/program, that one can be sure that it has no problems, specially after fixing many many bugs in it, and after restructuring/rewriting it many times. It certainly had problems, and i changed almost the whole code many times before claiming that it has no problems, so... definitely not my last words, i've only been in this programming game for 2 years, so i'm just getting started.


Anyway, i will post shortly again since i came to some rather interesting facts about this problem. Still conducting some tests now.

[TupacShakur]

Let me 1st say that i am running Win ME on my desktop PC. I was testing my program on a +500 MB file. At some point, while copying some files, and when observing the progress bar (the copy window in the Win OS's), it occured to me that Win copies files the same way like my code, in rather small buffers, in a loop. So i started copying all the files i was testing on from 1 location to another. Not so surprisingly, i got the same results i was getting when running my code: for example, copying the 500 MB file got my RAM down from 325 to 10 MB!!!
My next step was to test my program and the copying process on my Laptop running Win XP Home Edition - SP1. Also, the results were similar when copying or using my program. But this time, both results were extremely positive and perfect!!!!! I also tested on a 670 MB video file, using 1 MB buffer in my program, thus maximising the loop: A PERFECT RESULT (speed wise and memory wise)!! I then tested again on my cousin's desktop PC running Win XP, again perfect results..

Hence, i can basically say that the problem seems to be Windows related, and how Windows does something that i cant figure what it is since we're talking about compiled executables that presumably have machine level instructions only, and no API calls whatsoever in my code. Strange i guess...
Anyone can comment on this??

[Merri]

Have you tried taking out the in-most DoEvents? Now that I remember it, on Win9X you might get weird errors when you call DoEvents too long too often. The same might apply to opening and closing files, I'm not sure... it has been a long time I coded under a 9X.

[TupacShakur]

^^^
Yeah, i did do that, and also tried all possible placements for the DoEvents call, but nothing really had any effect on the result. Anyway, as i said, the problem seems to be not related to the code (maybe i wasnt clear enough, when i mentioned file copying, i meant in the browser, and not thru VB!!)...

How strange of win9x

[dglienna]

Try adding more Buffers in W98. That might help things.

[TupacShakur]

How and Where? (I am using Win ME)