[RESOLVED] Clearing array is fast in VB5 but very slow in VB6

[Karl77]

Hi there,

I have a large array of class objects to clear and release memory.

For i = LBound(prp_GTable) To UBound(prp_GTable) - 1
  ReDim Preserve prp_GTable(LBound(prp_GTable) To UBound(prp_GTable) - 1)
Next

GTable is a small class.

The above was originally coded in VB5.
Of course it 'works' in VB6 the same.

But it is ultra slow compared to when it runs under VB5, estimated factor 1000.
I can't imagine what the error could be, as the code is unaltered.

I did a little research, ZeroMemory sounds perfect instead of the loop.
But ZeroMemory doesn't accept GTable - type mismatch.

1)
What could be the cause for acting so much slower in VB6?
This should be a known problem...

2)
In which way could I hand over GTable to ZeroMemory?


Thanks,
Karl

[techgnome]

There must be more going on, because that code by itself makes no sense. It doesn't alter the array in any way... so what is it supposed to be doing?

-tg

[Karl77]

It doesn't alter the array in any way... so what is it supposed to be doing?

No, the array gets shrinked one by one.
This way it releases memory.
While erase doesn't.
And takes very long as well.

[DataMiser]

That code is not well written at all. It is using a loop and redim preserve which means it is running multiple times.
For example if you have 100 items in the array then you are making 100 copies of it.
There is no reason I can see to use a loop there, just the statement that is within the loop will do the same thing only lots faster.
If as above there are 100 items in the array then using the loop makes it 100 times slower possibly more if any paging kicks in.

As a general rule always avoid using redim preserve in a loop and never call it for every instance. Much better to have the array larger than needed than to go through it for every item with preserve on.

Just to be clear when you use redim preserve you are making a new array and coping the contents of the old array into the new one then dropping the old array so the more entries in the array the longer it takes.

Also if you are actually trying to "clear" the array then there is no reason at all to use preserve and there is no reason to decrease its size by one. If you just want to remove the last item it may be better to just clear out that item rather than redim the array but that depends on how the array is being used and may require other changes to work.

Edit: On second look it looks like you are removing the items 1 by one, but again there is no reason to do it that way. If you want to clear the array do not use preserve at all. and do not reduce its size over and over and over when a single call will do the job.

[techgnome]

That's pretty much what I was thinking... why bother with the preserve... just redim it down to a single index (0) and be done with it... don't use preserve and don't use the loop.

-tg

[Karl77]

Hmm.
The good thing is, the snippet was not invented by me.
The bad thing is, I don't fully understand what's going on.

Did some tests:

For i = LBound(prp_GTable) To UBound(prp_GTable) - 1
  ReDim Preserve prp_GTable(LBound(prp_GTable) To UBound(prp_GTable) - 1)
Next

120 sec.
Same code in VB5 2 sec.

Erase GTable()

118 sec.

For i = LBound(prp_GTable) To UBound(prp_GTable) - 1
  Set prp_GTable(i) = Nothing
Next

118 sec.

While I know Preserve is a performance killer, seems it is not so in this case.

Karl

[Karl77]

ReDim prp_GTable(0)

121 sec.

[Karl77]

Also interesting:
http://www.vbforums.com/showthread.p...-Zero-Fast-way

Olaf tested and, of course, found out ZeroMemory is the fastest.
Only in my case I can't use it for my array.
Or?

[Elroy]

EDIT4: Well crap, staring at my post, I noticed that I put 100 in for the first test and 1000 in for the others. I've corrected that now.

What I see that you're doing is reducing the array to one element (the first one). Now, you've stated that this is an array of class objects.

I see three ways to do it, or possibly only two. It depends on whether or not you need to save certain settings/properties in the single object that you're keeping. If that's the case, you can't just use Erase and then re-instantiate a single object. However, for testing purposes, I'll assume that that's okay.

To test, I just started a new project, and then put a single Class1 into it (with no code). And then, I put the following code into the Form_Load event. Also, just as an FYI, I changed the order of the three methods around, and that didn't seem to make any difference.

Option Explicit
'
Private Declare Function PerformCount Lib "kernel32" Alias "QueryPerformanceCounter" (lpPerformanceCount As Any) As Long
Private Declare Function PerformFreq Lib "kernel32" Alias "QueryPerformanceCounter" (lpPerformanceCount As Any) As Long
'

Private Sub Form_Load()
    Dim timeFreq As Currency
    Dim startTime As Currency
    Dim endTime As Currency
    Dim i As Long
    Dim a() As Variant

    PerformFreq timeFreq


    ReDim a(1 To 1000)
    For i = LBound(a) To UBound(a)
        Set a(i) = New Class1
    Next i
    '
    PerformCount startTime
    ReDim Preserve a(LBound(a) To LBound(a))
    PerformCount endTime
    MsgBox "Nanoseconds: " & Format$((endTime - startTime) / timeFreq * 1000000000, "0.000")



    ReDim a(1 To 1000)
    For i = LBound(a) To UBound(a)
        Set a(i) = New Class1
    Next i
    '
    PerformCount startTime
    For i = LBound(a) To UBound(a) - 1
      ReDim Preserve a(LBound(a) To UBound(a) - 1)
    Next
    PerformCount endTime
    MsgBox "Nanoseconds: " & Format$((endTime - startTime) / timeFreq * 1000000000, "0.000")



    ReDim a(1 To 1000)
    For i = LBound(a) To UBound(a)
        Set a(i) = New Class1
    Next i
    '
    PerformCount startTime
    Erase a
    ReDim a(1 To 1)
    Set a(1) = New Class1
    PerformCount endTime
    MsgBox "Nanoseconds: " & Format$((endTime - startTime) / timeFreq * 1000000000, "0.000")



    Unload Me
End Sub

And then, the following was my output:

Redim Preserve (preserving only the first element, all one statement, no loop):


Redim Preserve using original loop in original #1 post:


Using Erase and then Redim, Set (to create a new first object):


As per EDIT4, we can now see that there are no huge differences.

Redim Preserve to a single object is the way to go.

Regards,
Elroy

EDIT1: I bumped it up to 10000 (from 100) elements. Doing this, I got 53, 82, 51 nanoseconds. Just a single Redim Preserve, as opposed to the loop was still faster, but not by the order of magnitude with only 100 elements. Also, under these conditions, the Erase approach was neck-and-neck with the single Redim Preserve approach, with the loop still being slowest.

EDIT2: Running it with 100000 elements, I got 1667, 1925, 1660.

EDIT3: Also, since the jump from 10000 to 100000 was exactly a factor 10, but the timings were up by a factor of around 30, I'm thinking that Windows paging (virtual memory) is kicking in. Karl77, if you're running on a machine with a smallish amount of memory, that could explain a great deal of it. These days, I would call smallish in the 2GB range.

[Schmidt]

It's not an "array-problem" we have here (for which all the mentioned speedup-recommendations would apply),
but instead we have a "how to destroy large amounts of VB6-ClassInstances fast" problem ...
(which makes all the different Array-freeing-techniques irrelevant and only a "spectator on the sidelines").

Olaf

[Elroy]

Olaf, I certainly tend to agree. In fact, that was my first thought when I saw this thread. However, the Redim Preserve in a loop did bother me.

reexre, maybe you could outline what this thing is for. So many people say that we should abandon UDT structures in favor of Class modules. However, this may be a case where the opposite is true, that is, if a UDT will get it done.

Regards,
Elroy

[The trick]

When you change size or destroy an array the objects items that are being destroyed performs Class_Terminate event. Runtime sequentially calls obj->Release method for each instance. Do you have any code in Terminate event?

[Karl77]

Do you have any code in Terminate event?

No, nothing.

[The trick]

No, nothing.

Please show your class code, maybe it contains dependencies to other classes that have Terminate event.

[Karl77]

Option Explicit
Dim prp_GCode As String
    ' Information about the gcode line
Dim prp_File As Boolean '= False   ' Is line from file or MDI?
Dim prp_LineNum As Long
Dim prp_Status As String
Dim prp_Sent As Boolean '= False    ' Was the line sent?
Dim prp_Acked As Boolean '= False    ' Was the line Ack'd?
Dim prp_M0 As Boolean '= False    ' Does this line contain M0 Pause?
Dim prp_M6 As Boolean '= False     ' Does this line contain M06 Tool Change?

Public Property Get GCode() As String
    GCode = prp_GCode
End Property

Public Property Let GCode(ByVal Value As String)
    prp_GCode = Value
End Property

Public Property Get File() As Boolean
    File = prp_File
End Property

Public Property Let File(ByVal Value As Boolean)
    prp_File = Value
End Property

Public Property Get LineNum() As Long
    LineNum = prp_LineNum
End Property

Public Property Let LineNum(ByVal Value As Long)
    prp_LineNum = Value
End Property

Public Property Get Status() As String
    Status = prp_Status
End Property

Public Property Let Status(ByVal Value As String)
    prp_Status = Value
End Property
Public Property Get Sent() As Boolean
    Sent = prp_Sent
End Property

Public Property Let Sent(ByVal Value As Boolean)
    prp_Sent = Value
End Property

Public Property Get Acked() As Boolean
    Acked = prp_Acked
End Property

Public Property Let Acked(ByVal Value As Boolean)
    prp_Acked = Value
End Property


Private Sub Class_Initialize()
    prp_File = False
    prp_Sent = False
    prp_Acked = False
    prp_M0 = False
    prp_M6 = False
    
End Sub

Private Sub Class_Terminate()

End Sub

[Karl77]

we have a "how to destroy large amounts of VB6-ClassInstances fast" problem ...

Yes.
And I wonder why VB5 is better in this respect...

[Arnoutdv]

There seems to be no methods, there is only data storage.
So such a basic class could easily be implemented as an UDT.

[Karl77]

So such a basic class could easily be implemented as an UDT.

The class is in an ActiveX DLL.
I can declare the UDT in there, but can't access it from the main program.

"Use classes not UDTs" has it's advantages.

[The trick]

I can declare the UDT in there, but can't access it from the main program.

You can.

[Karl77]

You can.

But I'm too dense.
How can I?

[Arnoutdv]

Put your declarations in a class of the ActiveX DLL project.
The "instancing" property of this class has to be set to GlobalMultiUse.

[The trick]

But I'm too dense.
How can I?

You can create UDT as in the TLB (even through Remote Server Files) as in the ActiveX Dll public class. You should add the public type to an class.
Then you can use UDT both in an array and in a collection.

Option Explicit

Public Sub Main()
    Dim arr()   As PubType
    Dim col     As Collection
    
    ' // Using array access
    ReDim arr(1000000)
    
    arr(993).prp_GCode = "String member"
    
    Erase arr()
    
    ' // Using collection
    Dim index   As Long
    
    Set col = New Collection
    
    For index = 0 To 1000000
        col.Add PubType_New
    Next
    
    Set col = Nothing
    
End Sub

Private Function PubType_New() As PubType
    PubType_New.prp_GCode = "String member"
End Function

[Karl77]

Arnoutdv, The trick:

Thank you, that should help me to get started to solve the problem.
Especially the sample project group.

Isn't it interesting that VB5 is so much faster when it comes to destroy classes?
I wonder what is different in VB6...

[DEXWERX]

Possibly more overhead in the base Class of VB6. I know they improved on Implements handling in VB6, as well as thread safety.

Is it still faster in VB5 to go through the Array and Set each instance = Nothing, before Erase() 'ing the Array?
This explicitly Release's each Class instance.


How long does this take in VB5?

For i = LBound(prp_GTable) To UBound(prp_GTable) - 1
  Set prp_GTable(i) = Nothing
Next
Erase prp_GTable

[techgnome]

I don't think VB5 actually destroyed anything... I think that's what the difference is. Can't prove it. Don't really have the inclination to do so...(plus there's better people that could prove/disprove the theory) but that's my guess. VB5 was simply releasing the references w/o actually cleaning up. VB6 on the other hand does the responsible thing and does the cleanup.

-tg

[Karl77]

I don't think VB5 actually destroyed anything... I think that's what the difference is. Can't prove it. Don't really have the inclination to do so...(plus there's better people that could prove/disprove the theory) but that's my guess. VB5 was simply releasing the references w/o actually cleaning up. VB6 on the other hand does the responsible thing and does the cleanup.

You are right, to now understand why VB5 seems faster is worthless effort.
And I can't prove it either, I only have the exe compiled under VB5.

It seems VB5 didn't only release the reference, as it freed memory.
The original coder said to me Erase didn't release the memory.

I agree finding out the 'why' is not worth it.

[Karl77]

How long does this take in VB5?

For i = LBound(prp_GTable) To UBound(prp_GTable) - 1
  Set prp_GTable(i) = Nothing
Next
Erase prp_GTable

Can't test it in VB5.
Don't have it installed, and can't find the original CD anymore.

As said, it is not worth to investigate anyway.

[techgnome]

You are right, to now understand why VB5 seems faster is worthless effort.
And I can't prove it either, I only have the exe compiled under VB5.

It seems VB5 didn't only release the reference, as it freed memory.
The original coder said to me Erase didn't release the memory.

I agree finding out the 'why' is not worth it.

Oh, I don't know that... it might be interesting to see what's going on underneath the covers... my curiosity is piqued... I just don't have the means to go investigate it.

Can't test it in VB5.
Don't have it installed, and can't find the original CD anymore.

As said, it is not worth to investigate anyway.

??? If you don't have VB5 anymore then how'd you come up with these numbers?

Hmm.
The good thing is, the snippet was not invented by me.
The bad thing is, I don't fully understand what's going on.

Did some tests:

For i = LBound(prp_GTable) To UBound(prp_GTable) - 1
  ReDim Preserve prp_GTable(LBound(prp_GTable) To UBound(prp_GTable) - 1)
Next

120 sec.
Same code in VB5 2 sec.

Erase GTable()

118 sec.

For i = LBound(prp_GTable) To UBound(prp_GTable) - 1
  Set prp_GTable(i) = Nothing
Next

118 sec.

While I know Preserve is a performance killer, seems it is not so in this case.

Karl

-tg

[DEXWERX]

Yeah my hunch is that VB5 freed the Array Memory without Release'ing references. But can't say without testing

[Karl77]

??? If you don't have VB5 anymore then how'd you come up with these numbers?

The VB5 number(s) are told me from the co-worker.
Also I can test the VB5 executable and see the difference.

In fact it is only one number I have written for VB5: The green 2 sec. vs. 120 sec above.
All the others are VB6 numbers.

[Karl77]

You can create UDT as in the TLB (even through Remote Server Files) as in the ActiveX Dll public class. You should add the public type to an class.
Then you can use UDT both in an array and in a collection.

The problem is solved.
I changed to UDT instead of class in the real app.
Works like a charm - of course.

So much I can learn here, very good.

Thank you very much for the example!

Karl

[DEXWERX]

The problem is solved.
I changed to UDT instead of class in the real app.
Works like a charm - of course.

So much I can learn here, very good.

Thank you very much for the example!

Karl

As far as speed and overhead is concerned, a UDT is more equivalent to the Data Structures in other languages.

[Elroy]

Awww boooo. I recommended the UDT first (post #11), and Karl77 didn't see it.

Also, Karl77, if all of this is contained within your project (and not using any ActiveX or other referenced API calls with the UDT), you can just declare the UDT as Public in a standard (BAS) module, make sure all your form and class uses of it are in methods declared as Friend, and you're good to go.

Regards,
Elroy

[Schmidt]

Yeah my hunch is that VB5 freed the Array Memory without Release'ing references. But can't say without testing

IMO VB5 was/is (deep-)freeing the Class-instances when Arrays were "shortened" or Erased
(cannot test/compare this anymore to VB6) - but when VB5 was destroying these Instances,
it did so in a different (faster) way (compared to what VB6 does now).

As for how VB6 does it currently - there was a recent thread in the old MS-NewsGroup,
where I dug a bit deeper into VB6-Class-Instantiation/Freeing - and I'm quite sure that
what makes larger amounts of VB6-Classe-Instances that slow performance-wise is in all likelihood:
The slow de-registration from IConnectionPoint-Containers...

Because VB6-ClassInstances are registering themselves at such a Container, so that
Class_Initialize and Class_Terminate could be thrown not over a direct (and fast) Implementation-
Callback (which could be the reason why VB5 was performing better, in case it used these direct callbacks),
but instead VB6 is doing it now over the wellknown, but slow(er) COM-Eventmechanism.

What became also apparent (and is probably related to the COM-Event-Callback-Registration) is,
that VB6-ClassInstances are definitely entered into - and part of a double-linked-list (instead of
being "free-standing ones").

Here's a Demo for an approach I came up with, which significantly speeds up VB6-Class-Creation
(but is especially a booster, when it comes to the Destruction of larger amounts):
http://vbRichClient.com/Downloads/Fa...teDestroy3.zip


Alternatives to the Zipped Demo above are:
- to use UDTs instead (already the solution to Karls problem apparently)
- to use true lightweight COM-Classes/instances over a *.bas-module implementation

Olaf

[Karl77]

Hi Elroy,

Awww boooo. I recommended the UDT first (post #11), and Karl77 didn't see it.

No no, I saw it.
At that time I thought the UDT can't be used because of the ActiveX DLL.
Now I know better (from #21/22 on).

Also, Karl77, if all of this is contained within your project (and not using any ActiveX or other referenced API calls with the UDT), you can just declare the UDT as Public in a standard (BAS) module, make sure all your form and class uses of it are in methods declared as Friend, and you're good to go.

That's not new, that's natural.

Read again :"ActiveX DLL".
And see #19.
And perhaps try the sample project in #22.



Karl

[Karl77]

Hello Olaf,

wow, that was too interesting for you...
...impressive.

I studied your demo, also tried with a million objects - yes, it is really fast.
What you all found out and realized is far beyond my mind.
I won't use it, too complicated for me, so I prefer the easy solution (UDT).

Karl

[Elroy]

Ahhh, got it. Glad you got it worked out.

[DEXWERX]

@Olaf - you're saying it's COM Events that are slowing down Class Instantiation and Destruction.
Are you saying that VB6 Initialize and Terminate are Events (registered via IConnectionPoint) where VB5 they were direct callbacks?
OR are you saying that it's the overhead of setting up Class objects to use events that are slowing VB6 down?

To summarize your example - you sidestep the run-time's Allocation and Freeing, of Class Objects. By side stepping the run-time, you lose the ability for the classes to use Events (pros being speed, con's being the loss of asynchronous callbacks which no ones uses anyway, and you have to cleanup your objects and references manually).

[Schmidt]

@Olaf - you're saying it's COM Events that are slowing down Class Instantiation and Destruction.

Well, not the Events themselves, but their involved registration (and especially de-registration) from
an apparently "central instance" (which is perhaps related to the Instances-MemAllocator or other controlling-
or cleanup-implementations, perhaps useful when running in Debug-Mode in the IDE).

Are you saying that VB6 Initialize and Terminate are Events (registered via IConnectionPoint) where VB5 they were direct callbacks?

I guess so (if the information is right, that VB5 generated instances don't have such a teardown-performance-problem).
Don't have a VB5 anymore here for verification.

OR are you saying that it's the overhead of setting up Class objects to use events that are slowing VB6 down?

Yep, but "setting them up for Events" in the widest sense.
What I definitely know is, that any VB6-ClassInstance (of the same type) which was created per 'New':
- is layed out as an "outer" aggregating Instance (which wraps an inner "BaseClass-Instance" which consumes additional memory)
- is part of a linked List (with at least a back-pointing instance-pointer to the instance created before, directly after the VTable-pointer)

I assume, that it is the "Remove"-operation from that linked List, which is the main-culprit
for the increasingly bad teardown-performance, the more instances are members of that linked-list.

I suspect that this list is related to IConnectionPoint(Container)-stuff, to give a central (EventServer-)instance
a chance, to observe any concrete Object-destruction in that central place (which I imagine can
be useful, whilst running in the IDE) - and that in truly compiled code, this mechanism is not deactivated ("left as it is in IDE-mode").

Not removing the mechanism at (native)compile-time might be an oversight, might be laziness, but might
as well be deliberate (to set a clear "performance-barrier" for the main-competitor at that time -> VC6++).

Also this "outer intance-allocation" which is part of any VB6-ClassInstance, does consume
about half of the 120Bytes a "naked VB6-ClassInstance needs" in total (if that mechanism would be
left out at native compile-time, we would have only 64Byte per "naked VB6-Class-Instance").

Here's what I found out so far about the memory-layout: (by simple MemCopy-checks on the ObjPtr, not by disassembly)

Public Type VBClassHeader '64Bytes for a naked Class -> 16 32Bit Members
  pVTable As Long 
  pPreviousInstanceSameType As Long '<- here's the most interesting one 
  pUnkInstance As Long 'always denoting with 28Bytes Offs to our own ObjInstance-Ptr 
  pInstanceBaseClass As Long 'sidewards-allocation (extra memory) 
  pInstanceIConnectionPoint As Long 'sidewards-allocation (extra memory)
     YetToFindOut1 As Long 'usually at Zero 
     YetToFindOut2 As Long 'usually at Zero 
  pUnkVTable As Long 'the 7'th member after pVTable (= our 28Bytes Offs) 
  lRefCount As Long 
  lDataSourceBehaviourFlag As Long 
     YetToFindOut3 As Long 'usually at Zero 
  StateFlag As Long 'usually at &H100F, but at &H1C6E when terminating 
     YetToFindOut4 As Long 'usually at Zero 
'-> ...Class-private Vars will be inserted here, shifting the IClassModuleEvt-vTable down
  pVTableIClassModuleEvt As Long 
     YetToFindOut5 As Long 'usually at Zero 
     YetToFindOut6 As Long 'usually at Zero 
End Type

As said, the second member (pPreviousInstanceSameType) is the interesting one with regards to the teardowns.

To summarize your example - you sidestep the run-time's Allocation and Freeing, of Class Objects. By side stepping the run-time, you lose the ability for the classes to use Events (pros being speed, con's being the loss of asynchronous callbacks which no ones uses anyway, and you have to cleanup your objects and references manually).

Yes, that's it basically (although there's nothing asychronous about COM-Event-callbacks).

Olaf

[DEXWERX]

Yes, that's it basically (although there's nothing asychronous about COM-Event-callbacks).

Olaf

ah.. good point. COM-Event-Callbacks are not Async, anymore than any other form of callback.

Still curious about why VB5 was faster. What/Why was setup/teardown changed from VB5. Thanks for the insight.
This could all be fixed if MS would Open Source VB.