[VB6] Problem Using CopyMemory API to Get Collection Keys

[jbarnett74]

I'm maintaining a VB6 application that contains a function for getting collection keys from a collection, into an array. It calls the CopyMemory API. For the most part it works, with one exception. The last item in the collection is corrupt when the function exits. In most cases, a collection of strings are passed into the function and the last item contains a bunch of unrecognizable characters. However, I tracked down the problem after a collection of objects were passed in and VB would crash when accessing the final item.

Here's the code that is used. Could anyone tell me what I may be able to do differently to keep from corrupting the last item in the collection? Are there any cleanup practices that must be followed with the CopyMemory API or pointers?

Declare Sub CopyMemory Lib "kernel32.dll" Alias "RtlMoveMemory" _
            (hpvDest As Any, hpvSource As Any, ByVal cbCopy As Long)
Declare Function lstrlenW Lib "kernel32" _
            (ByVal lpString As Any) As Long

Public Type CollectionData
    Unk0(0 To 2) As Long
    Unk1 As Long
    ElementCount As Long
    Unk2 As Long
    PtrFirstElement As Long
    PtrLastElement As Long
    Unk3 As Long
    Unk4 As Long
    Unk5 As Long
End Type

Public Type CollectionElement
    Data As Variant
    Key As Long
    PtrPrev As Long
    PtrNext As Long
    Unk0 As Long
    Unk1 As Long
    Unk2 As Long
End Type

Public Function GetKeys(ByVal oCollection As Collection) As String()
    Dim asKeys() As String
    Dim lIdx As Long
    Dim tColl As CollectionData
    Dim tColElem As CollectionElement
    Dim sKey As String
    
    If oCollection.Count > 0 Then
        ReDim asKeys(0 To oCollection.Count - 1)
        CopyMemory tColl, ByVal ObjPtr(oCollection), LenB(tColl)
        tColElem.PtrNext = tColl.PtrFirstElement
        For lIdx = 0 To tColl.ElementCount - 1
            CopyMemory tColElem, ByVal tColElem.PtrNext, LenB(tColElem)
            
            sKey = String$(lstrlenW(tColElem.Key), 0)
            CopyMemory ByVal StrPtr(sKey), ByVal tColElem.Key, lstrlenW(tColElem.Key) * 2
            
            asKeys(lIdx) = sKey
        Next
    End If
    
    GetKeys = asKeys
End Function

[LaVolpe]

This is the line I suspect is causing the problem:
CopyMemory tColElem, ByVal tColElem.PtrNext, LenB(tColElem)

If tColElem contains data referenced by pointer, then that line basically creates an unreferenced copy of the data. Let's say it's string data in the tColElem.Data item. When your function exits, whatever is in that element is destroyed and when later referenced as part of the collection or collection is destroyed - crash.

You'll want to zero out the tColElem when the loop ends. Can do this with an array of LenB(tColElem) or FillMemory API.

Dim tNullData() as Byte
ReDim tNullData(1 to Lenb(tColElem))
For ...

Next ...
CopyMemory tColElem, tNullData(1), UBound(tNullData)
...

[Bonnie West]

You'll want to zero out the tColElem when the loop ends. Can do this with an array of LenB(tColElem) or FillMemory API.

Here are a couple of alternative methods:

CopyMemory ByVal VarPtr(tColElem), ByVal String$(LenB(tColElem), 0), LenB(tColElem)


Private Declare Sub ZeroMemory Lib "kernel32.dll" Alias "RtlZeroMemory" (ByRef Destination As Any, ByVal Length As Long)

ZeroMemory ByVal VarPtr(tColElem), LenB(tColElem)

BTW, the Key member of the CollectionElement UDT is actually a BSTR (there's a length field and null terminator just before & after the pointed to string data), so the following lines are completely unnecessary:

Declare Function lstrlenW Lib "kernel32" (ByVal lpString As Any) As Long

Dim sKey As String

sKey = String$(lstrlenW(tColElem.Key), 0)
CopyMemory ByVal StrPtr(sKey), ByVal tColElem.Key, lstrlenW(tColElem.Key) * 2

The assignment to the string array can therefore be simplified to:

CopyMemory ByVal VarPtr(tColElem), ByVal tColElem.PtrNext, LenB(tColElem)
asKeys(lIdx) = tColElem.Key

after changing the CollectionElement.Key's data type from Long to String.

[Schmidt]

The information about these VBA.Collection-related structs which jbarnett74 posted at the top,
were new to me, so if somebody (Bonnie?) has more information about the origination of these structs,
who has used/posted them first in a VB (or C++ context), would be nice.

It's quite interesting, since they allow a simple wrapping of the VB-Collection in a decent manner
(which avoids most of its pitfalls, and could have been implemented this way by the MS-devs
who wrote the VBA-collection, already in the first place).

Have put an implementation (which incorporates these Structs) for a "better VB-Col" now into the Codebank:
http://www.vbforums.com/showthread.p...Indexed-Access

Olaf

[LaVolpe]

FYI: First time I've seen the collection structure exposed was nearly 10 years ago. Most samples you find on the net (including one in our CodeBank by Merri), navigate the collection, but not necessarily describing the structure they are navigation

[Schmidt]

FYI: First time I've seen the collection structure exposed was nearly 10 years ago. Most samples you find on the net (including one in our CodeBank by Merri), navigate the collection, but not necessarily describing the structure they are navigation

Earliest I could find (but also only "Poking around wildly", not using any UDTs as shown by jbarnett74)
was from 2007 (by Ulli, on PSC):
http://www.planetsourcecode.com/vb/s...68075&lngWId=1

Perhaps jbarnett74 has some more comments about these Structure-Defs (from the
Author of the App he has now to maintain)?

Olaf

[jbarnett74]

Thanks everyone! I took LaVolpe's suggestion to zero out the tColElem when the loop ends. I also refactored some code based on Bonnie's suggestions. However, I don't think I implemented Bonnie's suggestion correctly, because it seemed to strip the keys from the collection.

As for credit to whomever came up with the structure declarations, I have no idea. I'm glad my question benefited others and wish I could give credit where it's due, but there are no comments in the code I'm working with to explain where it came from. I imagine the person who implemented the code in my application copied it from elsewhere, because the code style was inconsistent with the majority of the application.

I'm ok with the extra step to extract the key, so here's the working solution that I've implemented:

Private Declare Sub CopyMemory Lib "kernel32.dll" Alias "RtlMoveMemory" (hpvDest As Any, hpvSource As Any, ByVal cbCopy As Long)
Private Declare Function lstrlenW Lib "kernel32" (ByVal lpString As Any) As Long

Private Type CollectionData
    Unk0(0 To 2) As Long
    Unk1 As Long
    ElementCount As Long
    Unk2 As Long
    PtrFirstElement As Long
    PtrLastElement As Long
    Unk3 As Long
    Unk4 As Long
    Unk5 As Long
End Type

Private Type CollectionElement
    Data As Variant
    Key As String
    PtrPrev As Long
    PtrNext As Long
    Unk0 As Long
    Unk1 As Long
    Unk2 As Long
End Type

Public Function GetKeys(ByVal oCollection As Collection) As String()
    Dim asKeys() As String
    Dim lIdx As Long
    Dim tColl As CollectionData
    Dim tColElem As CollectionElement
    Dim tNulldata() As Byte
    Dim sKey As String
    
    If oCollection.Count > 0 Then
        ReDim tNulldata(1 To LenB(tColElem))
        ReDim asKeys(0 To oCollection.Count - 1)
        CopyMemory tColl, ByVal ObjPtr(oCollection), LenB(tColl)
        tColElem.PtrNext = tColl.PtrFirstElement
        For lIdx = 0 To tColl.ElementCount - 1
            CopyMemory tColElem, ByVal tColElem.PtrNext, LenB(tColElem)

            sKey = String$(lstrlenW(tColElem.Key), 0)
            CopyMemory ByVal StrPtr(sKey), ByVal tColElem.Key, lstrlenW(tColElem.Key) * 2

            asKeys(lIdx) = sKey
        Next
        CopyMemory tColElem, tNulldata(1), UBound(tNulldata)
    End If
    
    GetKeys = asKeys
End Function

[jbarnett74]

I did find something... the following link points to a solution very close to what was implemented in my application. The thread dates back to 2003.

http://www.44342.com/visual-basic-f948-t34789-p1.htm

[Bonnie West]

However, I don't think I implemented Bonnie's suggestion correctly, because it seemed to strip the keys from the collection.

CopyMemory tColElem, ByVal tColElem.PtrNext, LenB(tColElem)

Yep, you missed the ByVal VarPtr() part:

CopyMemory ByVal VarPtr(tColElem), ByVal tColElem.PtrNext, LenB(tColElem)

I did find something... the following link points to a solution very close to what was implemented in my application. The thread dates back to 2003.

http://www.44342.com/visual-basic-f948-t34789-p1.htm

Thanks for digging up that link! The guy there who provided a glimpse of the Collection's internal structure might very well be the same "dude" being referred to here.

There seems to be a problem in that site's newsgroup archive copy, though. For some unknown reason, a few sentences in the discussion there are truncated. Googling around, the archive versions here and here appears to be more faithful to the original thread.

If you have no more issues pertaining to this thread, you can now mark it Resolved! Thank you.

[dilettante]

Looks interesting and potentially useful. Once you go through the process it all looks simple enough that we'll probably each tend to hack out our own renditions. Mine is basically the same as other code above with some personal choices included.

I think my Key() As String function will prove far more useful and error-free than my Keys() As String() function. I say that mainly because Collections are rarely unchanging over the life of a program so caching Keys() and using it could result in hard to track down program bugs.

For that matter it is still faster to use error trapping instead of implementing a "Does Key Exist In Collection?" function by grabbing an array of keys and searching it. Well, except for very small Collections.


As for where the critical information came from...

Well it is possible that somebody worked it out by dumping memory and some trial and error. But what I suspect is that some time in the distant past some Microsoft Partner was fed the information by Microsoft, and this somehow managed to leak. The leak could have been somebody who happened to see some Partner-developed code, or maybe some Partner employee leaked it.

I doubt it was ever meant to reach the masses. The Declare signatures for entrypoints in various VB runtimes probably leaked in a similar manner.

[Jonney]

Looks interesting and potentially useful. Once you go through the process it all looks simple enough that we'll probably each tend to hack out our own renditions. Mine is basically the same as other code above with some personal choices included.

I think my Key() As String function will prove far more useful and error-free than my Keys() As String() function. I say that mainly because Collections are rarely unchanging over the life of a program so caching Keys() and using it could result in hard to track down program bugs.

For that matter it is still faster to use error trapping instead of implementing a "Does Key Exist In Collection?" function by grabbing an array of keys and searching it. Well, except for very small Collections.


As for where the critical information came from...

Well it is possible that somebody worked it out by dumping memory and some trial and error. But what I suspect is that some time in the distant past some Microsoft Partner was fed the information by Microsoft, and this somehow managed to leak. The leak could have been somebody who happened to see some Partner-developed code, or maybe some Partner employee leaked it.

I doubt it was ever meant to reach the masses. The Declare signatures for entrypoints in various VB runtimes probably leaked in a similar manner.

Could you please put the attachment to CodeBank for easy reference? Thank you,Sir.

[dilettante]

Could you please put the attachment to CodeBank for easy reference? Thank you,Sir.

I'm not sure that my implementation is any better than those the others posting here have written. Perhaps one of them would like to enter their own version as a CodeBank thread.

[dilettante]

This turns out to be a lot less useful than I imagined it might be. I went and looked at some smaller projects where I had been using small value classes to be stored into Collections.

Even when the only thing I defined in the class were a Value and a Name (String), I was concerned about case-insensitivity. I wanted a case-insensitive lookup by Key (Name), but I needed to be able to retrieve the "canonical" Name with proper casing.

For example a code editor handling a VB-like language (in my case it was VBScript).

I wanted the most recently loaded, edited, or entered "definition statement" to determine the case of the identifiers, just as in the VB6 IDE's code editing windows. For example if a script had:

Dim somethingSpecial

... and then I edited that to be:

Dim SomethingSpecial

... the new "SomethingSpecial" needed to become the stored symbol, as well as then being updated throughout the script to match the new casing. Removing and then re-Adding the item isn't especially efficient either.


So in the end I still needed a value class anyway. Fetching the Key based on Index really isn't as useful as one might be tempted to imagine, and for that matter you often need to store more than a single value per item anyway.

Maybe Microsoft knew what they were doing all along.

[Schmidt]

... the new "SomethingSpecial" needed to become the stored symbol, as well as then being updated throughout the script to match the new casing. Removing and then re-Adding the item isn't especially efficient either.

Why not?
It should work sufficiently fast even on a VB-Collection with a few thousand entries,
to "remove and re-add" such a "ReDimmed/Redefined" SymbolName - even whilst
the User is typing it.

Maybe Microsoft knew what they were doing all along.

Well, in case of the VB-Collection they definitely did not.
Looks more like "deliberate crippling" (or if one wants to put it mildly: "a job half-done"),
when you consider, what the structure as it currently is, would have allowed with only
a few lines of C-Code.

Thought that I demonstrated what's possible with the given structure with an
appropriate CodeBank-entry already (link to it was contained in my post #4).

Being able to enumerate both, Keys and Values - by index with "array-like speed"
(as shown in the CodeBank-entry), would surely have been a feature which would've
made the VB-Collection far more useful over the last decades in a whole lot of scenarios.

Olaf

[dilettante]

I see now that you did post something to the CodeBank. I must have missed that, thinking it was something else.