Persisting data in vb6 classes/objects

[axisdj]

Hello Group,

Hope everyone is having a great week.

trying to get perspective on what is the best way to save and load data into/out of my objects in my program. What I am doing now is creating structs with the same properties/variables as my classes and then saving/loading them from a binary file.

Is there a more efficient way to do this?

I looked at using this: http://www.vb-helper.com/howto_vb6_serialize.html, but then my program would have external config file and xml file for all my new changes/objects that need to be saved. My only concern with XML file is that my class design also become apparent. Also I feel the struct method above seem in-efficient.

Any suggestion would be welcome.

Thank you!

[Niya]

Binary is the most efficient way to save data. It is very fast and very compact. However, the disadvantage is that it is not extensible. What this means is that if you have to add, remove or reorder fields, any binary saves from the previous version will be incompatible so you have to throw it away. You can use reserved fields to allow for the addition of new fields in future version without breaking the format but this is a stopgap measure at best.

XML is far less efficient but quite easy to extend. If you add new fields to an XML format, you can still read older saves correctly since XML is very descriptive and nodes are not dependent upon their position within the file like binary.

There is no right way to approach to this. It depends on what is important to you. Both ways have advantages and disadvantages.

[jpbro]

You can use the PropertyBag object to serialize/deserialize your object properties.

For example (this is air code so there might be bugs, it's just to give you an idea):

Public Function Serialize() As Byte
   Dim lo_Pb as PropertyBag

   Set lo_Pb = New PropertyBag
   With lo_Pb
      .WriteProperty "SomeProperty", SomeValueVariable
      .WriteProperty "SomeOtherProperty", SomeOtherValueVariable
   End With

   Serialzie = lo_Pb.Contents
End Function

Public Sub Deserialize(pa_Contents() As Byte)
   Dim lo_Pb As PropertyBag

   Set lo_Pb = New PropertyBag
   lo_Pb.Contents = pa_Contents 

   SomeValueVariable = lo_Pb.ReadProperty("SomeProperty")
   SomeOtherValueVariable = lo_Pb.ReadProperty("SomeOtherProperty")
End Sub

The nice thing about the PropertyBag approach is that you can later add different properties to extend it, and include default values as a third parameter in case you are reading old serialized files that don't have one or more of your new extended properties.

You can also encrypt the byte array on write, and decrypt it on read if you are worried about tampering.

[LaVolpe]

If not using a property bag approach, give this some serious forethought. Likelihood that changes will occur down the road, you'll want to add new stuff to be saved, no longer use some stuff, change the datatype of something (say from string to date format), etc. To help with this, you may want to include version information in one of the 'fields' you are saving. When your app reads the file, it looks at the version and 2 things can happen at that time: 1) it knows what version the data is in and 2) whether it can process it or not. If the data is in a greater version than your current app is written for, it can't reliably read the data and should inform the user they are using an outdated version of your app or they need to use a previous version of the data format. Versioning also allows you to save data to a previous format

[jpbro]

To extrapolate on LaVolpe's point, this might also be a good reason to use a file database format like Sqlite. You can create a table/tables to hold all your serialized values (and have a version number for the entire database, or on a per-table basis depending on your needs). When upgrading from one version to another, you can put your upgrade logic in a transaction, so if you have any bugs in your update logic, it is trivial to rollback.

A database also gives you some interesting possibilities re: selecting subsets of data to do partial deserializations (if you think that kind of think might ever be necessary).

Databases are also relatively easy to modify by hand if necessary (using the appropriate GUI or command line tool).

An Sqlite wrapper like Olaf Schmidt's vbRichClient5 also let's you work against an encrypted database file if you are worried about snoopers.

[axisdj]

You can use the PropertyBag object to serialize/deserialize your object properties.

For example (this is air code so there might be bugs, it's just to give you an idea):

Public Function Serialize() As Byte
   Dim lo_Pb as PropertyBag

   Set lo_Pb = New PropertyBag
   With lo_Pb
      .WriteProperty "SomeProperty", SomeValueVariable
      .WriteProperty "SomeOtherProperty", SomeOtherValueVariable
   End With

   Serialzie = lo_Pb.Contents
End Function

Public Sub Deserialize(pa_Contents() As Byte)
   Dim lo_Pb As PropertyBag

   Set lo_Pb = New PropertyBag
   lo_Pb.Contents = pa_Contents 

   SomeValueVariable = lo_Pb.ReadProperty("SomeProperty")
   SomeOtherValueVariable = lo_Pb.ReadProperty("SomeOtherProperty")
End Sub

The nice thing about the PropertyBag approach is that you can later add different properties to extend it, and include default values as a third parameter in case you are reading old serialized files that don't have one or more of your new extended properties.

You can also encrypt the byte array on write, and decrypt it on read if you are worried about tampering.

Wow, I find new things everyday.. love the propertybag idea and it looks like you can serialize a complete class with almost no effort. https://msdn.microsoft.com/en-us/lib...(v=vs.60).aspx

Thanks for all the replies!

WP

[Schmidt]

And to expand on jpbro's suggestion further...

These days I'd serialize my Classes to JSON (especially when long-time-storage is planned),
to be more flexible with regards to providing also WebClients (Browsers) from that Base-Data
in the future (without having to write converters and stuff).

Here's a small Demo, which shows how you can make a Class JSON-serializable -
how to create and handle a list of such Class-Instances - later storing such a
"Class-List" in an InMemory-DB - finally saving the InMemory-DB into a real
DB-File on Disk (in encrypted SQLite-format).

Start a new, empty VB-Project and put a reference to vbRichClient5 into it:

Into a Project-Private Class, named: cSerializableClass

Option Explicit

Public Properties As cCollection

Private Sub Class_Initialize()
  Set Properties = New_c.JSONObject
  With Properties 'define default-values ... not each and every Class-Property needs to be set, ...
    .Prop("DateValue") = Now '.. here we init only the DateValue-Prop to a current DateTime-Stamp
  End With
End Sub
 
Public Property Get ID() As Long
  ID = Properties("ID")
End Property

Public Property Get Name() As String
  Name = Properties("Name")
End Property
Public Property Let Name(RHS As String)
  Properties.Prop("Name") = RHS
End Property
 
Public Property Get LongValue() As Long
  LongValue = Properties("LongValue")
End Property
Public Property Let LongValue(ByVal RHS As Long)
  Properties.Prop("LongValue") = RHS
End Property

Public Property Get DoubleValue() As Double
  DoubleValue = Properties("DoubleValue")
End Property
Public Property Let DoubleValue(ByVal RHS As Double)
  Properties.Prop("DoubleValue") = RHS
End Property

Public Property Get DateValue() As Date
  DateValue = Properties("DateValue")
End Property
Public Property Let DateValue(ByVal RHS As Date)
  Properties.Prop("DateValue") = RHS
End Property

And this here into a Form (Form1)

Option Explicit

Private MemDB As cMemDB

Private Sub Form_Load()
  Caption = "RC5-based serialization-scenario"
  SimpleTest_SingleInstance 'just to show the basic principle in action
End Sub

Private Sub Form_Click()
Const InstanceCount As Long = 1000
Dim ClassList As cCollection

  New_c.Timing True
    Set ClassList = Performance_ClassListCreate(InstanceCount)
  Print "Performance_ClassListCreate:" & New_c.Timing; " for "; ClassList.Count; " entries"; vbLf
  
  New_c.Timing True
    Performance_ClassListWrite ClassList
  Print "Performance_ClassListWrite:" & New_c.Timing; " for "; MemDB.GetCount("ClassList"); " entries"; vbLf
  
  'small test, selecting (and deserializing) a single Element out of the DB
  Dim Rs As cRecordset, Instance As cSerializableClass
  Set Rs = MemDB.GetRs("Select JSONContent From ClassList Where ID=123") 'select a single record with the ID 123
  Set Instance = New cSerializableClass
  Set Instance.Properties = New_c.JSONDecodeToCollection(Rs!JSONContent.Value)
  Print "ReadOut of Instance-Properties, after retrieval from the DB (under ID=123)"
  Print Instance.ID, Instance.Name, Instance.DoubleValue, Instance.DateValue; vbLf 'the date should be 123 days later than now
  
  'and here we finally write the InMemory-DB out to an encrypted DB-File (into the App.Path)
  WriteMemDBContentToDisk "c:\temp\MyClassSerializations.db3", "MyEncryptionKey"
End Sub
 
Private Sub SimpleTest_SingleInstance()
  With New cSerializableClass
    Debug.Print "First Readout: "; .Name, .LongValue, .DoubleValue, .DateValue
    
      .Name = "Name " & 1
      .LongValue = .LongValue + 1
      .DoubleValue = .DoubleValue + 1
      .DateValue = .DateValue + 1
    
    Debug.Print "Second Readout: "; .Name, .LongValue, .DoubleValue, .DateValue

    Debug.Print .Properties.SerializeToJSONString 'just to show, how the serialization-format ends up in the DB later on
  End With
End Sub

Private Function Performance_ClassListCreate(ByVal InstanceCount As Long) As cCollection
Dim i As Long, NewInstance As cSerializableClass, D As Date
    Set Performance_ClassListCreate = New_c.Collection(False)
    D = Now 'will be incremented "by a day" in each round
    For i = 1 To InstanceCount
      Set NewInstance = New cSerializableClass
          NewInstance.Name = "Name " & i
          NewInstance.LongValue = i
          NewInstance.DoubleValue = i + i / 10
          NewInstance.DateValue = D + i
        
      NewInstance.Properties.Prop("ID") = i
      Performance_ClassListCreate.Add NewInstance, i
    Next
End Function

Private Sub Performance_ClassListWrite(ClassList As cCollection)
  Set MemDB = New_c.MemDB 
      MemDB.Exec "Create Table ClassList(ID Integer, JSONContent Text)"
 
  MemDB.BeginTrans
    Dim Instance As cSerializableClass
    For Each Instance In ClassList
      MemDB.ExecCmd "Insert Into ClassList Values(?,?)", Instance.ID, Instance.Properties.SerializeToJSONString
    Next
  MemDB.CommitTrans
End Sub

Private Sub WriteMemDBContentToDisk(DBName As String, Optional EncrKey As String)
  If New_c.FSO.FileExists(DBName) Then New_c.FSO.DeleteFile DBName
  MemDB.Cnn.CopyDatabase DBName, EncrKey
  Print "encrypted SQLite-FileDB successfully saved under: "; DBName
End Sub

Here's a ScreenShot, what the Demo does and produces:



Olaf

[7edm]

Hi Olaf,

I was thinking that technique could also be used to serialize and persist object related values that only change during design time and stays unchanged during runtime but is loaded into a collection for use? I am working on a resizer control that currently stores anchor values in the controls .Tag property, but while practical in a sense it's not a good solution as it require extra workaround code to make the .Tag property available at run time. Also, there is no guaranty every control will have a .Tag property, I think. I setup this stuff in a Property Page. And I hope @axisdj doesn't have anything against that I broaden the subject of the topic a bit.

[Schmidt]

I was thinking that technique could also be used to serialize and persist object related values that only change during design time and stays unchanged during runtime but is loaded into a collection for use?

Sure, and when it's only a few .Left/.Top/.Width/.Height Values you have to store
(at Design-Time) - then you'd not need to wrap these Values as Properties behind
a dedicated Class - just working with a single cCollection (in JSON-mode) would be
enough, I think.

Here's a (very simplified) small example, to show what I mean:

Into a Form, which will host an instance of a few simple Controls: Command1, Text1, Check1,
and will look like e.g.:



...the following code needs to be pasted (depending on a Reference to vbRichClient5):

Option Explicit

Private CurZoom As Single, CtlCoords As cCollection

Private Sub Form_Load()
  CurZoom = 1
  ScaleMode = vbPixels
End Sub

Private Sub Form_Click()
Dim strJSON As String
    strJSON = SerializeControlsCoords()
  Debug.Print strJSON 'this is, how the serialized Data will look like
  
  '...now persist it somewhere - maybe even as a "single value in a PropertyBag", to spare yourself fiddling with the VB-Resource-Editor
  
  'now the deserialization from the (later "from somewhere" retrieved) JSON-String
  Set CtlCoords = New_c.JSONDecodeToCollection(strJSON)
  CurZoom = CtlCoords("Zoom") 'apply the stored, original Zoom-Factor
 
  ResizeControlsFromStoredCoords 'and move the Controls accordingly
End Sub

Private Sub ResizeControlsFromStoredCoords()
  Dim Ctl As Control
  For Each Ctl In Controls
    With CtlCoords.Prop(Ctl.Name)
      Ctl.Move .Prop("Left") * CurZoom, .Prop("Top") * CurZoom, .Prop("Width") * CurZoom, .Prop("Height") * CurZoom
    End With
  Next
End Sub

Private Function SerializeControlsCoords() As String
Dim Store As cCollection
  Set Store = New_c.JSONObject
      Store.Prop("Zoom") = CurZoom
      
  Dim Ctl As Control
  For Each Ctl In Controls
    Store.Prop(Ctl.Name) = AddControlPropertiesTo(New_c.JSONObject, Ctl)
  Next
  SerializeControlsCoords = Store.SerializeToJSONString
End Function

Private Function AddControlPropertiesTo(CtlStore As cCollection, Ctl As Control) As cCollection
    CtlStore.Prop("Left") = Ctl.Left / CurZoom
    CtlStore.Prop("Top") = Ctl.Top / CurZoom
    CtlStore.Prop("Width") = Ctl.Width / CurZoom
    CtlStore.Prop("Height") = Ctl.Height / CurZoom
  
  Set AddControlPropertiesTo = CtlStore
End Function

Olaf

[7edm]

Thanks for that example Olaf,

that code describes the process greatly and will help a lot even if the values I want to save isn't control coordinates but anchor information for each control. These are set on a property page during design time and then used at runtime when the form is resizing so the controls resize depending on how they are anchored. I found some old code on PSC serving as a good base but there are space for improvement. What I want to achieve here is to serialize the anchor configuration values for all controls and then at runtime read it strait into a cCollection object, using the ctrl.Name (+ index if part of control array) as key when stepping trough a For Each loop resizing. When done, maybe I can add it to the Code bank here.

[7edm]

Oh, just a question regarding the the .Prop method of the cCollection, is it equal in function to .Add and/or is it only for use when in JSON mode? Otherwise, can it be used to update an object slot already added to the collection (not in JSON mode)?

[Schmidt]

Oh, just a question regarding the the .Prop method of the cCollection, is it equal in function to .Add and/or is it only for use when in JSON mode? Otherwise, can it be used to update an object slot already added to the collection (not in JSON mode)?

.Prop in Write-direction ensures an "AddOrReplace" (an entry with the given Key) -
so I guess, the answer is: "Yes in both cases".

Olaf

[7edm]

I hope OP doesn't mind I continue to develop this topic even... @Olaf, I am having problems with cCollection and JSonObject. In my resizing UserControl I have this code:

'Declaration section
Option Explicit
...
Private mAnchors As vbRichClient5.cCollection
..........................................
Private Sub InitControls()

Dim sKey As String
Dim ctrl As Control
Dim cAnchor As clsAnchorSpec
Dim sJSON As String
Dim aProp As PropertyBag
...
   Set mAnchors = New_c.JSONObject

   For Each ctrl In Extender.Parent.Controls

      If IsSupportedControl(ctrl) Then
         Set cAnchor = New clsAnchorSpec
         sKey = GetCtrlNameWithIndex(ctrl)
         mAnchors.Prop(sKey) = cAnchor
      End If

   Next

   sJSON = mAnchors.SerializeToJSONString
   Set aProp = New PropertyBag
   aProp.WriteProperty "Anchors", sJSON, ""
...
End Sub

The problem is that although

Debug.Print mAnchors.Count

returns 19

sJSON = mAnchors.SerializeToJSONString

raises an Exception 13 : Type mismatch

The Sub is running at design-time when my UC is sited on the form. Is that the problem, although it seems strange as the collection as such seem to work fine and have swallowed 19 items added to it? I'm at loss because as I understand it execution of UC code during design-time basically works as code at run-time in the IDE but with a more limited scope maybe. I cannot see why SerializeToJSONString shouldn't work here though?

[Schmidt]

'Declaration section

      ...
      If IsSupportedControl(ctrl) Then
         Set cAnchor = New clsAnchorSpec
         sKey = GetCtrlNameWithIndex(ctrl)
         mAnchors.Prop(sKey) = cAnchor
      End If
      ...

I cannot see why SerializeToJSONString shouldn't work here though?

mAnchors (as a JSON-enabled cCollection) will allow only other
(JSON-array, or JSON-object flagged) cCollection in its Object-Members.

So, in the case above (I guess you have an internal "JSON-object-flagged"
cCollection inside clsAnchorSpec?) you would have to make a ReadOnly-
Property (or Friend-Property) within clsAnchorSpec to hand out that JSON-Collection,
and then add that instead - as e.g.:

      ...
      If IsSupportedControl(ctrl) Then
         Set cAnchor = New clsAnchorSpec
         sKey = GetCtrlNameWithIndex(ctrl)
         mAnchors.Prop(sKey) = cAnchor.InternalJSONCol
      End If
      ...

Then it should work.

Olaf

[7edm]

mAnchors (as a JSON-enabled cCollection) will allow only other
(JSON-array, or JSON-object flagged) cCollection in its Object-Members.

Hmm that wasn't clear to me, is this documented somewhere?

So, in the case above (I guess you have an internal "JSON-object-flagged"
cCollection inside clsAnchorSpec?) you would have to make a ReadOnly-
Property (or Friend-Property) within clsAnchorSpec to hand out that JSON-Collection,
and then add that instead - as e.g.:

      ...
      If IsSupportedControl(ctrl) Then
         Set cAnchor = New clsAnchorSpec
         sKey = GetCtrlNameWithIndex(ctrl)
         mAnchors.Prop(sKey) = cAnchor.InternalJSONCol
      End If
      ...

Then it should work.

Olaf

Not sure I understand, but the use case is this:
clsAnchorSpec is a normal data class with 4 properties Left, Top, Right and Bottom of type Long - although used as boolean flags for True/False as 1 and 0, where 1 means Anchored and 0 not.

The collection then keeps 1 instance for each (resizeable) control with the control's Name property as key. The whole purpose of using JSON was to be able to serialize the collection (and its object members) and then save it to the PropertyBag. But maybe it's not possible this way. My aim is to create a resizer control that can be fully configurated during design time and just have to bother about resizing at run-time.

Do you mean that instaed of a data class with 4 properties, I would make it a JSon array (what is that?) or JSONObject collection instead or add to the class a JSON collection that stores the property values? I am sorry, I don't know much about JSON more than it's a format to structure data as with XML but in a very different way and more storage effective I believe.

In short, I just want to have a collection that during design-time is added to and holds configuration data for each control and that this data, which may change due to reconfiguration (Property page), is persistence between design-time instances and finally is there at run-time. At run-time there are no changes to the collection - at least at this time I don't plan for dynamically added controls.

To use the controls Tag property is a hacky way to do it, but I was looking for a cleaner solution as with the tag you have to take into consideration it may needs/wants to be used for other things.

[7edm]

Ok maybe the dime has sunken in now and I think I have come up with a solution, with this code

   Set mAnchors = New_c.JSONObject

   For Each ctrl In Extender.Parent.Controls

      If IsSupportedControl(ctrl) Then
         Set cAnchor = New_c.JSONObject
         cAnchor.Prop("Left") = 1
         cAnchor.Prop("Top") = 1
         cAnchor.Prop("Right") = 0
         cAnchor.Prop("Bottom") = 0
         sKey = GetCtrlNameWithIndex(ctrl)
         mAnchors.Prop(sKey) = cAnchor
      End If

   Next

But, and I'm a bit unsure of this, think I could set all these 4 values in one go like

{"Left":1,"Top":1,"Right":0,"Bottom":0}

if I have understood JSON correctly. The question then is how I do this and if I plainly can simplify it to

   Set mAnchors = New_c.JSONObject

   For Each ctrl In Extender.Parent.Controls

      If IsSupportedControl(ctrl) Then
         sKey = GetCtrlNameWithIndex(ctrl)
        mAnchors.Prop(sKey) = New_c.JSONDecodeToCollection("{""Left"":1,""Top"":1,""Right"":0,""Bottom"":0}")
      End If

   Next

and the content of mAnchors would be the same with this code, so I later can take out the object from mAnchors, make a value change and put it back?

[Schmidt]

mAnchors (as a JSON-enabled cCollection) will allow only other
(JSON-array, or JSON-object flagged) cCollections in its Object-Members.

Hmm that wasn't clear to me, is this documented somewhere?

Yes, it follows (indirectly) from the JSON-standard (and the allowed DataTypes).
http://en.wikipedia.org/wiki/JSON#Da...ax_and_example

There's simple "Value-Types" (String, Number, Boolean and Null) - but also
a kind of "List-Types" (which are used to build hierarchies, where the Value
comes as a JSON-array or a JSON-object) - these "ListType-Members" then
usually sit behind Keys, that contain an 's'-Suffix (e.g. "Comstomers" - or -
as in your case - "Anchors").

In the "JSON-string-representation", those "ListKind-Values" start with
either a "[" (a JSON-array follows) ... or a "{" (a JSON-Object follows).

In a JSON-objectaccess-representation (e.g. after decoding a JSON-string
into a cCollection) - you will find these "ListKind-Members" represented
by "just another cCollection" (and that's how the Parent-Child-Hierarchies are
represented generically - always using a cCollection, when it goes "one level deeper").

... the use case is this:
clsAnchorSpec is a normal data class with 4 properties Left, Top, Right and Bottom of type Long - although used as boolean flags for True/False as 1 and 0, where 1 means Anchored and 0 not.

The collection then keeps 1 instance for each (resizeable) control with the control's Name property as key.

For such simple purposes, I'd not introduce a dedicated Class as your: clsAnchorSpec -
instead you could access the Anchors for each different CtlKey behind a Function which returns
a cCollection.

Here's all the code you would need, to make use of such an "Anchor-By-Key"-property
(including proper serializing of all Anchors in the UserControls PropertyBag per JSON).

Into a fresh UserControl:

Option Explicit
 
Private mAnchors As vbRichClient5.cCollection

Private Sub UserControl_Initialize()
  Set mAnchors = New_c.JSONObject
End Sub

Private Property Get Anchor(Key As String) As cCollection
  If Not mAnchors.Exists(Key) Then mAnchors.Prop(Key) = CreateNewAnchorObj
  Set Anchor = mAnchors(Key)
End Property

Private Function CreateNewAnchorObj() As cCollection
  Set CreateNewAnchorObj = New_c.JSONObject 'create a new Anchor (with Default-Values)
      CreateNewAnchorObj.Prop("Left") = 0
      CreateNewAnchorObj.Prop("Top") = 0
      CreateNewAnchorObj.Prop("Right") = 0
      CreateNewAnchorObj.Prop("Bottom") = 0
End Function

'the events below will automatically read or write all the Anchor-Objects from the Controls PropertyBag
Private Sub UserControl_ReadProperties(PropBag As PropertyBag)
  Set mAnchors = New_c.JSONDecodeToCollection(CStr(PropBag.ReadProperty("Anchors", "{}")))
End Sub
Private Sub UserControl_WriteProperties(PropBag As PropertyBag)
  PropBag.WriteProperty "Anchors", mAnchors.SerializeToJSONString
End Sub

With the above code in place, you can Read and Write the current Anchor-Props for a given
Control-Key at any time this way (e.g. for Control with Key: "ctlFoo"):

  MsgBox Anchor("ctlFoo")("Left")  'read-direction
  Anchor("ctlFoo")("Left") = 1  'write-direction

If you do the Write-direction at DesignTime (whilst showing your PropertyPage-Dialogue),
the serialization into your UserControls internal PropertyBag should happen automatically.

Olaf

[7edm]

Great Olaf! Or should I say "Olaf the Great" ;-) I think this now have fully solved the challenge for this part of my control, although there are other challenges still. The code above gives a great demonstration in how to work with the cCollection as a JSON object although I found my "shortcut" with mAnchors.Prop(sKey) = New_c.JSONDecodeToCollection(JSON_OBJ_STRING) also work. However, 2 further short questions

with your example

Private Sub UserControl_WriteProperties(PropBag As PropertyBag)
  PropBag.WriteProperty "Anchors", mAnchors.SerializeToJSONString
End Sub

IF mAnchors happens to be Nothing (or is empty of items), will "mAnchors.SerializeToJSONString" then produce a "{}" string, just as the default value in the UserControl_ReadProperties Sub?

Also, I would like to ask you, if the fastest way to iterate through a cCollection to get the key values is like:

   For I = 0 To mAnchors.Count - 1
      sKey = mAnchors.KeyByIndex(I)
      ....
   Next

or

  For I = mAnchors.Count - 1 To 0 Step -1
      sKey = mAnchors.KeyByIndex(I)
      ...
  Next

Or there is an even better way?

Again, thanks. It's really great how you share of your knowledge.

/Joakim

[LaVolpe]

Per MSDN... Tip: For better performance, use For Each to iterate over the items in a Collection object. For Each is significantly faster than iterating with the index.

[7edm]

Per MSDN... Tip: For better performance, use For Each to iterate over the items in a Collection object. For Each is significantly faster than iterating with the index.

Yes I know that, BUT, afaik it doesn't work if you want to get the Key values instead of the collections content items. Also, if you iterate through a coll to remove items based on the key value, I think you need to start from col.count -1 and step down to 0

[Schmidt]

IF mAnchors happens to be Nothing (or is empty of items), will "mAnchors.SerializeToJSONString" then produce a "{}" string, just as the default value in the UserControl_ReadProperties Sub?

The code I've posted in #17 will ensure, that the mAnchors-Variable will never be Nothing
(as long as the hosting UserControl lives).

As for the result of a fresh instantiated "JSON-cCollection", why not simply check that -
e.g. with an additional Debug.Print in:

Private Sub UserControl_Initialize()
  Set mAnchors = New_c.JSONObject
  Debug.Print mAnchors.SerializeToJSONString
End Sub

Which then prints out:
{}

Also, I would like to ask you, if the fastest way to iterate through a cCollection to get the key values is like:

   For I = 0 To mAnchors.Count - 1
      sKey = mAnchors.KeyByIndex(I)
      ....
   Next

or

  
  For I = mAnchors.Count - 1 To 0 Step -1
      sKey = mAnchors.KeyByIndex(I)
      ...
  Next

Or there is an even better way?

No, both ways are equally fast (the first loop enumerating the Keys in "Add-Order",
the second one in "inverse Add-Order").

@LaVolpe
the RC5-cCollection is a completely different animal from the VB.Collection ...well, it can also be used in a
VB.Collection compatible mode - but has none of the disadvantages of the VB.Collection... e.g. it is:
- much faster in any regards (Adding, Removing, KeyedAccess to stored Items)
- it always maintains two indexes (one for enumeration in Add-Order, one for enumeration in "KeySorted-Order")
- enumerations by Index (no matter if for Items or Keys, sorted or not) are equally fast to For Each

Olaf

[7edm]

The code I've posted in #17 will ensure, that the mAnchors-Variable will never be Nothing
(as long as the hosting UserControl lives).

As for the result of a fresh instantiated "JSON-cCollection", why not simply check that -
e.g. with an additional Debug.Print in:

Private Sub UserControl_Initialize()
  Set mAnchors = New_c.JSONObject
  Debug.Print mAnchors.SerializeToJSONString
End Sub

Which then prints out:
{}

Blimey! Pure obstruction of curiosity on my behalf and I pledge myself guilty of laziness. And thanks for that code.

No, both ways are equally fast (the first loop enumerating the Keys in "Add-Order",
the second one in "inverse Add-Order").

Ok, maybe should have had figured that your stuff is optimized in both directions. But basically what I was after is if this is the "only" and fastest way to iterate and dig out the keys, assuming For Each couldn't work here in any? Something like:

For Each item in ColOfItems
  Debug.Print = item.key
Next

Would be a nice feature, but it may introduce too much complexity inwardly or are my eyes missing something here?

There is also "KeyBySortKeyIndex" but I am not really sure what it means, exactly. Is it an sequential numeric index ordinal to the keys (alphabetically) sorted order? That's basically what I read out of the name at least.

But would it make the iteration faster in the cases above? Also, would it work internally the same way when it comes to removal of items? In one (clean up) function I iterate through the collection, starting from the high index, and check if the control that the key mimic still exists and if not remove the item. However, I am not sure exactly what's happening here internally, weather using KeyBySortKeyIndex or KeyByIndex, if removing an item triggers an automatic resort of the index or reindexing may be the proper word?

/Joakim

[Schmidt]

For Each item in ColOfItems
  Debug.Print = item.key
Next

Would be a nice feature, but it may introduce too much complexity inwardly or are my eyes missing something here?

That's impossible to implement, since the Item which is enumerated there,
is "an Object" - in case it's instances of your own Classes, you can
always place a Key-Property inside this Class and set it at runtime,
shortly before adding it to the Collection.

E.g. when working with the VB-Collection this "extra-effort" was the only
way to make the Key available in an enmueration of the Items.

Using the cCollection (in conjunction with an integer Loop-Variable),
makes these extra-efforts (to implement a special Key-Prop) obsolete,
since you can always enumerate both (the Key and the Item-values)
in the same loop.
For Each looks nice "on paper", but makes performance-wise no difference -
and it has the disadvantage, that you can only enumerate either the Items,
or the Keys... - though in many scenarios you will need both...

There is also "KeyBySortKeyIndex" but I am not really sure what it means, exactly. Is it an sequential numeric index ordinal to the keys (alphabetically) sorted order? That's basically what I read out of the name at least.

The name hints at the SortIndex of the cCollection (and this index is updated
in the Add-Method of the cCollection, without "resorting the whole bunch of
already existing entries").

This is done using Binary-Comparisons on the already sorted existing Content
(to find the new place for the new Keyed-Item really fast) - and then an
"optimized Insert" is done, to persist the new Entry in memory (not using
a Double-Linked-List, instead larger "Skip-List-like"-chunks are moved when
necessary, which is not always the case).

In case you plan to sort a List of Strings, this "sorting whilst adding" beats
most dedicated (array-based) Sorting-algorithms (in case you care to make a test).

Well, below is a small Example, which shows the difference between:
Col.KeyByIndex(i), Col.ItemByIndex(i)
and
Col.KeyBySortKeyIndex(i), Col.ItemBySortKeyIndex(i)

Private Sub Form_Load()
  Dim Col As cCollection, i As Long
  Set Col = New_c.Collection(False)
  
  Col.Add "Item 1", 1 '<- Integer-Keys (String-, Date- and Double-Types for the Keys will work also)
  Col.Add "Item 10", 10
  Col.Add "Item 2", 2
  
  Debug.Print vbLf; "Enumeration of the Key/Value-Pairs in Add-Order"
  For i = 0 To Col.Count - 1
    Debug.Print Col.KeyByIndex(i), Col.ItemByIndex(i)
  Next
  
  Debug.Print vbLf; "Enumeration of the Key/Value-Pairs in SortKey-Order"
  For i = 0 To Col.Count - 1
    Debug.Print Col.KeyBySortKeyIndex(i), Col.ItemBySortKeyIndex(i)
  Next
End Sub

But would it make the iteration faster in the cases above?

No, For Each has no "magical CPU-speedup"-capabilities - instead For Each involves
Overhead (since internally there's also an Index which needs to be "counted up" -
and in addition one has to create an instance of IEnumVariant first, before the Loop
is starting at all).

Also, would it work internally the same way when it comes to removal of items?
In one (clean up) function I iterate through the collection, starting from the high index, and check if the control that the key mimic still exists and if not remove the item. However, I am not sure exactly what's happening here internally, weather using KeyBySortKeyIndex or KeyByIndex, if removing an item triggers an automatic resort of the index or reindexing may be the proper word?

Enumerating (per Index) backwards over the Items and Keys is the right approach
in such a filter-scenario (then using .RemoveByIndex instead of .RemoveByKey).

And no, when a Key/Value-pair is removed, there's generally no "re-sorting" taking place -
just the "flagging" of the formerly occupied memory of that Key/Value-pair as "available" again
(either for a new added Key/Value-pair, or for freeing, when the last pair on a cluster was removed).

Olaf

[Niya]

And no, when a Key/Value-pair is removed, there's generally no "re-sorting" taking place -
just the "flagging" of the formerly occupied memory of that Key/Value-pair as "available" again
(either for a new added Key/Value-pair, or for freeing, when the last pair on a cluster was removed).

Does this mean your collection has something close to O(1) removal ? I say "something close to" because there still may be some kind of enumeration to find the index but indexing a memory location can be very fast if it boils down to simply reading a memory with a base/offset pair, something CPUs can do natively, which is why it is fast.

[Schmidt]

Does this mean your collection has something close to O(1) removal ?

O(1) surely not - but something between O(1) and O(log n) ... not easy to say or determine,
due to the clustered approach.

Each mem-alloc-cluster taking up 76 items (no, not 42 ), which came out (empirically)
as the "sweet-spot-number" (performancewise).

So in case there's a need to "move things around", then in most cases I have
only to do these sometimes needed rearrangements within such a given cluster,
not on the data as a whole.

Olaf

[Niya]

So what you're saying is that your collection treats the data internally as separate "buckets" of items ?

This has some interest to me as I recently tried to create a list implementation with faster than normal inserts and removals, especially when dealing with lists that have a large number of items.

[Schmidt]

So what you're saying is that your collection treats the data internally as separate "buckets" of items ?

Yes, buckets, clusters whatever you want to name them.

The above mentioned sweet-spot (76) might be a different one in your case though -
but it behaved nicely in my case, where each Key/Value-pair is represented
by an allocation of 16Bytes+16Bytes+8Bytes of Extra-Data (being:
Variant + Variant + the Extra-Info)...

And instead of allocating these 40Bytes separately (for each and every
newly added Key/Value-pair) - I'm always allocating 76*40Bytes in one go,
which is nearly as fast as allocating 40Bytes only, but spares the allocation
for the next 75 pairs which get added.

If there's something to remove, I rarely have to de-allocate - I do this only,
when the last pair is removed from some bucket (only then the bucket-allocation
in question is destroyed as well).

In an enumeration-scenario the overhead of managing a kind of "Skip-Indexing"
(running or skipping over the buckets - respecting the "current buckets count")
is barely noticable. And it leaves room for quite a lot of optimizations (e.g.
when you keep the last range of Indexes for the last touched bucket in Class-
Private-Variables, so that the next "real-Index" can be found much faster by
looking into that saved "last-accessed-bucket-range".

Not sure how something like that will play out with .NETs GC - when combined
with your own memory-mangement and pointer-access.

Olaf

[Niya]

Some nice ideas you have there. My own attempt at a custom collection was based on a doubly linked list. While it was faster to remove and insert, it presented some issues when indexing and enumerating. Indexing was the reason I tried to create such a list in the first place. The .Net Framework already has a Dictionary<T> class which is optimized for fast look ups but not so good to use as a normal indexable collection. I was trying to create something of an all rounder list class, with particular emphasis on fast removals. Your idea of using buckets is interesting. It merits experimentation on my part.

Not sure how something like that will play out with .NETs GC - when combined
with your own memory-mangement and pointer-access.

It will introduce some measure of indeterministic performance hits but the GC for the most part is pretty well behaved. Also, you can pin objects in memory to prevent the GC from moving it if you want to play around with their memory more directly.