VB6 - BasicBuffer, Binary Stream Class

[dilettante]

Description

A simple stream-style buffer class.

This is a kind of binary stream, similar to an ADO Steam object in Type = adBinary mode or an OLE IStream object. It accepts and returns Byte arrays, Integers, and Longs but not text String values.

It can be useful whenever you want a data structure that supports something similar to concatenating Byte arrays when you need to accumulate data arriving in bits and pieces and extract chunks for use. Things like Winsock control and MSComm control binary communication come to mind.

The properties and methods are similar to those of an ADO Stream.

This class has a lot of things in it to handle common needs. Remove anything you don't need in your programs.

I have done a lot of testing, but bug reports and fixes would be welcome.


Properties

ChunkSize As Long [R/W]
EOS As Boolean [RO]
HaveAtLeast(Length As Long) As Boolean [RO]
IsOpen As Boolean [RO]
Position As Long [R/W]
Size As Long [RO]


Methods

CloseBuf()
CurrentBytes() As Byte()
DeleteBytes(Optional ByVal Length As Long = -1)
ExtractBytes(Optional ByVal Length As Long = -1) As Byte()
ExtractInteger() As Integer
ExtractLong() As Long
OpenBuf()
ReadBytes(Optional ByVal Length As Long = -1) As Byte()
ReadInteger() As Integer
ReadLong() As Long
ScanForBytes(ByRef Target() As Byte) As Long
SetEOS()
ShrinkBuf()
WriteBytes(ByRef Bytes() As Byte)
WriteInteger(ByVal Value As Integer)
WriteLong(ByVal Value As Long)


Attachment

The attached archive contains BasicBuffer.cls as well as a testing Project.

It uses character data for easy reading/debugging here (converting to/from Unicode as needed) though BasicBuffers are normally for binary data.

This looks weird but tries to provide a test for correctness of as many operations as possible.


Requirements

No special requirements. You just need VB6. It may also work in Office VBA and VB5 (not tested).

See comments in the code for more help in understanding its use.

[dilettante]

Bug found and fixed. All of the ExtractXXX() methods were flawed.

Corrrected version reposted above.

[Tanner_H]

This is an excellent class: concise, well-commented, and with just the right amount of error-handling. Thank you for sharing it, dilettante.

A recent project required me to write a PSD-like file format that condensed a wide variety of data (multiple large image arrays, strings, various UDTs) into a single file, with support for parsing out individual bits of data later. I used this BasicBuffer class to assemble the various pieces of data into a contiguous stream, while keeping track of each entry's offset by using the .Position property of the class. (The individual offsets were placed in a lightweight directory at the front of the file.) Even when processing everything from small strings to multiple megabytes of pixel data, the BasicBuffer class worked flawlessly, and it greatly simplified my project.

Anyway, I realize this comment isn't particularly useful for improving the class further, but I thought it worthwhile to describe my use-case and positive experience. Thanks again.

[dilettante]

I'm glad it was helpful, even if only as a starting point for creating your own versions.

Too bad we never got something like this from Microsoft. Perhaps as part of a better and more comprehensive alternative to the Scripting Runtime? It would be useful either shipped as a Windows library not needing deployment or as new objects in a VB6-successor runtime.

Both in-memory and file-backed streams are a weak point for us in VB6, especially for binary data.

[Tanner_H]

Too bad we never got something like this from Microsoft. Perhaps as part of a better and more comprehensive alternative to the Scripting Runtime? It would be useful either shipped as a Windows library not needing deployment or as new objects in a VB6-successor runtime.

Both in-memory and file-backed streams are a weak point for us in VB6, especially for binary data.

Couldn't agree more. I've never understood why they didn't just provide an abstract stream interface like C/C++'s fread/seek/etc.

BTW, here are two trivial suggestions for this class if you ever revisit it: Read/Write/Extract functions for Strings, which handle the StrConv step for the user. (I found this preferable to handling the StrConv externally.)

I also found it helpful to add an optional Length parameter to the WriteBytes function, for writing a subset of a Byte array to the stream. My use-case for this: prior to writing my data to the stream, I compressed it via zLib, which returns the compressed data in a subsection of the original buffer. A length parameter in WriteBytes was more performance-friendly than a ReDim Preserve statement prior to writing the bytes.

[dilettante]

Excellent points. In addition to simple String methods one could even add optional alternate character encoding conversions handling UTF-8, UTF-7, various ANSI codepages, etc.

Often when I use this myself I customize it, for example delete the xxInteger and xxLong methods. I have even added an ExtractPicture() As StdPicture when the stream I'm accumulating/deblocking contains image data.

The ADO Stream object can be used to do some of these things and it's already there in Windows. However it isn't always the highest performing (working with Variants so much?). it also lacks some things I use more than you might think with custom protocols over TCP: the xxInteger and xxLong methods along with xxNetInteger and xxNetLong versions to deal with "network byte ordering."

[Schmidt]

I'm glad it was helpful, even if only as a starting point for creating your own versions.

Too bad we never got something like this from Microsoft. Perhaps as part of a better and more comprehensive alternative to the Scripting Runtime? It would be useful either shipped as a Windows library not needing deployment or as new objects in a VB6-successor runtime.

Both in-memory and file-backed streams are a weak point for us in VB6, especially for binary data.

Well, since you mentioned: "...as new objects in a VB6-successor runtime" - this thing is *here* - available already
for good old VB6, with a whole lot of stuff (File- and Memory-Streams - and other "serialization-related things"
as InMemory-Archives, a StringBuilder with a ToUTF8-method, serializable Collections- and Array-List-Classes,
fast JSON-support etc...

Seriously wished, that it would be more widely adopted by the community (it could need a bit more support
from the "recognized heavy-guns of the classic-community", you know... - a whole lot of questions could
be answered with only a few lines here in the forum, when it would become "common", to use this lib as a
"quasi-default" in a VB6-Startup-Template).

Also with regards to drawing-support, (this is more directed to Tanner) - it contains a whole lot of stuff,
which could ease the development of modern looking Apps like PhotoDemon (layered transforms, modern
blend-ops, Vector-support in drawing-primitives - direct Pixel-access and stuff).

If you two guys (dile and Tanner) are interested in the Sources - I have no problem to send them over
to you, until this stuff will finally land in a public repository in a year or two (the plan always existed, but
there's still some beautifying needed internally - found no time over the last months and years to do that) ...
- so, just drop me a mail (a real one), in case you're interested in the sources.

I've spread them already among a few community-members... maybe such a step can increase the
trust-level a bit, when it is known that the whole thing is not really a "one-man-show" anymore
(or that at least this "single-point-of-failure"-problem is already addressed, in case my poor self
should go "out of scope" at some point in time ).

Olaf

[dilettante]

I need to make the time investment to explore your library one of these days. Right now I only do Windows programming here and there or tinker with things brought up by forum questions as a time filler while waiting for phone calls.

The one drawback is the community's tendency to look down on "external dependencies" of any kind. A lot of this stems from lack of deployment skills, and more may come from "locked down" target PCs (no admin rights).

Of course there are good answers to that for many applications, either Windows SxS reg-free or your own technique.


Not to drift off topic too far but...

Another huge omission I see are cloud service API libraries for VB6 and VBA. Microsoft doesn't care ("move to .Net, use Interop, etc."). Google, Amazon, etc. do not want VB6/VBA coders for a mix of reasons, both political ("we hate VB") and practical (supporting legions of casual coders can be a nighmare).

[Niya]

I just glimpsed at the source for the Buffer class using NotePad(Didn't install VB6 on my last Windows re-install). I didn't notice any interfaces. I think you should make a generic stream interface and have that class implement it. That way you can implement stream based classes backed by different sources(memory, disk, network) but give developers the benefit of coding against a single interface. This would make it easy to use different stream types interchangeably. Just a humble suggestion.

[dilettante]

Frankly, IStream and other common stream interfaces are so feature-poor as to be worthless except for talking to some other code that already uses them. I had started out that way and quickly abandoned it since the "stock" stream interface members were hardly being used in any real programs.

This was really meant as a binary equivalent of a string builder class, which is why it lacks String methods as posted. As such it doesn't make a lot of sense as a gateway into a disk file or existing blob of memory.

Heck guys, it's a buffer not an operating system.


Since you have the source you can add or subtract arms and legs to meet your needs.

[dilettante]

That wasn't meant to "pick on" the idea. I have returned to it several times myself but it just didn't make much sense so I dropped it again.

Basically you use it to accumulate stuff that arrives in bits and pieces at the tail end and pick it off from the head end. That doesn't mean it couldn't be a disk file I/O class, I just haven't had a need for that since regular old I/O statements normally do just fine.

[Tanner_H]

@Olaf,

I sent an email about your comments to the address listed on your GitHub page. I thought that might be better than derailing poor dilettante's thread further. (I've done that to him enough lately!)

@Niya,

I just glimpsed at the source for the Buffer class using NotePad(Didn't install VB6 on my last Windows re-install). I didn't notice any interfaces. I think you should make a generic stream interface and have that class implement it. That way you can implement stream based classes backed by different sources(memory, disk, network) but give developers the benefit of coding against a single interface. This would make it easy to use different stream types interchangeably. Just a humble suggestion.

If someone else were to create a different stream handler (network, for example), I would love to see it use the same interface as dilettante's BasicBuffer class here, so they could share an interface. That is an excellent idea. Now we just need to find someone with enough free time to do such a thing...!

Actually, I already use dilettante's class as a file handler for my project mentioned above. Once I've verified the first few bytes of the file, I throw the whole thing into his BasicBuffer class, then process it from there. For my particular application, this proved to be much faster than using VB's Get statement repeatedly, probably because it avoids multiple trips back out to the hard drive. So in that sense, I think it might already work very well as a file interface.

@dilettante,

The ADO Stream object can be used to do some of these things and it's already there in Windows. However it isn't always the highest performing (working with Variants so much?). it also lacks some things I use more than you might think with custom protocols over TCP: the xxInteger and xxLong methods along with xxNetInteger and xxNetLong versions to deal with "network byte ordering."

One of the reasons I think your class is so helpful vs something like an ADO Stream - aside from its good performance, as you mention - is that yours is not a "black box." I found it really easy to write high-performing code against it, because I knew exactly what would happen when I appended data, or how it would handle manually moving the pointer around, or what errors it would catch vs what ones I needed to watch for. Even obnoxious little things like "is this property 0-based or 1-based" are so much easier to solve when you can actually poke around the code.

In that sense, I'm reluctantly grateful that so many obvious structures (streams, stacks, trees, etc) have to be created manually in VB. As nice as it would be to have these built-in to the language, they'd probably just be half-baked garbage like Collections. (Maybe garbage is a little strong, my apologies to the developers responsible for Collections... )

[dilettante]

The case of the Collection is particularly annoying.

Scripting.Dictionary is an improvement in some ways, and a step back in others. My understanding was that Microsoft had developed a 3rd alternative with some advantages from both of the others but it never saw general distribution. I wish I could find its description again but I can't remember what they were calling it.

Here's one blog post that indirectly mentions it: Arrrrr! Cap'n Eric be learnin' about threadin' the harrrrd way.

Ahh: strtable.dll, but the comments suggest it was a special-purpose thing anyway. Anyone tried the Motobit Multi Dictionary linked there in the last comment or so?

But yeah, we've drifted far off topic now.

[Schmidt]

The case of the Collection is particularly annoying.

Yep - that was one of the first things I addressed in the first versions of the RichClient -
and it was done decently (completely in VB6, but still the fastest Dictionary or Collection
implementations out there - even when compared with C++ implementations).

Check them out yourself (cSortedDictionary is a bit lighter and faster than cCollection - and whilst
the latter one does internal "sorting-whilst-adding" as well - it also keeps the Items original
Add-order intact - so you can eumerate Items on a cCollection in add- and in sorted order).

Anyone tried the Motobit Multi Dictionary linked there in the last comment or so?

Didn't know about that - but it claims to be faster than the Scripting.Dictionary,
so I downloaded it (60 days trial-period) - and compared vs. cSortedDictionary.

And no - also this C++ implementation is slower than the VB6-implemented alternatives.

Here's what came out in a small test (code further below):

RC5-Dictionary, Add 100000 Items: 189,42msec
MultiDictionary, Add 100000 Items: 365,28msec

RC5-Dictionary, Exists on 100000 Items: 151,59msec
MultiDictionary, Exists on 100000 Items: 177,53msec

RC5-Dictionary, Removed dupes to 31715 Items: 189,25msec
MultiDictionary, Removed dupes to 31715 Items: 433,54msec

RC5-Dictionary, Remove single items to 10534 Items: 88,62msec
MultiDictionary, Remove single items to 10534 Items: 146,56msec

RC5-Dictionary, Remove All to 0 Items: 13,84msec
MultiDictionary, Remove All to 0 Items: 18,30msec

Option Explicit

Private Const TestCount& = 100000

Private Keys() As String, DRC5 As vbRichClient5.cSortedDictionary, DMUL As Multi.Dictionary

Private Sub Form_Load()
  Set DRC5 = New_c.SortedDictionary
      DRC5.UniqueKeys = False
      DRC5.StringCompareMode = TextCompare
  
  Set DMUL = New Multi.Dictionary
      DMUL.UniqueKeys = False
      DMUL.CompareMode = TextCompare
      
  Dim i As Long
  ReDim Keys(0 To TestCount - 1)
  Rnd -1
  For i = 0 To UBound(Keys) 'we prepare the Keys beforehand, to avoid this overhead in the comparison
    Keys(i) = "SomeKey" & Int(Rnd * TestCount / 3)
  Next i
End Sub

Private Sub Form_Click()
  CompareAddPerformance
  CompareExistsPerformance
  CompareRemoveDuplicatesPerformance
  CompareRemoveSingleItemsPerformance
  CompareRemoveAllPerformance
End Sub

Private Sub CompareAddPerformance()
Dim i As Long
  New_c.Timing True
    For i = 0 To UBound(Keys)
      DRC5.Add Keys(i), i
    Next i
  Debug.Print "RC5-Dictionary, Add"; DRC5.Count; "Items:"; New_c.Timing
  DoEvents
  
  New_c.Timing True
    For i = 0 To UBound(Keys)
      DMUL.Add Keys(i), i
    Next i
  Debug.Print "MultiDictionary, Add"; DMUL.Count; "Items:"; New_c.Timing
  DoEvents
  Debug.Print
End Sub

Private Sub CompareExistsPerformance()
Dim i As Long
  New_c.Timing True
    For i = 0 To UBound(Keys)
      If Not DRC5.Exists(Keys(i)) Then MsgBox "Shouldn't happen"
    Next i
  Debug.Print "RC5-Dictionary, Exists on"; DRC5.Count; "Items:"; New_c.Timing
  DoEvents
  
  New_c.Timing True
    For i = 0 To UBound(Keys)
      If Not DMUL.Exists(Keys(i)) Then MsgBox "Shouldn't happen"
    Next i
  Debug.Print "MultiDictionary, Exists on"; DMUL.Count; "Items:"; New_c.Timing
  DoEvents
  Debug.Print
End Sub

Private Sub CompareRemoveDuplicatesPerformance()
Dim i As Long, Keys(), Items(), DRC5Tmp As cSortedDictionary
  New_c.Timing True
    Set DRC5Tmp = New_c.SortedDictionary(TextCompare, False)
    For i = 0 To DRC5.Count - 1
      If Not DRC5Tmp.Exists(DRC5.KeyByIndex(i)) Then DRC5Tmp.Add DRC5.KeyByIndex(i), DRC5.ItemByIndex(i)
    Next i
    Set DRC5 = DRC5Tmp
  Debug.Print "RC5-Dictionary, Removed dupes to"; DRC5.Count; "Items:"; New_c.Timing
  DoEvents
  
  New_c.Timing True
    Keys = DMUL.Keys
    Items = DMUL.Items
    DMUL.RemoveAll
    For i = 0 To UBound(Keys)
      If Not DMUL.Exists(Keys(i)) Then DMUL.Add Keys(i), Items(i)
    Next i
  Debug.Print "MultiDictionary, Removed dupes to"; DMUL.Count; "Items:"; New_c.Timing
  DoEvents
  Debug.Print
End Sub
 
Private Sub CompareRemoveSingleItemsPerformance()
Dim i As Long
  New_c.Timing True
    For i = 0 To UBound(Keys) Step 3
      If DRC5.Exists(Keys(i)) Then DRC5.Remove Keys(i)
    Next i
  Debug.Print "RC5-Dictionary, Remove single items to"; DRC5.Count; "Items:"; New_c.Timing
  DoEvents
  
  New_c.Timing True
    For i = 0 To UBound(Keys) Step 3
      If DMUL.Exists(Keys(i)) Then DMUL.Remove Keys(i)
    Next i
  Debug.Print "MultiDictionary, Remove single items to"; DMUL.Count; "Items:"; New_c.Timing
  DoEvents
  Debug.Print
End Sub

Private Sub CompareRemoveAllPerformance()
Dim i As Long
  New_c.Timing True
    DRC5.RemoveAll
  Debug.Print "RC5-Dictionary, Remove All to"; DRC5.Count; "Items:"; New_c.Timing
  DoEvents
  
  New_c.Timing True
    DMUL.RemoveAll
  Debug.Print "MultiDictionary, Remove All to"; DMUL.Count; "Items:"; New_c.Timing
  DoEvents
  Debug.Print
End Sub

Olaf

[dilettante]

Well it does offer Share dictionary values over processes/remote computers

You can also install the dictionary as a COM+ application and share variables between several application pools, isolated IIS processes, between windows scripting host, VBA applications or any other applications. You can also share dictionary variables between more computers over local network, over internet or over http connection.

So it may have some redeeming qualities, but I'm not sure I want to invest time and energy into evaluating it for use in future applications.


What I need to do is set aside some time to do more with VB6 than just the occasional project and forum question "quickies." Right now Android and JVM work is offering more paying work though. At least B4A and B4J help speed development on those runtimes.

The death of WinXP combined with the Win8 debacle has been frightening a lot of my clients away from Windows so I find a lot of small "proof of concept" jobs. These don't pay well because of the hand-holding required (these people don't know what they want or what to expect) but I want to get "my foot in the door" for bigger projects. I may even have to look at Chrome OS development, because interest is growing there as well.