VB6 - BasicBuffer, Binary Stream Class-VBForums
Results 1 to 15 of 15

Thread: VB6 - BasicBuffer, Binary Stream Class

  1. #1

    Thread Starter
    PowerPoster
    Join Date
    Feb 2006
    Posts
    15,940

    Lightbulb VB6 - BasicBuffer, Binary Stream Class

    Description

    A simple stream-style buffer class.

    This is a kind of binary stream, similar to an ADO Steam object in Type = adBinary mode or an OLE IStream object. It accepts and returns Byte arrays, Integers, and Longs but not text String values.

    It can be useful whenever you want a data structure that supports something similar to concatenating Byte arrays when you need to accumulate data arriving in bits and pieces and extract chunks for use. Things like Winsock control and MSComm control binary communication come to mind.

    The properties and methods are similar to those of an ADO Stream.

    This class has a lot of things in it to handle common needs. Remove anything you don't need in your programs.

    I have done a lot of testing, but bug reports and fixes would be welcome.


    Properties

    ChunkSize As Long [R/W]
    EOS As Boolean [RO]
    HaveAtLeast(Length As Long) As Boolean [RO]
    IsOpen As Boolean [RO]
    Position As Long [R/W]
    Size As Long [RO]



    Methods

    CloseBuf()
    CurrentBytes() As Byte()
    DeleteBytes(Optional ByVal Length As Long = -1)
    ExtractBytes(Optional ByVal Length As Long = -1) As Byte()
    ExtractInteger() As Integer
    ExtractLong() As Long
    OpenBuf()
    ReadBytes(Optional ByVal Length As Long = -1) As Byte()
    ReadInteger() As Integer
    ReadLong() As Long
    ScanForBytes(ByRef Target() As Byte) As Long
    SetEOS()
    ShrinkBuf()
    WriteBytes(ByRef Bytes() As Byte)
    WriteInteger(ByVal Value As Integer)
    WriteLong(ByVal Value As Long)



    Attachment

    The attached archive contains BasicBuffer.cls as well as a testing Project.

    It uses character data for easy reading/debugging here (converting to/from Unicode as needed) though BasicBuffers are normally for binary data.

    This looks weird but tries to provide a test for correctness of as many operations as possible.


    Requirements

    No special requirements. You just need VB6. It may also work in Office VBA and VB5 (not tested).

    See comments in the code for more help in understanding its use.
    Attached Files Attached Files
    Last edited by dilettante; Nov 3rd, 2013 at 05:33 PM. Reason: replaced attachment

  2. #2

    Thread Starter
    PowerPoster
    Join Date
    Feb 2006
    Posts
    15,940

    Re: VB6 - BasicBuffer, Binary Stream Class

    Bug found and fixed. All of the ExtractXXX() methods were flawed.

    Corrrected version reposted above.

  3. #3
    Fanatic Member
    Join Date
    Aug 2013
    Posts
    621

    Re: VB6 - BasicBuffer, Binary Stream Class

    This is an excellent class: concise, well-commented, and with just the right amount of error-handling. Thank you for sharing it, dilettante.

    A recent project required me to write a PSD-like file format that condensed a wide variety of data (multiple large image arrays, strings, various UDTs) into a single file, with support for parsing out individual bits of data later. I used this BasicBuffer class to assemble the various pieces of data into a contiguous stream, while keeping track of each entry's offset by using the .Position property of the class. (The individual offsets were placed in a lightweight directory at the front of the file.) Even when processing everything from small strings to multiple megabytes of pixel data, the BasicBuffer class worked flawlessly, and it greatly simplified my project.

    Anyway, I realize this comment isn't particularly useful for improving the class further, but I thought it worthwhile to describe my use-case and positive experience. Thanks again.
    Check out PhotoDemon, a pro-grade photo editor written completely in VB6. (Full source available at GitHub.)

  4. #4

    Thread Starter
    PowerPoster
    Join Date
    Feb 2006
    Posts
    15,940

    Re: VB6 - BasicBuffer, Binary Stream Class

    I'm glad it was helpful, even if only as a starting point for creating your own versions.

    Too bad we never got something like this from Microsoft. Perhaps as part of a better and more comprehensive alternative to the Scripting Runtime? It would be useful either shipped as a Windows library not needing deployment or as new objects in a VB6-successor runtime.

    Both in-memory and file-backed streams are a weak point for us in VB6, especially for binary data.

  5. #5
    Fanatic Member
    Join Date
    Aug 2013
    Posts
    621

    Re: VB6 - BasicBuffer, Binary Stream Class

    Quote Originally Posted by dilettante View Post
    Too bad we never got something like this from Microsoft. Perhaps as part of a better and more comprehensive alternative to the Scripting Runtime? It would be useful either shipped as a Windows library not needing deployment or as new objects in a VB6-successor runtime.

    Both in-memory and file-backed streams are a weak point for us in VB6, especially for binary data.
    Couldn't agree more. I've never understood why they didn't just provide an abstract stream interface like C/C++'s fread/seek/etc.

    BTW, here are two trivial suggestions for this class if you ever revisit it: Read/Write/Extract functions for Strings, which handle the StrConv step for the user. (I found this preferable to handling the StrConv externally.)

    I also found it helpful to add an optional Length parameter to the WriteBytes function, for writing a subset of a Byte array to the stream. My use-case for this: prior to writing my data to the stream, I compressed it via zLib, which returns the compressed data in a subsection of the original buffer. A length parameter in WriteBytes was more performance-friendly than a ReDim Preserve statement prior to writing the bytes.
    Check out PhotoDemon, a pro-grade photo editor written completely in VB6. (Full source available at GitHub.)

  6. #6

    Thread Starter
    PowerPoster
    Join Date
    Feb 2006
    Posts
    15,940

    Re: VB6 - BasicBuffer, Binary Stream Class

    Excellent points. In addition to simple String methods one could even add optional alternate character encoding conversions handling UTF-8, UTF-7, various ANSI codepages, etc.

    Often when I use this myself I customize it, for example delete the xxInteger and xxLong methods. I have even added an ExtractPicture() As StdPicture when the stream I'm accumulating/deblocking contains image data.

    The ADO Stream object can be used to do some of these things and it's already there in Windows. However it isn't always the highest performing (working with Variants so much?). it also lacks some things I use more than you might think with custom protocols over TCP: the xxInteger and xxLong methods along with xxNetInteger and xxNetLong versions to deal with "network byte ordering."

  7. #7
    PowerPoster
    Join Date
    Jun 2013
    Posts
    2,835

    Re: VB6 - BasicBuffer, Binary Stream Class

    Quote Originally Posted by dilettante View Post
    I'm glad it was helpful, even if only as a starting point for creating your own versions.

    Too bad we never got something like this from Microsoft. Perhaps as part of a better and more comprehensive alternative to the Scripting Runtime? It would be useful either shipped as a Windows library not needing deployment or as new objects in a VB6-successor runtime.

    Both in-memory and file-backed streams are a weak point for us in VB6, especially for binary data.
    Well, since you mentioned: "...as new objects in a VB6-successor runtime" - this thing is *here* - available already
    for good old VB6, with a whole lot of stuff (File- and Memory-Streams - and other "serialization-related things"
    as InMemory-Archives, a StringBuilder with a ToUTF8-method, serializable Collections- and Array-List-Classes,
    fast JSON-support etc...

    Seriously wished, that it would be more widely adopted by the community (it could need a bit more support
    from the "recognized heavy-guns of the classic-community", you know... - a whole lot of questions could
    be answered with only a few lines here in the forum, when it would become "common", to use this lib as a
    "quasi-default" in a VB6-Startup-Template).

    Also with regards to drawing-support, (this is more directed to Tanner) - it contains a whole lot of stuff,
    which could ease the development of modern looking Apps like PhotoDemon (layered transforms, modern
    blend-ops, Vector-support in drawing-primitives - direct Pixel-access and stuff).

    If you two guys (dile and Tanner) are interested in the Sources - I have no problem to send them over
    to you, until this stuff will finally land in a public repository in a year or two (the plan always existed, but
    there's still some beautifying needed internally - found no time over the last months and years to do that) ...
    - so, just drop me a mail (a real one), in case you're interested in the sources.

    I've spread them already among a few community-members... maybe such a step can increase the
    trust-level a bit, when it is known that the whole thing is not really a "one-man-show" anymore
    (or that at least this "single-point-of-failure"-problem is already addressed, in case my poor self
    should go "out of scope" at some point in time ).

    Olaf

  8. #8

    Thread Starter
    PowerPoster
    Join Date
    Feb 2006
    Posts
    15,940

    Re: VB6 - BasicBuffer, Binary Stream Class

    I need to make the time investment to explore your library one of these days. Right now I only do Windows programming here and there or tinker with things brought up by forum questions as a time filler while waiting for phone calls.

    The one drawback is the community's tendency to look down on "external dependencies" of any kind. A lot of this stems from lack of deployment skills, and more may come from "locked down" target PCs (no admin rights).

    Of course there are good answers to that for many applications, either Windows SxS reg-free or your own technique.


    Not to drift off topic too far but...

    Another huge omission I see are cloud service API libraries for VB6 and VBA. Microsoft doesn't care ("move to .Net, use Interop, etc."). Google, Amazon, etc. do not want VB6/VBA coders for a mix of reasons, both political ("we hate VB") and practical (supporting legions of casual coders can be a nighmare).

  9. #9
    Angel of Code Niya's Avatar
    Join Date
    Nov 2011
    Posts
    5,346

    Re: VB6 - BasicBuffer, Binary Stream Class

    I just glimpsed at the source for the Buffer class using NotePad(Didn't install VB6 on my last Windows re-install). I didn't notice any interfaces. I think you should make a generic stream interface and have that class implement it. That way you can implement stream based classes backed by different sources(memory, disk, network) but give developers the benefit of coding against a single interface. This would make it easy to use different stream types interchangeably. Just a humble suggestion.
    Treeview with NodeAdded/NodesRemoved events | BlinkLabel control | Calculate Permutations | Object Enums | ComboBox with centered items | .Net Internals article(not mine) | Wizard Control | Understanding Multi-Threading | Simple file compression | Demon Arena


    C++ programmers will dismiss you as a cretinous simpleton for your inability to keep track of pointers chained 6 levels deep and Java programmers will pillory you for buying into the evils of Microsoft. Meanwhile C# programmers will get paid just a little bit more than you for writing exactly the same code and VB6 programmers will continue to whitter on about "footprints". - FunkyDexter

    There's just no reason to use garbage like InputBox. -jmcilhinney

  10. #10

    Thread Starter
    PowerPoster
    Join Date
    Feb 2006
    Posts
    15,940

    Re: VB6 - BasicBuffer, Binary Stream Class

    Frankly, IStream and other common stream interfaces are so feature-poor as to be worthless except for talking to some other code that already uses them. I had started out that way and quickly abandoned it since the "stock" stream interface members were hardly being used in any real programs.

    This was really meant as a binary equivalent of a string builder class, which is why it lacks String methods as posted. As such it doesn't make a lot of sense as a gateway into a disk file or existing blob of memory.

    Heck guys, it's a buffer not an operating system.


    Since you have the source you can add or subtract arms and legs to meet your needs.

  11. #11

    Thread Starter
    PowerPoster
    Join Date
    Feb 2006
    Posts
    15,940

    Re: VB6 - BasicBuffer, Binary Stream Class

    That wasn't meant to "pick on" the idea. I have returned to it several times myself but it just didn't make much sense so I dropped it again.

    Basically you use it to accumulate stuff that arrives in bits and pieces at the tail end and pick it off from the head end. That doesn't mean it couldn't be a disk file I/O class, I just haven't had a need for that since regular old I/O statements normally do just fine.

  12. #12
    Fanatic Member
    Join Date
    Aug 2013
    Posts
    621

    Re: VB6 - BasicBuffer, Binary Stream Class

    @Olaf,

    I sent an email about your comments to the address listed on your GitHub page. I thought that might be better than derailing poor dilettante's thread further. (I've done that to him enough lately!)

    @Niya,

    Quote Originally Posted by Niya View Post
    I just glimpsed at the source for the Buffer class using NotePad(Didn't install VB6 on my last Windows re-install). I didn't notice any interfaces. I think you should make a generic stream interface and have that class implement it. That way you can implement stream based classes backed by different sources(memory, disk, network) but give developers the benefit of coding against a single interface. This would make it easy to use different stream types interchangeably. Just a humble suggestion.
    If someone else were to create a different stream handler (network, for example), I would love to see it use the same interface as dilettante's BasicBuffer class here, so they could share an interface. That is an excellent idea. Now we just need to find someone with enough free time to do such a thing...!

    Actually, I already use dilettante's class as a file handler for my project mentioned above. Once I've verified the first few bytes of the file, I throw the whole thing into his BasicBuffer class, then process it from there. For my particular application, this proved to be much faster than using VB's Get statement repeatedly, probably because it avoids multiple trips back out to the hard drive. So in that sense, I think it might already work very well as a file interface.

    @dilettante,

    Quote Originally Posted by dilettante View Post
    The ADO Stream object can be used to do some of these things and it's already there in Windows. However it isn't always the highest performing (working with Variants so much?). it also lacks some things I use more than you might think with custom protocols over TCP: the xxInteger and xxLong methods along with xxNetInteger and xxNetLong versions to deal with "network byte ordering."
    One of the reasons I think your class is so helpful vs something like an ADO Stream - aside from its good performance, as you mention - is that yours is not a "black box." I found it really easy to write high-performing code against it, because I knew exactly what would happen when I appended data, or how it would handle manually moving the pointer around, or what errors it would catch vs what ones I needed to watch for. Even obnoxious little things like "is this property 0-based or 1-based" are so much easier to solve when you can actually poke around the code.

    In that sense, I'm reluctantly grateful that so many obvious structures (streams, stacks, trees, etc) have to be created manually in VB. As nice as it would be to have these built-in to the language, they'd probably just be half-baked garbage like Collections. (Maybe garbage is a little strong, my apologies to the developers responsible for Collections... )
    Check out PhotoDemon, a pro-grade photo editor written completely in VB6. (Full source available at GitHub.)

  13. #13

    Thread Starter
    PowerPoster
    Join Date
    Feb 2006
    Posts
    15,940

    Re: VB6 - BasicBuffer, Binary Stream Class

    The case of the Collection is particularly annoying.

    Scripting.Dictionary is an improvement in some ways, and a step back in others. My understanding was that Microsoft had developed a 3rd alternative with some advantages from both of the others but it never saw general distribution. I wish I could find its description again but I can't remember what they were calling it.

    Here's one blog post that indirectly mentions it: Arrrrr! Cap'n Eric be learnin' about threadin' the harrrrd way.

    Ahh: strtable.dll, but the comments suggest it was a special-purpose thing anyway. Anyone tried the Motobit Multi Dictionary linked there in the last comment or so?

    But yeah, we've drifted far off topic now.
    Last edited by dilettante; Apr 11th, 2014 at 11:07 PM.

  14. #14
    PowerPoster
    Join Date
    Jun 2013
    Posts
    2,835

    Re: VB6 - BasicBuffer, Binary Stream Class

    Quote Originally Posted by dilettante View Post
    The case of the Collection is particularly annoying.
    Yep - that was one of the first things I addressed in the first versions of the RichClient -
    and it was done decently (completely in VB6, but still the fastest Dictionary or Collection
    implementations out there - even when compared with C++ implementations).

    Check them out yourself (cSortedDictionary is a bit lighter and faster than cCollection - and whilst
    the latter one does internal "sorting-whilst-adding" as well - it also keeps the Items original
    Add-order intact - so you can eumerate Items on a cCollection in add- and in sorted order).

    Quote Originally Posted by dilettante View Post
    Anyone tried the Motobit Multi Dictionary linked there in the last comment or so?
    Didn't know about that - but it claims to be faster than the Scripting.Dictionary,
    so I downloaded it (60 days trial-period) - and compared vs. cSortedDictionary.

    And no - also this C++ implementation is slower than the VB6-implemented alternatives.

    Here's what came out in a small test (code further below):

    RC5-Dictionary, Add 100000 Items: 189,42msec
    MultiDictionary, Add 100000 Items: 365,28msec

    RC5-Dictionary, Exists on 100000 Items: 151,59msec
    MultiDictionary, Exists on 100000 Items: 177,53msec

    RC5-Dictionary, Removed dupes to 31715 Items: 189,25msec
    MultiDictionary, Removed dupes to 31715 Items: 433,54msec

    RC5-Dictionary, Remove single items to 10534 Items: 88,62msec
    MultiDictionary, Remove single items to 10534 Items: 146,56msec

    RC5-Dictionary, Remove All to 0 Items: 13,84msec
    MultiDictionary, Remove All to 0 Items: 18,30msec

    Code:
    Option Explicit
    
    Private Const TestCount& = 100000
    
    Private Keys() As String, DRC5 As vbRichClient5.cSortedDictionary, DMUL As Multi.Dictionary
    
    Private Sub Form_Load()
      Set DRC5 = New_c.SortedDictionary
          DRC5.UniqueKeys = False
          DRC5.StringCompareMode = TextCompare
      
      Set DMUL = New Multi.Dictionary
          DMUL.UniqueKeys = False
          DMUL.CompareMode = TextCompare
          
      Dim i As Long
      ReDim Keys(0 To TestCount - 1)
      Rnd -1
      For i = 0 To UBound(Keys) 'we prepare the Keys beforehand, to avoid this overhead in the comparison
        Keys(i) = "SomeKey" & Int(Rnd * TestCount / 3)
      Next i
    End Sub
    
    Private Sub Form_Click()
      CompareAddPerformance
      CompareExistsPerformance
      CompareRemoveDuplicatesPerformance
      CompareRemoveSingleItemsPerformance
      CompareRemoveAllPerformance
    End Sub
    
    Private Sub CompareAddPerformance()
    Dim i As Long
      New_c.Timing True
        For i = 0 To UBound(Keys)
          DRC5.Add Keys(i), i
        Next i
      Debug.Print "RC5-Dictionary, Add"; DRC5.Count; "Items:"; New_c.Timing
      DoEvents
      
      New_c.Timing True
        For i = 0 To UBound(Keys)
          DMUL.Add Keys(i), i
        Next i
      Debug.Print "MultiDictionary, Add"; DMUL.Count; "Items:"; New_c.Timing
      DoEvents
      Debug.Print
    End Sub
    
    Private Sub CompareExistsPerformance()
    Dim i As Long
      New_c.Timing True
        For i = 0 To UBound(Keys)
          If Not DRC5.Exists(Keys(i)) Then MsgBox "Shouldn't happen"
        Next i
      Debug.Print "RC5-Dictionary, Exists on"; DRC5.Count; "Items:"; New_c.Timing
      DoEvents
      
      New_c.Timing True
        For i = 0 To UBound(Keys)
          If Not DMUL.Exists(Keys(i)) Then MsgBox "Shouldn't happen"
        Next i
      Debug.Print "MultiDictionary, Exists on"; DMUL.Count; "Items:"; New_c.Timing
      DoEvents
      Debug.Print
    End Sub
    
    Private Sub CompareRemoveDuplicatesPerformance()
    Dim i As Long, Keys(), Items(), DRC5Tmp As cSortedDictionary
      New_c.Timing True
        Set DRC5Tmp = New_c.SortedDictionary(TextCompare, False)
        For i = 0 To DRC5.Count - 1
          If Not DRC5Tmp.Exists(DRC5.KeyByIndex(i)) Then DRC5Tmp.Add DRC5.KeyByIndex(i), DRC5.ItemByIndex(i)
        Next i
        Set DRC5 = DRC5Tmp
      Debug.Print "RC5-Dictionary, Removed dupes to"; DRC5.Count; "Items:"; New_c.Timing
      DoEvents
      
      New_c.Timing True
        Keys = DMUL.Keys
        Items = DMUL.Items
        DMUL.RemoveAll
        For i = 0 To UBound(Keys)
          If Not DMUL.Exists(Keys(i)) Then DMUL.Add Keys(i), Items(i)
        Next i
      Debug.Print "MultiDictionary, Removed dupes to"; DMUL.Count; "Items:"; New_c.Timing
      DoEvents
      Debug.Print
    End Sub
     
    Private Sub CompareRemoveSingleItemsPerformance()
    Dim i As Long
      New_c.Timing True
        For i = 0 To UBound(Keys) Step 3
          If DRC5.Exists(Keys(i)) Then DRC5.Remove Keys(i)
        Next i
      Debug.Print "RC5-Dictionary, Remove single items to"; DRC5.Count; "Items:"; New_c.Timing
      DoEvents
      
      New_c.Timing True
        For i = 0 To UBound(Keys) Step 3
          If DMUL.Exists(Keys(i)) Then DMUL.Remove Keys(i)
        Next i
      Debug.Print "MultiDictionary, Remove single items to"; DMUL.Count; "Items:"; New_c.Timing
      DoEvents
      Debug.Print
    End Sub
    
    Private Sub CompareRemoveAllPerformance()
    Dim i As Long
      New_c.Timing True
        DRC5.RemoveAll
      Debug.Print "RC5-Dictionary, Remove All to"; DRC5.Count; "Items:"; New_c.Timing
      DoEvents
      
      New_c.Timing True
        DMUL.RemoveAll
      Debug.Print "MultiDictionary, Remove All to"; DMUL.Count; "Items:"; New_c.Timing
      DoEvents
      Debug.Print
    End Sub
    Olaf
    Last edited by Schmidt; Apr 12th, 2014 at 03:44 PM.

  15. #15

    Thread Starter
    PowerPoster
    Join Date
    Feb 2006
    Posts
    15,940

    Re: VB6 - BasicBuffer, Binary Stream Class

    Well it does offer Share dictionary values over processes/remote computers

    You can also install the dictionary as a COM+ application and share variables between several application pools, isolated IIS processes, between windows scripting host, VBA applications or any other applications. You can also share dictionary variables between more computers over local network, over internet or over http connection.
    So it may have some redeeming qualities, but I'm not sure I want to invest time and energy into evaluating it for use in future applications.


    What I need to do is set aside some time to do more with VB6 than just the occasional project and forum question "quickies." Right now Android and JVM work is offering more paying work though. At least B4A and B4J help speed development on those runtimes.

    The death of WinXP combined with the Win8 debacle has been frightening a lot of my clients away from Windows so I find a lot of small "proof of concept" jobs. These don't pay well because of the hand-holding required (these people don't know what they want or what to expect) but I want to get "my foot in the door" for bigger projects. I may even have to look at Chrome OS development, because interest is growing there as well.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Featured


Click Here to Expand Forum to Full Width

Survey posted by VBForums.