Creating a large string in a private heap

[vbwins]

So I have a project that must run on 32bit and 64 bit windows. On 32 bit windows I sometimes get the dreaded "out of string space" error. I have tried as best I can to modify the code to use string buffers etc but it still happens sometimes

Using the information contained in this post I thought about creating a private heap to keep my large strings in.

https://www.vbforums.com/showthread....ocess-s-memory

The rough trial code looks like this. Is this a terrible idea? If it is then what would I need to do to make it workable. If its not what have I left out. I am not checking heap function status but I will. Does anyone know if out of string space means some limit has been hit or just that the default heap is too fragmented for the new allocation when I need to copy a string or pass it back up the stack. ( i guess this makes a new string)

 Dim lngPointer As Long
      Dim strString As String
      Dim lngPointer1 As Long
      Dim lngLength As Long
      Dim newLength As Long
      Dim oldBstrPtr As Long
      Dim oldStrPtr As Long
      Dim oldBStrLength As Long

10    newLength = 2048

20    strString = "a"

30    lngPointer = StrPtr(strString)

      ' save the old string pointer

40    oldStrPtr = lngPointer
      ' save the pointer to the old bStr

50    CopyMemory oldBstrPtr, ByVal lngPointer, 4

      ' save the old length

60    CopyMemory oldBStrLength, ByVal lngPointer - 4, 4
      '
      ' The 4 bytes immediately before the BSTR is the length of the string
      '
      ' Create a heap

70    hHeap = HeapCreate(0, 1024& * 1024, 0)

      ' allocate some memory

80    ptrHBlock = HeapAlloc(hHeap, 0, 2048)

      ' copy the address of the memory we allocated to to the old bstr address

90    CopyMemory ByVal lngPointer, ptrHBlock, 4

      ' copy the new length we want
      
100   CopyMemory ByVal lngPointer - 4, newLength, 4
    
' in theory I now have a string in a private heap

' Put it all back the way it was

110   CopyMemory ByVal lngPointer, oldBstrPtr, 4

120   CopyMemory ByVal lngPointer - 4, oldBStrLength, 4

'clean up
130 HeapFree hHeap, 0, ptrHBlock

140 HeapDestroy hHeap

[Arnoutdv]

How big is big?
Often the problem arises when you are on the edge of your memory buffer and new space needs to be allocated.
Then at least twice the memory is needed, the original buffer and the new increased buffer.

[vbwins]

The project allocates and deallocates lots of small strings so I think things get pretty fragmented. Its linked largeAddressAware on 32 and 64 bit. The problem does not seem to happen on 32 bit

I this case I need some decompression buffers of about 20 megabytes big. Only 3 at the moment. If for any reason I need to grow them (and I might) the thing gets an out of string space error even though memory utilization is around 1.3 gig. Close to the wind on 32 bits I know.

But is this s terrible idea?

[vbwins]

Interestingly I allocate 40 megs of memory, set the length to 40 million. Len(strString) returns 20meg, lenb(strString) returns 40 meg. So far so good.

But if I try to touch any memory after the first 2 bytes (1 unicode char) which was the size of the thing orionally it crashes

EDIT - Forgot the terminator 2 bytes

[Arnoutdv]

I'm using plain VB6 code to deal with large string (a StringBuilder class)
No fancy memory APIs and pointer managed.
I also deal with huge strings sometimes (text files in memory).
The critical point is when increasing the StringBuilder buffer, then you need twice the memory.

A string can have a maximum size of 2GB.
How do you allocate new memory space when exceeding the for example the 1GB buffer?
Do you double the capacity or increase it in smaller steps?

[vbwins]

So it been running for a few hours and just stopped in Windbg trying to allocate a plain old dynamic string

unCompressedString = Space$(containedUncompressedLen + 256000)

where containedUncompressedLen = 22884489

It got passed a compressed (zlib) string of length 1340757. The compressed string has the previously uncompressed length in the header in plain text. 22884489.

All it was trying to do is make space for the decompression.

For what its worth Task Manager committed memory is about 1.4 gig for the process.

[Arnoutdv]

If containedUncompressedLen = 22.884.489 then the string space allocated is: 2 * (containedUncompressedLen + 256.000)
But why are you using Strings instead of byte arrays?

[vbwins]

If containedUncompressedLen = 22.884.489 then the string space allocated is: 2 * (containedUncompressedLen + 256.000)
But why are you using Strings instead of byte arrays?

It's unicode data just comes from another app over a network and it needs to be parsed in memory. A byte array being returned is possible but sooner or later it needs to be a string.

I am hoping someone will chime in on the heap thing. Been playing with it today with partial success

[fafalone]

Perhaps you should look into using higher memory ranges since most systems have more than 2 or 4GB these days, using AWE. Another advantage of this method is you can use it to guarantee the memory isn't paged out to disk.

[Elroy]

If it's got to run on Windows 32-bit (as well as Windows WoW 64-bit), then stuffing the strings into any "far memory" isn't going to help any. And neither is LAA compiling/linking.

All the memory a 32-bit Windows machine will address is 4GB, so let's assume you've got one of these 32-bit Windows machine packed with memory. The Windows OS is going to take a sizeable chunk, and then all your "in-memory" services and start-up apps are going to take another sizable chunk. In total, that's probably, in some cases, going to approach 2GB. So, all that's left for VB6 is the remaining 2GB, which can be addressed with no concern about LAA linking.

Also, that entire 2GB is available to VB6 without any concerns of far memory, or private heap, or anything else. So, if you're getting out of memory (or out of string space) errors, about the only alternative is to start writing things to disk, and then reading pieces as you need them. That's sort of why we have disks in the first place.

---

In my primary application, I never had out of memory/string errors, but I was concerned about it. So, for all the 100s (if not 1000s) of places where I might show a MsgBox, I wrote a function that fetches the caption string from an MDB file. All these strings are just cross-referenced with a number (a Long). Because it's a MsgBox (which are basically never processor speed sensitive), there's no slowdown in my program whatsoever.

If you want me to, I'll post the little function that fetches these strings from the MDB file. Here's the gist of it, but I can post a more detailed explanation of it:

Public Function DbTxt(ID As Long, ParamArray sArgs() As Variant) As String
    ' If sArgs are passed, make SURE they're all strings.
    '
    Dim i As Long
    '
    rsStrings.Seek "=", ID
    If rsStrings.NoMatch Then
        If InIDE Then
            MsgBox "IDE Development Error!!!  The Strings.mdb message " & Format$(ID) & " wasn't found."
        End If
        Exit Function
    End If
    DbTxt = rsStrings![English]
    '
    ' And now see if we've got any replacements to make.
    For i = LBound(sArgs) To UBound(sArgs)
        DbTxt = Replace$(DbTxt, "%%", sArgs(i), 1&, 1&, vbBinaryCompare)
    Next
    '
    ' The following is necessary because MS-Access strips trailing spaces and vbCrLf from any Text/Memo fields.
    ' It's a bit unusual, but it does happen, particularly trailing vbCrLf occurrences.
    If InStr(DbTxt, "¶") <> 0 Then DbTxt = Replace$(DbTxt, "¶", vbCrLf, 1&, -1&, vbBinaryCompare)
    If InStr(DbTxt, "§") <> 0 Then DbTxt = Replace$(DbTxt, "§", " ", 1&, -1&, vbBinaryCompare)
End Function

Oh, and just to say it, yeah, virtual memory may kick in to give us a bit more than 2GB, but, if we're that close, we truly need to be writing/reading stuff to/from disk.

[xiaoyao]

You can try to use the memory mapping function.
For example, create 10 processes, each of which creates a memory file object.
And then write the string into it.

[vbwins]

LAA is useful on 64 bit though which is why I use it.

I want to avoid disk if possible for performance reasons.

Does anybody know what "out of string space" actually means? Is it a simple heap allocation failure? Is it because of heap fragmentation? Is there some special area of memory VB reserves for string?

What is the difference between "out of string space" and "out of memory"?

If I can understand this it may give me a clue as to something else I can change.

Thanks for the suggestions I will consider them all.

[vbwins]

Does anybody have an idea on the origional question. Perhaps The Trick?

[Elroy]

CSharedMemory-class-for-dynamic-memory-allocation-in-shared-memory

MMapper-Memory-Mapped-File-Demo

Arrays-in-Far-Memory

You can use this "far memory" as shared memory across different processes, but it can also just be used by a single process. Again, this is only going to help on 64-bit systems, as it is a way for VB6 to address memory out of the 32-bit 4GB range.

Also, as a note on LAA, any third-party DLLs and/or OCXs you use, be sure they're all LAA compatible as well. Or, as Dilettante has previously stated, you're setting a time-bomb for yourself. When specifying LAA, you're not "just" specifying it for your source code. You're also specifying it for anything your source code calls as well.

[vbwins]

Thanks will take a look. All the code is mine. I tried over negative addresses being returned by some small c++ items years ago but thanks for the heads up.

I will check out your post above. I think however whioe sleeping I figured out a way to make tye private heap thing work. Will try Monday and report back.

[vbwins]

Took a look at arrays-in-far memory.

How do you specify that CreateFileMapping will use memory outside our 32 bit address space? I dont see that in the code.

[Elroy]

Took a look at arrays-in-far memory.

How do you specify that CreateFileMapping will use memory outside our 32 bit address space? I dont see that in the code.

https://learn.microsoft.com/en-us/wi...tefilemappinga

All I can do is refer you to Microsoft's reference. And to say, "it just does".

That's why we get two Long arguments (instead of just one) to address that memory (dwMaximumSizeLow & dwMaximumSizeHigh). It takes two (or a 8 byte LongLong) to address a full 64-bit memory address space.

Also, the fact that this memory can be shared by two (or more) separate processes is further proof that it's outside the address space of a single VB6 program.

Vbwins, good luck with your project.

[Elroy]

Also, here's a very similar project developed by Eduardo:

https://www.vbforums.com/showthread....fit-in-memory)

His is nice in that he has a variable length string option. And he also broke the task up into several independent CLS files, so you can pick and choose according to your needs.

[Niya]

What is the difference between "out of string space" and "out of memory"?

I'd very much like to know the answer to this as well. Strings are really just binary data interpreted as text so it uses memory just like any other kind of binary data. This error suggests that there is some abstraction in the VB6 runtime that treats strings differently in memory.

[Elroy]

I'd very much like to know the answer to this as well. Strings are really just binary data interpreted as text so it uses memory just like any other kind of binary data. This error suggests that there is some abstraction in the VB6 runtime that treats strings differently in memory.

I'll listen in for an answer as well.

It might be something as simple as what caused it to run out (the very last thing to happen): A request to create a new string? or, A request for other kinds of memory (local variables, etc). But I really don't know.

[Niya]

I'll listen in for an answer as well.

It might be something as simple as what caused it to run out (the very last thing to happen): A request to create a new string?

I suspect something like this as well.

I also briefly took a look at the SysAllocString family of functions and they return NULL on the condition that there is insufficient memory. This may mean that a VB6 "Out of string space" is really just saying that a SysAllocString call has failed in the VB6 runtime.

[vbwins]

I suspect something like this as well.

I also briefly took a look at the SysAllocString family of functions and they return NULL on the condition that there is insufficient memory. This may mean that a VB6 "Out of string space" is really just saying that a SysAllocString call has failed in the VB6 runtime.

Heap alloc and virtualalloc also return null although you can configure heap allocations to throw on out of memory. On that note there is a string builder class using heaps that gave me some food for thought although it has a serious bug jn the alloc routine which returns a null pointer if the allocation fails and then tries to copy data to that pointer.

Boom.

I must add a note to the code bank entry.

[vbwins]

@elroy
Thanks will take a look. I have actually used CreateFileMapping and MapViewOfFile in another project and I thought the address space was inside process limits. Sharing is simply a matter of assigning the same physical pages to virtual pages in multiple processes page tables. In that way virtual memory points to the same physical memory. I remember this from my days as a vms internals support guy.

[Niya]

Heap alloc and virtualalloc also return null although you can configure heap allocations to throw on out of memory.

The SysAllocString family of functions allocate BSTR strings which is what VB6 uses as it's native string type. They do a little more than simply allocate memory. It would make sense to throw an "out of string space" error when a call to one of these functions fail. It would more accurately reflect that we are dealing with a higher level abstraction, in this case, a BSTR. That's my instinct anyway.

[vbwins]

The SysAllocString family of functions allocate BSTR strings which is what VB6 uses as it's native string type. They do a little more than simply allocate memory. It would make sense to throw an "out of string space" error when a call to one of these functions fail. It would more accurately reflect that we are dealing with a higher level abstraction, in this case, a BSTR. That's my instinct anyway.

Yes I thing you are right. I will catch the error and go back to view the dissambly in windbg which at least will show me what is being called. Tye arguments are beyond me,

[xiaoyao]

@elroy
Thanks will take a look. I have actually used CreateFileMapping and MapViewOfFile in another project and I thought the address space was inside process limits. Sharing is simply a matter of assigning the same physical pages to virtual pages in multiple processes page tables. In that way virtual memory points to the same physical memory. I remember this from my days as a vms internals support guy.

In this way, you can share your ah, a string variable or a string array.It's just that the best way is to just use it to read, not to use the string destruction method directly.