VB6 SP6 IDE array allocation limit

[FWinds]

Hi everyone,

Is there any known workaround to the apparent limit of array allocation in VB6 (SP6) IDE?

For example, this one will fail with an "Out of memory" error in IDE:

Option Explicit
Private sngArray() As Single
Private Sub Form_Load()
    ReDim sngArray(150000000) '150M
End Sub

But the compiled .exe will run this fine:

Option Explicit
Private sngArray() As Single
Private Sub Form_Load()
    ReDim sngArray(400000000) '400M
End Sub

Thanks in advance!

edit: Compiling to P-Code will also successfully run the 400M test

[Arnoutdv]

I believe the total amount of memory which can be used is 1GB.
When running in the IDE this also include the VB6.exe process itself.

[LaVolpe]

Honestly don't know the answer definitively. My guess is it may have to do with the way VB allocates memory during IDE (for tracking purposes or other purposes) vs. when compiled, stand-alone executable. Just a guess & depending on the hardware/paging, different systems may give you different limits?

Think another question may be, is there a better way to access your data vs declaring 0.6 to 1.6 GB of contiguous bytes of memory for an array?

[FWinds]

Thanks for the replies.

Definitely the problem seems to be the declaration of contiguous bytes and not the total bytes allocated.

On a 12 GB machine, I've been able to allocate at least 1.5GB of bytes, which runs fine inside the IDE using e.g.

Option Explicit
Private Type udtSngArray
    sngArray() As Single
End Type
Private sngArrays() As udtSngArray

Private Sub Form_Load()
  Dim i As Long
    ReDim sngArrays(1 To 20000)
    For i = 1 To 20000
        ReDim sngArrays(i).sngArray(1 To 20000)
    Next i
End Sub

Most likely the IDE introduces some kind of overhead or limitation, that prohibits contiguous memory allocation, but I have not managed to find any documentation on this issue.

Also, interestingly enough, fixed-size arrays seem to have a much bigger toll inside the VB6 IDE. For example this one fails with "Out of memory" in VB6 IDE:

Option Explicit
Private sngArray(13000000) As Single 'Just 13M

But this will happily run compiled:

Option Explicit
Private sngArray(400000000) As Single '400M

[Arnoutdv]

Okay, but what do you want to accomplish?

[FWinds]

Okay, but what do you want to accomplish?

Without going into details, the basic problem is getting a quite big database table into memory and performing some analysis (multicriteria optimization) on the data.

Table can typically be 0.5 to 2 million rows with about 50 to 200 columns of 32bit float datatype. So upper bound is about 400 million float variables.

The application is performing "random" access to the data into several nested loops, so having the data reside into RAM is several orders of magnitude faster than a buffered I/O approach.

A 1D or 2D array of 400M elements just does not seem to work in the IDE, so the easiest modification option seems to be to store each column in a separate array inside a UDT.

[Arnoutdv]

I should have asked this first, why does it also have to work in the IDE?

[Schmidt]

Without going into details, the basic problem is getting a quite big database table into memory and performing some analysis (multicriteria optimization) on the data.

Table can typically be 0.5 to 2 million rows with about 50 to 200 columns of 32bit float datatype.

Cannot imagine a kind of calculation, which woud involve *all* of the "50-200 columns"
in a formula or something.

Often there's not even all of the rows needed (unless you perform some kind of aggregates like Sum or Avg -
but that is usually done better at the SQL-Level).

So, what about specifying only the concrete amount of columns, which you will need in a certain calculation?
I'm quite sure you will not need all of them to calculate something.

Without more info about your current problem (the involved formulas, and why they need to be
performed on all the rows - and all the columns), we cannot really help.

When working with such large amounts of DB-Data, it becomes more and more important,
to specify and Select "only the needed Sub-Set" (for a given, smaller task).

Olaf

[FWinds]

I should have asked this first, why does it also have to work in the IDE?

Just for debugging purposes.

[FWinds]

Cannot imagine a kind of calculation, which woud involve *all* of the "50-200 columns"
in a formula or something.

Often there's not even all of the rows needed (unless you perform some kind of aggregates like Sum or Avg -
but that is usually done better at the SQL-Level).

So, what about specifying only the concrete amount of columns, which you will need in a certain calculation?
I'm quite sure you will not need all of them to calculate something.

Without more info about your current problem (the involved formulas, and why they need to be
performed on all the rows - and all the columns), we cannot really help.

When working with such large amounts of DB-Data, it becomes more and more important,
to specify and Select "only the needed Sub-Set" (for a given, smaller task).

Olaf

Let's just say that each row is an "agent" or "individual" and the columns are several attributes of that individual that can change, depending on the elapsed time, interactions with other individuals or due to an event happening. It is similar to a simulation and yes, all data are relevant at all times and may change at every iteration.

[Schmidt]

Let's just say that each row is an "agent" or "individual" and the columns are several attributes of that individual that can change, depending on the elapsed time, interactions with other individuals or due to an event happening. It is similar to a simulation and yes, all data are relevant at all times and may change at every iteration.

Reads like an "online-game"-scenario (with that amount of "2Mio. individuals, which can possibly interact").

So maybe we should get a bit more concrete - and look at the problem from an (Online-Clients) side.

A certain Client (one of the individuals) will have "a scene" on the screen -
and this scene involves coordinates which place it in the "game-world" ...

Now the entities the client-avatar in the game can "interact with" are limited already
by those scene-coordinates.

So you will only need to consider other "objects", which are "near the given coordinates" -
and perform appropriate (fast, indexed) Range-Selects against your "players-table" -
against your "monsters-table" - and against your "mystical-objects-table" - and select
only those, which are "near the scene-coords" - and you will end up with perhaps
10 or 20 entities of each of the three groups.

Now the interaction among those 20 other individuals, 20 monsters and 20 mystical objects
is quite possible to handle with a certain amount of CPU-Power on the client (even per JavaScript).

If your simulation is not like in the above scenario, you can surely find other "limiting factors"
by which you could reduce the amount of "interacting entities", which are considered
"to be able to interact at all".

Olaf

[FWinds]

Reads like an "online-game"-scenario (with that amount of "2Mio. individuals, which can possibly interact").

So maybe we should get a bit more concrete - and look at the problem from an (Online-Clients) side.

A certain Client (one of the individuals) will have "a scene" on the screen -
and this scene involves coordinates which place it in the "game-world" ...

Now the entities the client-avatar in the game can "interact with" are limited already
by those scene-coordinates.

So you will only need to consider other "objects", which are "near the given coordinates" -
and perform appropriate (fast, indexed) Range-Selects against your "players-table" -
against your "monsters-table" - and against your "mystical-objects-table" - and select
only those, which are "near the scene-coords" - and you will end up with perhaps
10 or 20 entities of each of the three groups.

Now the interaction among those 20 other individuals, 20 monsters and 20 mystical objects
is quite possible to handle with a certain amount of CPU-Power on the client (even per JavaScript).

If your simulation is not like in the above scenario, you can surely find other "limiting factors"
by which you could reduce the amount of "interacting entities", which are considered
"to be able to interact at all".

Olaf

Hi Olaf,

Thanks for your reply! The online game analogy is great. My case is a crowd microsimulation model, so individuals are just AI agents, but the concepts of proximity and limiting factors definitely apply. My goal is for the simulation to run as fast as possible and anything that involves database operations, however optimized, is having a significant performance impact. Using a big array it is also straight forward to dump the current contents of the array(s) into a binary file and continue from a stored scenario at a later time, even if no access to the actual database exist. In addition, in order to have concurrent processes /.exes running with different settings would require to have as many tables in the database as the number of simulation processes, as UPDATEs would be unique for each process.

But implementation details aside, the original observation still stands, VB6 IDE has issues when allocating contiguous amounts of memory compared to a standalone exe and I would be interested to find any resources / background on that.

[Arnoutdv]

But implementation details aside, the original observation still stands, VB6 IDE has issues when allocating contiguous amounts of memory compared to a standalone exe and I would be interested to find any resources / background on that.

But even if you find some other experiences or even documentation which confirms what you already experienced, how will that help you?
I can only advice you to run simulations in the IDE using a smaller population, the size of the population should not be influencing the correctness of your application.

[FWinds]

OMG!! I just did the unthinkable and it WORKED!!!!

I patched VB6.exe so that /LARGEADDRESSAWARE is set to true (this can be done for example by the usage of a PE Editor tool such as CFF Explorer or 4gb_patch). And magically.... everything works as expected (can allocate slightly less than 540M elements single array in vb IDE, either fixed size or dynamic)... of course a 64bit OS would be needed to take advantage of it.

I can't believe it... Woot!!!

[dilettante]

I suspected as much but didn't follow through to try it. There might be side effects if the p-code engine relies on the upper bit to mark addresses, but I don't expect that in VB6. VB5 maybe.

I doubt 64-bit Windows makes any difference since (a.) we have this thing called virtual memory we use, and (b.) even 32-bit Windows can use far more than 2GB of RAM depending on the CPU, BIOS, and some other factors. The machine I'm on now has far more:

Private Type MEMORYSTATUSEX
    dwLength As Long
    dwMemoryLoad As Long                'Number between 0 and 100 that specifies the
                                        'approximate percentage of physical memory
                                        'that is in use.

    ullTotalPhys As Currency            'Amount of actual physical memory.

    ullAvailPhys As Currency            'Amount of physical memory currently available.

    ullTotalPageFile As Currency        'Current committed memory limit for the system
                                        'or the current process, whichever is smaller.

    ullAvailPageFile As Currency        'Maximum amount of memory the current process
                                        'can commit.

    ullTotalVirtual As Currency         'Size of the user-mode portion of the virtual
                                        'address space of the calling process.

    ullAvailVirtual As Currency         'Amount of unreserved and uncommitted memory
                                        'currently in the user-mode portion of the
                                        'virtual address space of the calling process.

    ullAvailExtendedVirtual As Currency 'Reserved.  Currently always 0.
End Type

Private Declare Sub GlobalMemoryStatusEx Lib "kernel32" (ByRef Buffer As MEMORYSTATUSEX)

Mulitply the Currency items by 10000@ to get whole numbers (they're really 64-bit integers).

[FWinds]

Hi,

Yes, you are right - 32 bit or 64 bit Windows should not matter. I am running some tests now with as much RAM utilization as possible inside the IDE and see if I encounter any unusual behavior.



Edit: Seems to be working like a charm - I have found no side effects of setting LARGEADDRESSAWARE to True, so far.

[Arnoutdv]

See this article by Karl E. Peterson, by changing the LARGEADDRESSAWARE bit in the header then even on Win7 32bits you can address more memory.