dcsimg
Results 1 to 17 of 17

Thread: VB6 SP6 IDE array allocation limit

  1. #1

    Thread Starter
    New Member
    Join Date
    Mar 2015
    Posts
    8

    VB6 SP6 IDE array allocation limit

    Hi everyone,

    Is there any known workaround to the apparent limit of array allocation in VB6 (SP6) IDE?

    For example, this one will fail with an "Out of memory" error in IDE:

    Code:
    Option Explicit
    Private sngArray() As Single
    Private Sub Form_Load()
        ReDim sngArray(150000000) '150M
    End Sub
    But the compiled .exe will run this fine:

    Code:
    Option Explicit
    Private sngArray() As Single
    Private Sub Form_Load()
        ReDim sngArray(400000000) '400M
    End Sub
    Thanks in advance!

    edit: Compiling to P-Code will also successfully run the 400M test
    Last edited by FWinds; Mar 16th, 2015 at 02:56 PM. Reason: additional info

  2. #2
    PowerPoster Arnoutdv's Avatar
    Join Date
    Oct 2013
    Posts
    3,820

    Re: VB6 SP6 IDE array allocation limit

    I believe the total amount of memory which can be used is 1GB.
    When running in the IDE this also include the VB6.exe process itself.

  3. #3
    VB-aholic & Lovin' It LaVolpe's Avatar
    Join Date
    Oct 2007
    Location
    Beside Waldo
    Posts
    18,993

    Re: VB6 SP6 IDE array allocation limit

    Honestly don't know the answer definitively. My guess is it may have to do with the way VB allocates memory during IDE (for tracking purposes or other purposes) vs. when compiled, stand-alone executable. Just a guess & depending on the hardware/paging, different systems may give you different limits?

    Think another question may be, is there a better way to access your data vs declaring 0.6 to 1.6 GB of contiguous bytes of memory for an array?
    Insomnia is just a byproduct of, "It can't be done"

    Classics Enthusiast? Here's my 1969 Mustang Mach I Fastback. Her sister '67 Coupe has been adopted

    Newbie? Novice? Bored? Spend a few minutes browsing the FAQ section of the forum.
    Read the HitchHiker's Guide to Getting Help on the Forums.
    Here is the list of TAGs you can use to format your posts
    Here are VB6 Help Files online


    {Alpha Image Control} {Memory Leak FAQ} {Unicode Open/Save Dialog} {Resource Image Viewer/Extractor}
    {VB and DPI Tutorial} {Manifest Creator} {UserControl Button Template} {stdPicture Render Usage}

  4. #4

    Thread Starter
    New Member
    Join Date
    Mar 2015
    Posts
    8

    Re: VB6 SP6 IDE array allocation limit

    Thanks for the replies.

    Definitely the problem seems to be the declaration of contiguous bytes and not the total bytes allocated.

    On a 12 GB machine, I've been able to allocate at least 1.5GB of bytes, which runs fine inside the IDE using e.g.

    Code:
    Option Explicit
    Private Type udtSngArray
        sngArray() As Single
    End Type
    Private sngArrays() As udtSngArray
    
    Private Sub Form_Load()
      Dim i As Long
        ReDim sngArrays(1 To 20000)
        For i = 1 To 20000
            ReDim sngArrays(i).sngArray(1 To 20000)
        Next i
    End Sub
    Most likely the IDE introduces some kind of overhead or limitation, that prohibits contiguous memory allocation, but I have not managed to find any documentation on this issue.

    Also, interestingly enough, fixed-size arrays seem to have a much bigger toll inside the VB6 IDE. For example this one fails with "Out of memory" in VB6 IDE:

    Code:
    Option Explicit
    Private sngArray(13000000) As Single 'Just 13M
    But this will happily run compiled:

    Code:
    Option Explicit
    Private sngArray(400000000) As Single '400M
    Last edited by FWinds; Mar 16th, 2015 at 06:00 PM.

  5. #5
    PowerPoster Arnoutdv's Avatar
    Join Date
    Oct 2013
    Posts
    3,820

    Re: VB6 SP6 IDE array allocation limit

    Okay, but what do you want to accomplish?

  6. #6

    Thread Starter
    New Member
    Join Date
    Mar 2015
    Posts
    8

    Re: VB6 SP6 IDE array allocation limit

    Quote Originally Posted by Arnoutdv View Post
    Okay, but what do you want to accomplish?
    Without going into details, the basic problem is getting a quite big database table into memory and performing some analysis (multicriteria optimization) on the data.

    Table can typically be 0.5 to 2 million rows with about 50 to 200 columns of 32bit float datatype. So upper bound is about 400 million float variables.

    The application is performing "random" access to the data into several nested loops, so having the data reside into RAM is several orders of magnitude faster than a buffered I/O approach.

    A 1D or 2D array of 400M elements just does not seem to work in the IDE, so the easiest modification option seems to be to store each column in a separate array inside a UDT.

  7. #7
    PowerPoster Arnoutdv's Avatar
    Join Date
    Oct 2013
    Posts
    3,820

    Re: VB6 SP6 IDE array allocation limit

    I should have asked this first, why does it also have to work in the IDE?

  8. #8
    PowerPoster
    Join Date
    Jun 2013
    Posts
    4,884

    Re: VB6 SP6 IDE array allocation limit

    Quote Originally Posted by FWinds View Post
    Without going into details, the basic problem is getting a quite big database table into memory and performing some analysis (multicriteria optimization) on the data.

    Table can typically be 0.5 to 2 million rows with about 50 to 200 columns of 32bit float datatype.
    Cannot imagine a kind of calculation, which woud involve *all* of the "50-200 columns"
    in a formula or something.

    Often there's not even all of the rows needed (unless you perform some kind of aggregates like Sum or Avg -
    but that is usually done better at the SQL-Level).

    So, what about specifying only the concrete amount of columns, which you will need in a certain calculation?
    I'm quite sure you will not need all of them to calculate something.

    Without more info about your current problem (the involved formulas, and why they need to be
    performed on all the rows - and all the columns), we cannot really help.

    When working with such large amounts of DB-Data, it becomes more and more important,
    to specify and Select "only the needed Sub-Set" (for a given, smaller task).

    Olaf

  9. #9

    Thread Starter
    New Member
    Join Date
    Mar 2015
    Posts
    8

    Re: VB6 SP6 IDE array allocation limit

    Quote Originally Posted by Arnoutdv View Post
    I should have asked this first, why does it also have to work in the IDE?
    Just for debugging purposes.

  10. #10

    Thread Starter
    New Member
    Join Date
    Mar 2015
    Posts
    8

    Re: VB6 SP6 IDE array allocation limit

    Quote Originally Posted by Schmidt View Post
    Cannot imagine a kind of calculation, which woud involve *all* of the "50-200 columns"
    in a formula or something.

    Often there's not even all of the rows needed (unless you perform some kind of aggregates like Sum or Avg -
    but that is usually done better at the SQL-Level).

    So, what about specifying only the concrete amount of columns, which you will need in a certain calculation?
    I'm quite sure you will not need all of them to calculate something.

    Without more info about your current problem (the involved formulas, and why they need to be
    performed on all the rows - and all the columns), we cannot really help.

    When working with such large amounts of DB-Data, it becomes more and more important,
    to specify and Select "only the needed Sub-Set" (for a given, smaller task).

    Olaf
    Let's just say that each row is an "agent" or "individual" and the columns are several attributes of that individual that can change, depending on the elapsed time, interactions with other individuals or due to an event happening. It is similar to a simulation and yes, all data are relevant at all times and may change at every iteration.

  11. #11
    PowerPoster
    Join Date
    Jun 2013
    Posts
    4,884

    Re: VB6 SP6 IDE array allocation limit

    Quote Originally Posted by FWinds View Post
    Let's just say that each row is an "agent" or "individual" and the columns are several attributes of that individual that can change, depending on the elapsed time, interactions with other individuals or due to an event happening. It is similar to a simulation and yes, all data are relevant at all times and may change at every iteration.
    Reads like an "online-game"-scenario (with that amount of "2Mio. individuals, which can possibly interact").

    So maybe we should get a bit more concrete - and look at the problem from an (Online-Clients) side.

    A certain Client (one of the individuals) will have "a scene" on the screen -
    and this scene involves coordinates which place it in the "game-world" ...

    Now the entities the client-avatar in the game can "interact with" are limited already
    by those scene-coordinates.

    So you will only need to consider other "objects", which are "near the given coordinates" -
    and perform appropriate (fast, indexed) Range-Selects against your "players-table" -
    against your "monsters-table" - and against your "mystical-objects-table" - and select
    only those, which are "near the scene-coords" - and you will end up with perhaps
    10 or 20 entities of each of the three groups.

    Now the interaction among those 20 other individuals, 20 monsters and 20 mystical objects
    is quite possible to handle with a certain amount of CPU-Power on the client (even per JavaScript).

    If your simulation is not like in the above scenario, you can surely find other "limiting factors"
    by which you could reduce the amount of "interacting entities", which are considered
    "to be able to interact at all".

    Olaf

  12. #12

    Thread Starter
    New Member
    Join Date
    Mar 2015
    Posts
    8

    Re: VB6 SP6 IDE array allocation limit

    Quote Originally Posted by Schmidt View Post
    Reads like an "online-game"-scenario (with that amount of "2Mio. individuals, which can possibly interact").

    So maybe we should get a bit more concrete - and look at the problem from an (Online-Clients) side.

    A certain Client (one of the individuals) will have "a scene" on the screen -
    and this scene involves coordinates which place it in the "game-world" ...

    Now the entities the client-avatar in the game can "interact with" are limited already
    by those scene-coordinates.

    So you will only need to consider other "objects", which are "near the given coordinates" -
    and perform appropriate (fast, indexed) Range-Selects against your "players-table" -
    against your "monsters-table" - and against your "mystical-objects-table" - and select
    only those, which are "near the scene-coords" - and you will end up with perhaps
    10 or 20 entities of each of the three groups.

    Now the interaction among those 20 other individuals, 20 monsters and 20 mystical objects
    is quite possible to handle with a certain amount of CPU-Power on the client (even per JavaScript).

    If your simulation is not like in the above scenario, you can surely find other "limiting factors"
    by which you could reduce the amount of "interacting entities", which are considered
    "to be able to interact at all".

    Olaf
    Hi Olaf,

    Thanks for your reply! The online game analogy is great. My case is a crowd microsimulation model, so individuals are just AI agents, but the concepts of proximity and limiting factors definitely apply. My goal is for the simulation to run as fast as possible and anything that involves database operations, however optimized, is having a significant performance impact. Using a big array it is also straight forward to dump the current contents of the array(s) into a binary file and continue from a stored scenario at a later time, even if no access to the actual database exist. In addition, in order to have concurrent processes /.exes running with different settings would require to have as many tables in the database as the number of simulation processes, as UPDATEs would be unique for each process.

    But implementation details aside, the original observation still stands, VB6 IDE has issues when allocating contiguous amounts of memory compared to a standalone exe and I would be interested to find any resources / background on that.

  13. #13
    PowerPoster Arnoutdv's Avatar
    Join Date
    Oct 2013
    Posts
    3,820

    Re: VB6 SP6 IDE array allocation limit

    But implementation details aside, the original observation still stands, VB6 IDE has issues when allocating contiguous amounts of memory compared to a standalone exe and I would be interested to find any resources / background on that.
    But even if you find some other experiences or even documentation which confirms what you already experienced, how will that help you?
    I can only advice you to run simulations in the IDE using a smaller population, the size of the population should not be influencing the correctness of your application.

  14. #14

    Thread Starter
    New Member
    Join Date
    Mar 2015
    Posts
    8

    Re: VB6 SP6 IDE array allocation limit

    OMG!! I just did the unthinkable and it WORKED!!!!

    I patched VB6.exe so that /LARGEADDRESSAWARE is set to true (this can be done for example by the usage of a PE Editor tool such as CFF Explorer or 4gb_patch). And magically.... everything works as expected (can allocate slightly less than 540M elements single array in vb IDE, either fixed size or dynamic)... of course a 64bit OS would be needed to take advantage of it.

    I can't believe it... Woot!!!
    Last edited by FWinds; Mar 17th, 2015 at 10:15 AM.

  15. #15
    PowerPoster
    Join Date
    Feb 2006
    Posts
    20,958

    Re: VB6 SP6 IDE array allocation limit

    I suspected as much but didn't follow through to try it. There might be side effects if the p-code engine relies on the upper bit to mark addresses, but I don't expect that in VB6. VB5 maybe.

    I doubt 64-bit Windows makes any difference since (a.) we have this thing called virtual memory we use, and (b.) even 32-bit Windows can use far more than 2GB of RAM depending on the CPU, BIOS, and some other factors. The machine I'm on now has far more:

    Name:  sshot.png
Views: 302
Size:  13.5 KB

    Code:
    Private Type MEMORYSTATUSEX
        dwLength As Long
        dwMemoryLoad As Long                'Number between 0 and 100 that specifies the
                                            'approximate percentage of physical memory
                                            'that is in use.
    
        ullTotalPhys As Currency            'Amount of actual physical memory.
    
        ullAvailPhys As Currency            'Amount of physical memory currently available.
    
        ullTotalPageFile As Currency        'Current committed memory limit for the system
                                            'or the current process, whichever is smaller.
    
        ullAvailPageFile As Currency        'Maximum amount of memory the current process
                                            'can commit.
    
        ullTotalVirtual As Currency         'Size of the user-mode portion of the virtual
                                            'address space of the calling process.
    
        ullAvailVirtual As Currency         'Amount of unreserved and uncommitted memory
                                            'currently in the user-mode portion of the
                                            'virtual address space of the calling process.
    
        ullAvailExtendedVirtual As Currency 'Reserved.  Currently always 0.
    End Type
    
    Private Declare Sub GlobalMemoryStatusEx Lib "kernel32" (ByRef Buffer As MEMORYSTATUSEX)
    Mulitply the Currency items by 10000@ to get whole numbers (they're really 64-bit integers).
    Last edited by dilettante; Mar 17th, 2015 at 10:37 AM.

  16. #16

    Thread Starter
    New Member
    Join Date
    Mar 2015
    Posts
    8

    Re: VB6 SP6 IDE array allocation limit

    Hi,

    Yes, you are right - 32 bit or 64 bit Windows should not matter. I am running some tests now with as much RAM utilization as possible inside the IDE and see if I encounter any unusual behavior.

    Name:  vb6_4gb.jpg
Views: 317
Size:  31.9 KB

    Edit: Seems to be working like a charm - I have found no side effects of setting LARGEADDRESSAWARE to True, so far.
    Last edited by FWinds; Mar 17th, 2015 at 11:48 AM.

  17. #17
    PowerPoster Arnoutdv's Avatar
    Join Date
    Oct 2013
    Posts
    3,820

    Re: VB6 SP6 IDE array allocation limit

    See this article by Karl E. Peterson, by changing the LARGEADDRESSAWARE bit in the header then even on Win7 32bits you can address more memory.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Featured


Click Here to Expand Forum to Full Width