Is there a 2 GB run-time size limit on a VB6 application?

[jfhoff]

I'm trying to declare an array in a vb6 application that is a little larger than 2GB in size; it gets an "out of memory" error at run time. I distilled the application down to a single Dim (removing all code), as shown below:

' "comment out" one of these two:
'   Private A(1 To &H65F00000) As Byte ' works with this active
    Private A(1 To &H66000000) As Byte ' fails with this active

The app runs when the first line is used, and fails at run time (NOT at compile time) with an "out of memory" message when the second line is used.

It looks like there is some kind of 2GB limit on the size of an application at run time.
It doesn't help if I divide the array in half and put the two arrays in separate modules.
It doesn't help if I assign the arrays dynamically at run time.

I looked here for some specification from Microsoft, but saw nothing pertinent.

Am I correct that there is a 2GB limit on application size? Where is it specified?

What is the easiest workaround?

• Use a system call to allocate memory and address it manually (would this be exempt from the 2GB limit?)
• Call two separate custom DLLs, each containing half of the array
• Some other technique I haven't thought of

[DataMiser]

Maybe something here would be useful?
http://www.codeguru.com/vb/gen/vb_mi...tores-Data.htm

[techgnome]

It's not an application size limit... the app doesn't change size because an array is dimmed at 100 vs 10000. If I remember right, it's actually a memory address limit... and it's because the largest memory address VB6 can deal with is a 32-bit address, which maxes out around 2Gig...

But then again, I could be wrong about that...

-tg

[Bonnie West]

... it gets an "out of memory" error at run time.

How much is your PC's RAM + Virtual Memory (page file)?

I looked here for some specification from Microsoft, but saw nothing pertinent.

Am I correct that there is a 500MB limit on application size? Where is it specified?

According to Arrays:

The upper bound cannot exceed the range of a Long data type (-2,147,483,648 to 2,147,483,647).

[jfhoff]

Thanks Bonnie. You're right about the 2GB max size on an array, but then why, when I use a static size of over 2GB, doesn't it error out at compile time? Also, it fails below the 2GB point (see the distilled one-line code example above). Thirdly, it doesn't help if I use two arrays so that the size of each array is within the limit.

This fails:

Private A(1 To &H50000000) As Byte
Private B(1 To &H50000000) As Byte

[Bonnie West]

Are you sure your PC's RAM can satisfy your app's memory allocation requests? The error description you've given sounds like it can't.

[jfhoff]

I have 6GB of RAM, lightly used. Also, I can start 3 instances of the app (< 2GB) with no problems; I just can't have one instance with more than something less than 2GB of memory used.

[Bonnie West]

I see. However, at this point, I have no idea why your app fails to create such a huge array at run time (even though it is still within the limits). Could you describe your program a bit more? What are you using the enormous Byte array for?

[jfhoff]

The array in the actual application is a table containing about 60 bytes of data for each of 32M accounts (If you've heard of Bitcoin, this is a client for an alternate incarnation). The accesses are relatively infrequent, but the response must be fast.

[Niya]

The array in the actual application is a table containing about 60 bytes of data for each of 32M accounts...

I take it 32M means 32 million ? Why aren't you using a database engine like Access or SQL Server for this ?

[Bonnie West]

Have you tried a 2-dimensional Byte array? Your first workaround in post #1 (allocate memory via API) might work in conjunction with a SAFEARRAY hack.

[jfhoff]

I take it 32M means 32 million ? Why aren't you using a database engine like Access or SQL Server for this ?

Didn't consider it. Would it be as fast?

[jfhoff]

Have you tried a 2-dimensional Byte array? Your first workaround in post #1 (allocate memory via API) might work in conjunction with a SAFEARRAY hack.

Yes, in fact my actual implementation is a 2-dimensional array (0 to &H1FFFFFF, 0 to 49).

[Niya]

Didn't consider it. Would it be as fast?

Nothing is as fast as reading from system memory except reading from CPU registers which is way faster.

[HanneSThEGreaT]

Don't know if I'm too late for this party ( I brought beer, though ) but here is an interesting article about the 2GB file limit and getting past it :

http://www.codeguru.com/cpp/w-d/doc_...File-Limit.htm

[Phill.W]

No, it wouldn't be. Not even close.
But then if you were thinking "database", then you wouldn't even consider reading all 32 million items in single hit anyway! You'd only read what you needed to read. So ...

Take a step away from the keyboard, take a deep breath and then tell us why you want to read 32 million of anything into memory.

(Please don't say "drop-down list" )

Regards, Phill W.

[Tanner_H]

Hi jfhoff. On 32-bit systems, Windows itself imposes a 2gb per-process limit on memory usage. This is regardless of programming language, available RAM, or pagefile size. See here for details:

http://msdn.microsoft.com/en-us/libr...8VS.85%29.aspx

That article links to a topic called 4GT, or "4-gigabyte tuning", which is a way to increase the limit to 3gb per-process:

http://msdn.microsoft.com/en-us/libr...=vs.85%29.aspx

"To enable 4GT, use the BCDEdit /set command to set the increaseuserva boot entry option to a value between 2048 (2 GB) and 3072 (3 GB). "

It's not a great solution for end-users, but for your PC, it will increase the amount of addressable space by another gigabyte, which may be helpful.

[Zvoni]

As far as i understand it: You don't get an compile-error because at compile-time you're not assigning the memory. You're assigning it at Run-Time.
It's the same as with any other variable: You can declare it, but if you never use it, you don't pay the penalty for used memory.

There is also something else to think about (as far as i remember it): You can even run into the Out-of-Memory problem with distinctly smaller arrays, if your memory is too "cluttered", since an array
needs a consecutive memory-block. An array is actually a collection of memory-adresses, which point to the respective members of the array, and every adress of an array-member is 4 bytes farther away than the previous member
Address of MyArray(1) = 12345678
Address of MyArray(2) = 12345682 (Address of MyArray(1) + 4)
Address of MyArray(3) = 12345686 (Address of MyArray(2) + 4)
and so on....

[Niya]

...You can even run into the Out-of-Memory problem with distinctly smaller arrays, if your memory is too "cluttered", since an array
needs a consecutive memory-block.

The word you're looking for is "fragmented" not "cluttered".

[Zvoni]

Yepp. That's the one ^^

[jfhoff]

Tanner_H, thanks for your information. That exactly answers the question I was asking, though the answer is disappointing. I can't use 4GT because, as you say, it's not a great solution for end-users. I can't use system calls to allocate more memory because they would be using the same 2GB-limited address space. A database wouldn't work, because it would still store the data in a file and access it as required (which is really slow). I can't use a DLL because, once again, it shares the same address space. The only possible solution I see is to launch two more processes, one for each half of the data array (thus bringing each below 2GB). The problem there is that the data can't be accessed without swapping to another process, which could be much slower. Oh, well, I'll have to find some way to live with one of the above options...

[Tanner_H]

Yeah, sorry for the bad news.

Let me provide one more clarification, just in case people read this thread in the future and want to activate 4-gigabyte tuning for their own PC or VB project. Even if you enable support for this at the OS level, Windows won't automatically allow applications access to 3gb of memory. Per this link (http://msdn.microsoft.com/en-us/libr.../gg487508.aspx):

It would be ineffective to automatically provide every application with a 3-GB address space.

Executables that can use the 3-GB address space are required to have the bit IMAGE_FILE_LARGE_ADDRESS_AWARE set in their image header. If you are the developer of the executable, you can specify a linker flag (/LARGEADDRESSAWARE).

To set this bit, you must use Microsoft Visual Studio Version 6.0 or later and the Editbin.exe utility, which has the ability to modify the image header (/LARGEADDRESSAWARE) flag. For more information on setting this flag, see the Microsoft Visual Studio documentation.

Some manufacturers preconfigure their applications to use application memory tuning, making it unnecessary for you to make this change. For more information, see your application documentation and contact your application vendor to determine whether they support Large Address Awareness or whether you can enable it in their application.

[Niya]

The only possible solution I see is to launch two more processes, one for each half of the data array.

Ok, this is veering off into the absurd now.

A database wouldn't work, because it would still store the data in a file and access it as required (which is really slow)

Works well enough for every website in the world. Works well enough for banks. Works well enough for just about every accounting software on the planet. Works well enough for every computerized POS system on Earth. Works well enough for the CIA, FBI, and the NSA.

Save yourself all this headache and use a DB.

[jfhoff]

Niya,

Works well enough for every website in the world. Works well enough for banks. Works well enough for just about every accounting software on the planet. Works well enough for every computerized POS system on Earth. Works well enough for the CIA, FBI, and the NSA. Save yourself all this headache and use a DB.

File accesses are very slow, compared to an in-memory array access, limited by the spin rate of the disk, plus many other system factors. I'm trying to find a way to access a large body of data at speeds comparable to a RAM access.

[szlamany]

Back in my mainframe/mini days we had what was called "virtual arrays" that actually resided on disk.

You could easily fashion the same thing with a CLASS that "looks like an array" - but actually "loads" a "smaller" memory array with disk contents when those array slots are asked for.

So you have half the array in memory (or even less) and "swap" in and out the data to disk - this is actually a really common practice from prior days.

But you have to tell us more - do you run the array "sequentially"? Jump all over the place?

Give us a bit more info on what you are really doing with this data.

Also - I find it hard to believe that you chose VB6 as a platform to do this analysis - I would think if you really needed speed and performance you would go to C++. With that you would get an exponential increase in performance and maybe the disk access would be of less importance.

[jfhoff]

szlamany, This is a client for a real-time distributed system having up to 32 million nodes. Every client retains a copy of every node's data (~50-60 bytes per node), and all nodes concurrently update the ledger-database. I'm shooting for updates at a 10-second rate, with up to 10K nodes submitting updates during that 10-second period. The accesses to the array are totally random. More information is here. As the paper describes, each node is linked to hundreds of other nodes in a very regular pattern, so that updates can be distributed to all nodes in very few "hops" (for 32M nodes, 5 hops are necessary).

I am an electrical engineer, not a programmer, by training, but I'm quite strong in VB6 (and some C++6). Looks like it's time for me to bite the bullet and graduate to a 21st-century language.

[Tanner_H]

jfhoff: I feel it's important to note that there is no "magical" benefit in switching to another development language for this task. All 32-bit programs - regardless of language - are subject to the 2gb limit on Windows. That's a Windows limit, not a VB6 limit.

If you were to switch to a language with 64-bit support, that is a different story, but then you would be limiting your client base to users of 64-bit versions of Windows, only - maybe this is acceptable to you, maybe not. Something to think about.

If you are serious about writing a Bitcoin or Bitcoin-like client, you'd be better off investigating ways to offload the calculation work onto the GPU instead of the CPU. (The parallellism of bitcoin-type calculations makes this a MUST if you are serious about building your own client.) Among other benefits, assuming you are familiar with GPU programming, that would allow you to tap into much higher-bandwidth GDDR memory, and if you could pool both video RAM and regular RAM, that would help.

At the end of the day, though, you'd really be better off with some kind of hashing system that would enable you to reduce the program's memory consumption needs. Regardless of development language, you generally don't want to be monopolizing more than 2gb of RAM. I would investigate how various bitcoin clients handle this, because unless you want to limit yourself to a very small number of users, the 2gb problem is something you will eventually need to solve.

[szlamany]

I've not tried - but maybe with C++ you can "address" a chuck of memory the size of what you need (32 million * 60 bytes - right).

I do large scale document analysis with C++, calling those unmanaged functions from VB.Net (for the convenience of loading the file into memory and such)...

[szlamany]

Read this thread before switching to VB.Net - I've already been down the path of maximum array sizes in .Net

http://www.vbforums.com/showthread.p...o-s&highlight=

[jfhoff]

Tanner_H:

If you are serious about writing a Bitcoin or Bitcoin-like client, you'd be better off investigating ways to offload the calculation work onto the GPU instead of the CPU. (The parallellism of bitcoin-type calculations makes this a MUST if you are serious about building your own client.) Among other benefits, assuming you are familiar with GPU programming, that would allow you to tap into much higher-bandwidth GDDR memory, and if you could pool both video RAM and regular RAM, that would help.

The GPU suggestion sounds really good, but wouldn't that restrict the client base even more than a 64-bit Windows requirement? And wouldn't that mean programming for various GPU's (or do they use a common programming language)?

[Tanner_H]

Tanner_H:

The GPU suggestion sounds really good, but wouldn't that restrict the client base even more than a 64-bit Windows requirement? And wouldn't that mean programming for various GPU's (or do they use a common programming language)?

https://en.wikipedia.org/wiki/Opencl

[Niya]

I'm trying to find a way to access a large body of data at speeds comparable to a RAM access.

Well if you insist.

The only sensible way to do this without worrying about any size limits is by paging. Store your data on a file on harddisk and then device a system that can swap the data in and out of memory as you access specific records. The challenge here would be to find the optimal page size to gain the best performance. Then you have to consider other things like locality of reference which also needs to be factored in for getting the best performance. This is the about the best option you have.

[szlamany]

Back in my mainframe/mini days we had what was called "virtual arrays" that actually resided on disk.

You could easily fashion the same thing with a CLASS that "looks like an array" - but actually "loads" a "smaller" memory array with disk contents when those array slots are asked for.

So you have half the array in memory (or even less) and "swap" in and out the data to disk - this is actually a really common practice from prior days.

As I said back in post #25...

[Niya]

Ah, I seem to have missed that bit

[szlamany]

Great minds think alike

[Niya]

.....