[RESOLVED] How can we hold an array has 20 MILLION PARAMETERS in VB?

I am working on a scientific program which have an array with 6 variables that is

tnar(totalD, a2tot, a1tot, b2tot, b1tot, 3)

and each variable has a minimum, maximum and a step values that user can adjust (look at attached picture). When this variables are adjusted sometimes tnar array is reaching over 20 million parameters for example when totalD is 42, a2tot is 31, a1tot is 11, b2tot is 31 and b1tot is 20 then tnar has an 26,638,920 array indices.

but in my pc VB is giving an "Out of memory" error.

is there another way to hold these values? can i do that with Collection?

If they are all on one form, they you could try parent form with frames or something, so you can have say, Form2 content on Form1. I don't know if this will help or not.

this is an array variable that each one has uniqe value and it is not related with forms so

tnar(1,1,1,1,1,1)=1.2345
tnar(1,1,1,1,1,2)=2.3456
tnar(1,1,1,1,1,3)=-0.456
.
.
tnar(1,1,1,1,2,1)=11.239
tnar(1,1,1,1,2,2)=1.8965
.
.
tnar(1,1,1,3,1,1)=-1.233
.
.
and so on.
attached picture is showing values used in formula that each has different start and end values, but VB can not hold about over 20 milion variable at once

one can know the way holding all values in VB

i thing you better post a sample code.
coz can't see how you have declared variables, variable data types etc.

Quote:

---

When this variables are adjusted sometimes ..

---

Adjusted mean ?

my main problem is that vb is giving "out of memory" error

this means vb can not hold information of array or can not handle an array with over 20 million indices

main variable can be double or string and this is not very important, i tried every one but each time i have same error

maybe collection structure can be used but i dont know how could i manage it because i have to hold each parameter used in formula for example

a2 is changing from 0 to 0.03 with step 0.001
a1 is changing from 1.3 to 1.4 with step 0.01 and so on

roughly i colud say that i have a formula like

f(x)=a2*x^2+a1*x+b2*x^2 +b1*x

each value is used in formula and result is stored in variable "tnar"

How about using an UDT?
Or even a class??

Code:

---

PublicType typTnar
   
    totalD As Double
    a2tot As Double
    a1tot As Double
    b2tot As Double
    b1tot As Double
    UnknownVal As Double ' This I dont know what its named as 3???
End Type

public tnar() as typTnar

---

This could be expanded from this simple solution. If its going to be multi dimentional array, in the UDT, make members arrays also

:wave:

Quote:

---

main variable can be double or string and this is not very important,

---

I think it is very important. If its going to hold large data sets, you have to think about memory and performance.

And not to talk about acuracy of calculations it performs. I m no expert on choosing the right data type for you but you have to consider about it.

:wave:

at start of program

Code:

---

      Dim tnar()

---

when other variables are adjusted for new calculation

Code:

---

    a2tot = (maxa2 - mina2) / stepa2
    a1tot = (maxa1 - mina1) / stepa1
    b2tot = (maxb2 - minb2) / stepb2
    b1tot = (maxb1 - minb1) / stepb1

    redim tnar(totalD, a2tot, a1tot, b2tot, b1tot, 3)


    for totalD = 0 to ntot
        a2 = mina2
        for a2Count = 0 to a2tot
              a1 = mina1
              for a1Count = 0 to a1tot
                  b2 = minb2
                  for b2Count = 0 to b2tot
                        b1 = minb1
                        for b1Count = 0 to b1tot
                             
                            tnar(totalD, a2Count, a1Count, b2Count, b1Count, 1) = some calculation ...
                            tnar(totalD, a2Count, a1Count, b2Count, b1Count, 2) = other calculation ...
                            tnar(totalD, a2Count, a1Count, b2Count, b1Count, 3) = another calculation ...

                      b1 = b1 + stepb1
                      next b1Count

                  b2 = b2 + stepb2
                  next b2Count

              a1 = a1 + stepa1
              next a1Count

          a2 = a2 + stepa2
          next a2Count
      next totalD

---

tnar is readjusted again and so i am allocating each variable in formule

Can't follow what you are doing, but sounds to me like a random file might be a good solution. You can read and write records a thousand or a million at a time, which makes your program and memory only deal with a small fraction at a time.

you are right really, it is a scientific calculation and it is very difficult to understand and things that you know is that program is producing an array which has totaly over 20 milion subcripts.

your SUGGESTION maybe a solution but if i could do this i must produce three file and maybe each file will have a size about 10-30MB

Could i read data from each other indepently?

Out of memory probably will occur regardless of how you store the data. One recommendation might be to use a file, store the data to file, and extract pieces of it, JIT (just in time) for the calculations. 20 million equates to what, about 80 million bytes (Longs) or 160 million bytes (doubles). Organizing the file for quick retrieval would be the most difficult part, but not too difficult if you think about it in advance.

Edited: Gotta read other posts more closely. Sorry Blue Rose, I see VBZeroPlus and I have the same idea.

Are you using a database?

How much memory deos the computer have?

Atleast this works fine for me:

Code:

---

    Dim tnar() As Double
    ReDim tnar(42, 31, 11, 31, 20, 3)
    MsgBox (VarPtr(tnar(42, 31, 11, 31, 20, 3)) - VarPtr(tnar(0, 0, 0, 0, 0, 0))) \ 1024 \ 1024 & " megabytes!"

---

This gives me 338 megabytes, so it fits nicely within the 2 GB of memory that I have. It also may be that some other thing causes the out of memory error, but it is very hard to tell what it might be. For example, you get out of memory error if you're using some API call to create something, but then don't destroy it from memory.


Edit!
Yup, I got out of memory error when I switched to Variant, as I ran out of free memory.

Why you've created one big array? Do you need it for optimization for calculating other results based on values in the array? How randomly you need a specific value from the array, do you need like a value from the beginning, from the middle and the end at the same time at some point?

Hi Merri

thanks for your attention, as you can see picture of my first text of this thread there are some variables which can be changed by user and as a result of this changed variables, i am calculating some values by using a related formula and also calculating error values about this parameters

as you can see my main aim is to find minimum error values which was calculated with variables entered by user.

therefore every parameter in use must be stored in an array, and end of calculation this values must be declared to user because this is a scientific program the values also must be presented in detail (10-15 digit after decimal) i am using decimal type for declaration therefore array must be declared as variant

Quote:

---

Originally Posted by BlueRose

... therefore every parameter in use must be stored in an array...

---

Actually, using a file for storage of array data could still be used, and retrieved for calculations. This will negate the out of memory errors. When calculations are done, retrieve the needed values from the file. Of course this means a little more work since you would be writing data to file vs to an array and subsequently retrieving some data into an array or other variables from the file. Are you saying you need 20 million values for a single process? Or only part of the array for a single process? Either way, fle can still be used even if it requires writing some calculation results to the file to be retrieved later in the process.

Quote:

---

Originally Posted by BlueRose

... and end of calculation this values must be declared to user because this is a scientific program the values also must be presented in detail (10-15 digit after decimal) i am using decimal type for declaration therefore array must be declared as variant

---

You can use double vs variant. Double uses 8 bytes of memory, variant uses 16 and the data in that variant is only 8 bytes anyway with the other 8 bytes used by the variant for header/description information describing that 8 byte value. Try this for some validation:

Code:

---

Private Sub Command1_Click()
    Dim v As Variant
    v = 14322.1234123421
    Debug.Print (VarType(v) = vbDouble) ' variant is holding a double
End Sub

---

Simply change the array to a function.

Add a 7th and 8th parameter.

7th parameter is boolean - passed as false it will retrieve the value from that "array position".

If the 7th parameter is passed as true then you also pass the 8th parameter as the double value to set at that "array position".

The function stores the values on disk - as described in other posts here. I'm not familar enough with "random file" access on PC's - but on mainframes we could easily grab record x - getting 512 bytes returned - from any file. And also write that record as well.

It's up to you to determine what "array position" logic is needed to find you double value in some record on the disk.

If you made this a property of a class then you would not need the 7th or 8th parameter because you could SET/GET the values more naturally.

Logic in the function to "page" what "record" is already in place will allow you to speed this up.

To LaVolpe

i tried to write file but i takes very long time and allocates very very much place (250MB for 3% of calculation) for text file this means it is not useful for this purpose output of this text file like below

Code:

---

0.0028001110111966269477543538:0 1.3 -0.6 -3:0.000462837667985896
0.0028001110111966269477543538:0 1.3 -0.6 -3:0.000331897303655402
0.0028001110111966269477543538:0 1.3 -0.6 -3:0.000221273850109942
0.0028001110111966269477543538:0 1.3 -0.6 -3:0.000149302133905207
0.0028001110111966269477543538:0 1.3 -0.6 -3:0.000102348957119539
0.0028001110111966269477543538:0 1.3 -0.6 -3:0.0000575684975376337
0.0028001110111966269477543538:0 1.3 -0.6 -3:0.0000291431488864269
0.0028001110111966269477543538:0 1.3 -0.6 -3:0.0000154164786133479
0.0028001110111966269477543538:0 1.3 -0.6 -3:0.0000084739858499305
0.0028001110111966269477543538:0 1.3 -0.6 -3:0.00000483242850838852
0.0028001110111966269477543538:0 1.3 -0.6 -3:0.00000285118016152584
0.0028001110111966269477543538:0 1.3 -0.6 -3:0.00000173468560369513
0.0028001110111966269477543538:0 1.3 -0.6 -3:0.00000108878362399447

---

maybe it must be written in binary to hold less space but also this will not be useful


To szlamany

you can be right also i was thought this dont know the way because i am not familiar changing array to function

could you show an example or explain this property?

it sounds good at first sight but need explain in detail for me

You have several issues to overcome...

First - you need to examine how you are going to do random access to a file. Using a sequential access text file is not going to work for you.

Then you need to be able to convert a DOUBLE in memory to it's 8-byte string representation - as you want to store the 8 byte value - not some displayed values of a number.

I've never done either of those two things with VB - but I'm sure it's possible.

i just want to know the way of doing change array to function

could you show me an example written except VB

i can understant the algorithm of it if i see some code

Something like this

Code:

---

Private Sub Form_Load()

Dim SomeDouble As Double

Call MyArray(1, 2, 3, 4, 5, 6, True, 999.111)      ' Set a value

SomeDouble = MyArray(9, 8, 7, 6, 5, 4, False, 0)    ' Return a value

End Sub

Private Function MyArray(x1 As Integer, x2 As Integer, x3 As Integer, x4 As Integer, x5 As Integer, x6 As Integer _
                                        , booSet As Boolean, dblValue As Double) As Double


If booSet Then
    Debug.Print "Let's change the values at location "; x1; " "; x2; " "; x3; " "; x4; " "; x5; " "; x6; " to "; dblValue
Else
    Debug.Print "Let's return the value from "; x1; " "; x2; " "; x3; " "; x4; " "; x5; " "; x6;
End If

End Function

---

will return in the immediate window

Code:

---

Let's change the values at location  1  2  3  4  5  6  to  999.111
Let's return the value from  9  8  7  6  5  4

---

It's up to you to determine where in the random access file the value is stored - that's simply math. And then you need to be able to convert the DOUBLE value to it's eight-byte string representation - since a file only really stores strings (binary strings as it is here).

Again, have you considered using a database and implementing this as a stored procedure with parameters?

Just as a sample on how to write a single 8 byte file with a Double:

Code:

---

Open Filename For Binary Access Write As #1
    Put #1, , dblValue
Close #1

---

Getting the information is just as easy, you just open for reading and use Get instead. Arrays can be filled this way, for example, you could hold 6 first bytes in the file to tell the dimensions, then redim and then read the whole thing to the array at once. Writing is easy as well.

Of course, things do get more complicated when you start adding in the requirement of sequential reading, but my suggestion in this case: make a simple test project that makes what you want, then with the knowledge you've gained you can start off implementing it into the real thing (just reminding of this to help you avoid making a big mess).

To leimad31

I dont think to use a database becasue this program produces very very large data approximately 200 milion records and this is a average value so it could be much more because the number of the records are dependent to min - max values of the variables changed by user

one record is like this

Code:

---

0.0028001110111966269477543538:0 1.3 -0.6 -3:0.00000108878362399447

---

Writing and reading to a database or a file will take long time and this is not undesirable condition for an user

To szlamany and Merri

I want to understand what you said, therefore must tell to me step by step, mostly very clear to you but not to me there are huge records to write a file and one of these is shown above. As you see from the record there are six or more parameters in one record and i must read one record at once

at this point how can i write one record and read again?

So your saying the user will view/inspect/scroll through all 200 million records? I doubt that... think of your data as residing in a data warehouse. But if you prefer to maintain an array locally then so be it.

No not exactly like that

I am holding record for reading back to show suitable parameters, user will not see record while program is working. But after all calculation done user can want to see this record in a file.

@bluerose - you do not want to store the values as they are displayed.

a DOUBLE is internally stored in memory as 8 bytes.

The question you need to get answered first is how to put those 8 memory bytes into a file - not the values that is displayed when you print the dblVariable.

@merri/leinad - is this possible in VB??

1. Some estimates:
1.1. (8-byte Double * 5 columns) * 20M = 800MB binary file excluding delimiters. And you'll have to use copymemory to get the bits from string/bytes to double and vice versa.
1.2. (2-bytes per string character * 70 characters per line) * 20M = 2.8GB text file.

2. Even if split up into several files, will your users really want to skim through the files? Wouldn't it be easier to SELECT tnar WHEN other columns are at particular values?
2.1. What application/viewer will they be using? Can it support 500MB files? How small do you need each file to be in order for them to view the content?

3. On failure processing has to start all over, array has to be repopulated.

4. When primary memory exhausted, secondary memory is paged... essentially reading from a textfile to be stored back on swap file on harddisk.

5. Databases are supposed to hold lots of info with some performance trade-offs that is to be expected. But there are databases that can multi thread query processing, your VB app is single threaded.
5.1. Once your app runs, it will hog resources making PC worthless for other tasks... it would have been better off as a server specification PC.
5.2. You should really be asking, which is more important; how fast the data is generated (array), or how the data is to be used (query, summations, average, sort, multi-user, centralized management of data, export into csv/excel, report generation, data mining, statistics, etc)? Can't the users wait or at the very least understand that such things take time unless they'll buy a mainframe? Wouldn't they prefer that it be run asynchronously on the database and they'll be sent an email notification by DB (don't know if SQL Server supports this) when it completes leaving their workstation free for other tasks?
5.3. You don't have to send each record as one insert query... simply send the stepped values then have the DB create a cartesian join on the data. Pass the starting values, procedure does cartesian joins (batched rather than single join) and calculates tnar accordingly nd finally performs some sort of notification when its done.

6. Some frameworks allow processing at middle-tier, so you can load balance the proicess on the application servers then have them commit their results back to the database. Not something you can do with VB and an array. Your implementation won't scale.

@leinad31...

I appreciate databases as much as the next person - but this situation - being able to retrieve a value based on an easily calculated position - does not warrant a relational database at all.

I believe that I understand from the original post that the user fills in some "calculation criteria" and the 20 million values are then produced with the logic of the calculation.

The user does not need to see or scroll or review these 20 million values - they are simply the by-product of the calculation.

I'm trying to do a COPYMEMORY on an ARRAY of Doubles

Code:

---

Private Declare Sub CopyMemory Lib "kernel32.dll" Alias "RtlMoveMemory" (ByRef Destination As Any, ByRef Source As Any, ByVal Length As Long)

Private Sub Form_Load()

Dim i As Integer, strHold As String, cmLen As Long

Dim WorkArray(100) As Double
Dim TestArray(100) As Double
Dim TestString As String * 8000

For i = 0 To 100
    WorkArray(i) = CDbl(i) * 100
Next i

Debug.Print "WorkArray looks like "; WorkArray(0); WorkArray(1); WorkArray(100)

cmLen = 8 * 2  ' If I make this 8 * 100 it blows up

CopyMemory TestString, WorkArray(0), cmLen

CopyMemory TestArray(0), TestString, cmLen

Debug.Print "TestArray looks like "; TestArray(0); TestArray(1); TestArray(100)

Stop

End Sub

---

produces this in the immediate window

Code:

---

WorkArray looks like  0  100  10000
TestArray looks like  0  100  0

---

So I've proven I can move an array to a string and back to a different array.

But if I change cmLen to 8000 it blows up - the IDE actually crashes.

Why is that??

Quote:

---

Originally Posted by szlamany

@leinad31...

I appreciate databases as much as the next person - but this situation - being able to retrieve a value based on an easily calculated position - does not warrant a relational database at all.

I believe that I understand from the original post that the user fills in some "calculation criteria" and the 20 million values are then produced with the logic of the calculation.

The user does not need to see or scroll or review these 20 million values - they are simply the by-product of the calculation.

---

Exactly, what are the so called by products for? Your all concentrating on the intermediate output rather than the overall system process. Databases aren't exclusively for normalized relational data, they're also designed for volume.

All the effort can be reduced so several lines of procedural SQL (data on 4 columns result of cartesian, last column result of function on other 4 columns, no GUI development) run at the database.

szlamany: fixed length strings are a different beast than variable length strings. I can't get myself to remember how to handle them, so I just switched to a variable length array and fixed it a bit:

Code:

---

Option Explicit

Private Declare Sub CopyMemory Lib "kernel32.dll" Alias "RtlMoveMemory" (ByRef Destination As Any, ByRef Source As Any, ByVal Length As Long)

Private Sub Form_Load()
    Dim i As Integer, strHold As String, cmLen As Long
   
    Dim WorkArray(100) As Double
    Dim TestArray(100) As Double
    Dim TestString As String
   
    TestString = Space$(8000)
   
    Debug.Print "Length of string in bytes: " & LenB(TestString)
   
    For i = 0 To 100
        WorkArray(i) = CDbl(i) * 100
    Next i
   
    Debug.Print "WorkArray looks like "; WorkArray(0); WorkArray(1); WorkArray(100)
   
    cmLen = 8 * 101  ' If I make this 8 * 100 it blows up
   
    CopyMemory ByVal StrPtr(TestString), WorkArray(0), cmLen
   
    CopyMemory TestArray(0), ByVal StrPtr(TestString), cmLen
   
    Debug.Print "TestArray looks like "; TestArray(0); TestArray(1); TestArray(100)
   
    Stop
End Sub

---

I leave it for you to figure out why these fixes work :)

Ok - for some strange reason I had to increase the LEN of the file by 2 for it to first the record.

I've got this now

[edit]Code removed from this post - as it wasn't working and has been refined and corrected in a post #39 - sz[/edit]

REALLY REALLY YOU ARE MARVELLOUS FRIENDS

i want to thank to each of you

to leinad31 for valuable description, yes you're right it's very confusing case for programmer. There are huge data to hold memory or other somewhere

to szlamany for example of valuable program codes
to merri for developing codes of program

you are very very fast people i could not be able to reach you

first i have to examine the your codes and try to understand what you did

Keep in mind that my example has some bug that I have not worked out for you yet.

That value returned at the end of my example should have been 999.111.

I was hoping that Merri or someone else might be able to correct that.

szlamany: you're hitting the Unicode problem. I forgot to write about this because my Firefox joined in with VB's crash when I attempted to read the clipboard (= right click menu in this textbox I'm writing).

When you save a string to a file, it is changed to ANSI. Since you're using both bytes of a character, you get tons of question mark characters into the file each time the upper byte is being used. Actually I don't see why you're using strings in the middle at all, because everything should work perfectly even without using a string, you can save arrays into a file directly and you can read them directly, too.

there is one thing that prepossess me

your are writing variable in file at specific position and variables length is not fixed therefore the allocted place of variable will be change, to use this algoritm variable that is saved is must be fixed length or we must allocate a fixed length place for variables

Doubles are always of the same length. They're not variable length. There is a difference between saving as binary and saving as text.

Print #1, dblValue

This coerces dblValue to a string, thus data is variable length and takes tons of space. It also adds a line change.

Put #1, , dblValue

This makes no coercion, instead it directly writes the 8 bytes from the memory to disk, and nothing else. Thus the result you see in the file is always 8 bytes long.

When saving, you simply make sure the data always stays the same length, and if there is something that may change in length, you can save information on the length of the variable data so that you can read it.

Ok - BlueRose - the function is done - here it is.

Code:

---

Public b1 As Long, b2 As Long, b3 As Long
' Use these to retain the "bounds" of our array

Private Type UDT_Array
    WA(999) As Double
End Type
' This is our work array - we will page into
' memory from the disk 1000 doubles at a time

Private dblPage As UDT_Array
' This is our current page

Private CurWorkRecord As Long
' This will retain the "current" page in memory

Private Sub Form_Load()

Dim SomeDouble As Double

Call DimArray(100, 500, 200)
' This dimensions our array on disk

Call MyArray(1, 2, 3, True, 999.111)        ' Set a value

SomeDouble = MyArray(1, 2, 3, False, 0)    ' Return a value

Debug.Print SomeDouble

Debug.Print "This just tested setting and returning a value at 1,2,3"
Debug.Print ""

Call MyArray(100, 500, 200, True, 888.222)      ' Set a value

SomeDouble = MyArray(100, 500, 200, False, 0)  ' Return a value

Debug.Print SomeDouble

Debug.Print "This just tested setting and returning a value at 100,500,200"
Debug.Print ""

SomeDouble = MyArray(1, 2, 3, False, 0)        ' Return a value

Debug.Print SomeDouble

Debug.Print "This just tested regetting the value from 1,2,3"
Debug.Print "This proves that we are properly paging the work array"

End Sub

Private Function DimArray(x1 As Integer, x2 As Integer, x3 As Integer)

Dim i As Long

b1 = x1
b2 = x2
b3 = x3
' Retain the bounds that were passed

Open "C:\ArraySpace.tmp" For Random As #1 Len = LenB(dblPage)
' Create the RANDOM access disk file with page size to match our work array

For i = 1 To (b1 * b2 * b3) / 1000
    Put #1, i, dblPage
Next i
' Initialize the disk file to all zeroes

CurWorkRecord = -1
' Mark as "no current work record" paged

End Function

Private Function MyArray(x1 As Integer, x2 As Integer, x3 As Integer _
                                        , booSet As Boolean, dblValue As Double) As Double

' booSet as false will return a value from x1, x2, x3
' booSet as true will set the value at x1, x2, x3 to the dblValue passed

Dim WorkSlot As Long, WorkRecord As Long

WorkSlot = Location(x1, x2, x3) - 1
' Turn the x1, x2, x3 into a single-bound location

WorkRecord = WorkSlot \ 1000
' Divide by 1000 to find out what page that slot is in

WorkRecord = WorkRecord + 1
' Add 1 since the division will return 0 for the first 1000 slots

If CurWorkRecord <> WorkRecord Then    ' We need a fresh read from disk
    If CurWorkRecord <> -1 Then        ' We need to write record back to disk
        Put #1, CurWorkRecord, dblPage
    End If
    Get #1, WorkRecord, dblPage        ' Now get the page we need
    CurWorkRecord = WorkRecord          ' And remember what page is in memory
End If

If booSet Then                          ' Set the value
    dblPage.WA(WorkSlot - ((WorkRecord - 1) * 1000)) = dblValue
    Debug.Print "Let's change the values at location "; x1; " "; x2; " "; x3; " to "; dblValue
    Debug.Print "Really at slot "; WorkSlot - ((WorkRecord - 1) * 1000); " in record "; CurWorkRecord
Else                                    ' Return the value
    Debug.Print "Let's return the value from "; x1; " "; x2; " "; x3
    Debug.Print "Really at slot "; WorkSlot - ((WorkRecord - 1) * 1000); " in record "; CurWorkRecord
    MyArray = dblPage.WA(WorkSlot - ((WorkRecord - 1) * 1000))
End If

End Function

Private Function Location(x1 As Integer, x2 As Integer, x3 As Integer) As Long

' 1000 x 500 x 200 is the array size

' 1st double is for 0,0,0
' 2nd double is for 0,0,1
' 201st double is for 0,0,200

' 202nd double is for 0,1,0
' 203rd double is for 0,1,1
' 402nd double is for 0,1,200

' 403rd double is for 0,2,0
' .
' .
' 201 for each x3
' repeated 501 times for each x2
' 201 * 501 = 100701 spots x1 = 0
'
' 100702 spot is for 1,0,0

Location = (x1 * ((b2 + 1) * (b3 + 1))) + (x2 * (b3 + 1)) + x3 + 1

End Function

---

When it is run it returns the following in the immediate window

Code:

---

Let's change the values at location  1  2  3  to  999.111
Really at slot  106  in record  102
Let's return the value from  1  2  3
Really at slot  106  in record  102
 999.111
This just tested setting and returning a value at 1,2,3

Let's change the values at location  100  500  200  to  888.222
Really at slot  800  in record  10171
Let's return the value from  100  500  200
Really at slot  800  in record  10171
 888.222
This just tested setting and returning a value at 100,500,200

Let's return the value from  1  2  3
Really at slot  106  in record  102
 999.111
This just tested regetting the value from 1,2,3
This proves that we are properly paging the work array

---

Of course this example is for a 3 dimension array - but it's just a matter of making minor changes to make it any number of dimensions.

Adding my 2 cents worth. If I were to try to replicate a huge array as a file, I would mimic the array structure. This means the "ReDim Preserve" would simply change the EOF marker in the file, adding/subtracting (API file operations, not VB). "ReDim" without preserve would require both EOF marker and zeroing out file data. Retrieval can be easily accomplished by calculating the offset of the array element within the file and intelligent retrieval should be used to read an entire range of data at once, as much as possible; otherwise individual element reads could prove slow.

Another note. Are the huge arrays absolutely necessary? Can their size be reduced? Is it possible to recalculate formulas vs storing calculated values some or most of the time?

Back in our mini/mainframe days we had virtual arrays that were disk stored as part of the syntax in BASIC (DEC PDP-11 machines).

I mimiced that concept.

I arbritrarily set the "page" size to 1000 elements. Since it's double's for this example that meant the page was 8000 bytes (a double is 8 bytes - right?). That page is held in memory and used until a different page is required - and only then is the page written back to disk.

It appears that RANDOM files in VB have a max of 16000 bytes per record anyway.

Paging in 1000 elements at a time is a good idea if you frequently use elements in the same area.

That size can and should be adjusted if the locations hit are more "all over" the range of the bounds - as it would be faster to page less elements from disk.

A REDIM would be easy to accomplish - probably best to "move" the elements to a new "file" on disk with those bounds. That's what VB does with a real REDIM in memory anyway.

Quote:

---

Originally Posted by szlamany

A REDIM would be easy to accomplish - probably best to "move" the elements to a new "file" on disk with those bounds. That's what VB does with a real REDIM in memory anyway.

---

True, but when redim with/without "preserve" is used by VB, VB has to do it that way because it needs contiguous memory space. With a file, you just increase/decrease the file length (assuming disk has space) and you don't have to rewrite any data (if preserve) -- much faster. Without preserving, same principle could be used, but the data should be zero'd out; or a new file could be used.

Another note: should a file array be used. Recommend user write loops to reference array from right to left vs left to right. With a 6 dimensional array, look at it like the following.

Dim Data(u, v, w, x, y, z)
u = a letter
v = a word
w = a paragraph
x = a page
y = a chapter
z = a book

All the letters for each word for each paragraph for each page for each chapter for each book are in consecutive memory. Ideally, you'd want to retrieve the data in similar form: a book, chapter, page or paragraph at a time vs a letter or word at a time.

If For:Loops were stacked from z to u, top to bottom, you would get better performance retrieving blocks of data from the file.

But if For:Loops were stacked from u to z, you would be jumping all over the file trying to retrieve array data.

Here's an example of how a multi-dimensional array looks, be it in memory or written to a file. I think you will see how the order of the For:Loops can reduce number of block retrievals.

Code:

---

Private Declare Sub CopyMemory Lib "kernel32.dll" Alias "RtlMoveMemory" (ByRef Destination As Any, ByRef Source As Any, ByVal Length As Long)

Private Sub Command1_Click()

Dim x As Long, y As Long, z As Long, w As Long, v As Long
Dim lptr As Long, s As String, e As Long
Dim xyz(1 To 3, 1 To 2, 1 To 2, 1 To 3, 0 To 2) As String
' array elements for this example are v, w, x, y, z

' fill test array
For v = LBound(xyz, 1) To UBound(xyz, 1)
    For w = LBound(xyz, 2) To UBound(xyz, 2)
        For x = LBound(xyz, 3) To UBound(xyz, 3)
            For y = LBound(xyz, 4) To UBound(xyz, 4)
                For z = LBound(xyz, 5) To UBound(xyz, 5)
                    xyz(v, w, x, y, z) = v & "." & w & "." & x & "." & y & "." & z
                Next
            Next
        Next
    Next
Next

Debug.Print "How array is stored in memmory"
lptr = VarPtr(xyz(1, 1, 1, 1, 0))
For v = LBound(xyz, 1) To UBound(xyz, 1)
    For w = LBound(xyz, 2) To UBound(xyz, 2)
        For x = LBound(xyz, 3) To UBound(xyz, 3)
            For y = LBound(xyz, 4) To UBound(xyz, 4)
                For z = LBound(xyz, 5) To UBound(xyz, 5)
                    CopyMemory ByVal VarPtr(s), ByVal lptr, 4&
                    lptr = lptr + 4
                    Debug.Print s; "  ";
                Next
                Debug.Print "elements"; e; "to"; e + 4 - 1
                e = e + 4
            Next
        Next
    Next
Next
CopyMemory ByVal VarPtr(s), 0&, 4&

Debug.Print "Looping from left to right - jumping around array happens for every record retrieved"
For v = LBound(xyz, 1) To LBound(xyz, 1)
    For w = LBound(xyz, 2) To LBound(xyz, 2)
        For x = LBound(xyz, 3) To LBound(xyz, 3)
            For y = LBound(xyz, 4) To LBound(xyz, 4)
                For z = LBound(xyz, 5) To UBound(xyz, 5)
                    Debug.Print xyz(v, w, x, y, z); "  ";
                Next
                Debug.Print ""
            Next
        Next
    Next
Next

Debug.Print "Looping from right to left - retrieves records in same order as in memory; consecutively"
For z = LBound(xyz, 5) To LBound(xyz, 5)
    For y = LBound(xyz, 4) To LBound(xyz, 4)
        For x = LBound(xyz, 3) To LBound(xyz, 3)
            For w = LBound(xyz, 2) To LBound(xyz, 2)
                For v = LBound(xyz, 1) To UBound(xyz, 1)
                    Debug.Print xyz(v, w, x, y, z); "  ";
                Next
            Next
        Next
    Next
Next

End Sub

FYI: To calculate the memory position for any element in a multi-dimensional array
formula: from left to right array elements (E)
(E1-LBound(Dim1) +
  (E2-LBound(Dim2))*(NrElemOfDim1) +
    (E3-LBound(Dim3))*(NrElemOfDim1*nrElemOfDim2) +
      (E4-LBound(Dim4))*(NrElemOfDim1*nrElemOfDim2*nrElemOfDim3)
        (E5-LBound(Dim5))*(NrElemOfDim1*nrElemOfDim2*nrElemOfDim3*nrElemOfDim4) ) * element length in bytes

---

Whether this is more efficienct for actual memory vs file, haven't tested it.

again YOU ARE MARVELLOUS FRIENDS

thanks a milion to each of you

it seems to be resolved but i want to test to be sure

Keep in mind that this function in my routine

Code:

---

Private Function Location(x1 As Integer, x2 As Integer, x3 As Integer) As Long

' 1000 x 500 x 200 is the array size

' 1st double is for 0,0,0
' 2nd double is for 0,0,1
' 201st double is for 0,0,200

' 202nd double is for 0,1,0
' 203rd double is for 0,1,1
' 402nd double is for 0,1,200

' 403rd double is for 0,2,0
' .
' .
' 201 for each x3
' repeated 501 times for each x2
' 201 * 501 = 100701 spots x1 = 0
'
' 100702 spot is for 1,0,0

Location = (x1 * ((b2 + 1) * (b3 + 1))) + (x2 * (b3 + 1)) + x3 + 1

End Function

---

determines the location of the value.

I've always felt that the right-most array slot should contain values that are most closely related. As the left-most array slot changes "large" scale differences are being related.

This LOCATION function is written with that in mind.

It can easily be changed to work any way you like.

specially to szlamany and lavolpe

i test the code sent it's working

i could not do full test in my program, there is something to arrange from me to adopt code to program and this will take time for a while

before this i have some questions

first the user type UDT_Array isa double type, how can we change to string type? as we could this with from "WA(999) as Double" to "WA(999) as String" will it enough or need something? because i want to reduce some dimension and to store some parameters within a string variable spaced with a space character.

second while program is runing, writing and reading to file of array is ok, but i could not see the values in the file when i look it with a text editor. After program run and ArraySpace.tmp file is created, i could not read values directly from the file, it is shown all to "0". Why?

third can we do a class module for this type array? because i will use three array and must be saved with a differrent file names

ok?

I thought you were dealing with DOUBLE values? Why do you want to change it to a string array? Please give very specific details for this.

As for the "visual" appearance of the ArraySpace.tmp file...

We mentioned in several posts about DOUBLES - and I think the point is being missed. I will try to be much more clear about this.

A double appears as a "number" when displayed in VB. It appears to work as a number when it's used in a calculation. But in reality - in memory - all doubles are stored as a a binary-representation that takes 8 bytes of memory. You might see 123.456 or .0123123123e6 or whatever - but that value takes 8 bytes of memory to be stored.

It's the memory we are storing. We have that WA() array of 1000 items. Each item is 8 bytes. That's 8000 bytes of storage.

The perfection of this approach is that there is no need to handle converting the number to a string for storing - or the fact that each number "appears" as a different length in VB when looked at numerically.

No delimiters are required in the TMP file.

That is required to maintain some level of speed here. If you had to "convert" 1000 elements to strings with delimeters for storage this concept would fail miserably.

That's why I asked for specific reasons for why you want to change the datatype of the array.

it is just reduce the number of dimension

Also i want to know whether it is possible to change datatype or not

do you have any answer for my 2nd and 3rd question?

Strings are variable length.

So no you cannot store strings with this method.

The point of storing the doubles is that each is fixed length. That is what allows this "paging" of array data to disk to work.

There is no reason to reduce the number of dimensions - as this routine turns them into a single-line array position.

Again - I'm more then willing to help if you give me very specific information regarding your requirements.

Yes this is the working program sample for array on disk or FileArray (named by me) maybe someone could call differently

i want to express that it is really handling very must parameters that i expected aproximately 700-800 million records that i tested maybe much more canbe handled

Just try it

Bluerose

I'm glad this worked for you!

If you don't mind I will create a codebank thread with both my explanation of how this works and you code sample.

Is that ok?

Sure ok?

Almostly it is yours

It's now in the code bank

http://www.vbforums.com/showthread.php?t=504649