Re: How can we hold an array has 20 MILLION PARAMETERS in VB?
Quote:
Originally Posted by szlamany
A REDIM would be easy to accomplish - probably best to "move" the elements to a new "file" on disk with those bounds. That's what VB does with a real REDIM in memory anyway.
True, but when redim with/without "preserve" is used by VB, VB has to do it that way because it needs contiguous memory space. With a file, you just increase/decrease the file length (assuming disk has space) and you don't have to rewrite any data (if preserve) -- much faster. Without preserving, same principle could be used, but the data should be zero'd out; or a new file could be used.
Re: How can we hold an array has 20 MILLION PARAMETERS in VB?
Another note: should a file array be used. Recommend user write loops to reference array from right to left vs left to right. With a 6 dimensional array, look at it like the following.
Dim Data(u, v, w, x, y, z)
u = a letter
v = a word
w = a paragraph
x = a page
y = a chapter
z = a book
All the letters for each word for each paragraph for each page for each chapter for each book are in consecutive memory. Ideally, you'd want to retrieve the data in similar form: a book, chapter, page or paragraph at a time vs a letter or word at a time.
If For:Loops were stacked from z to u, top to bottom, you would get better performance retrieving blocks of data from the file.
But if For:Loops were stacked from u to z, you would be jumping all over the file trying to retrieve array data.
Here's an example of how a multi-dimensional array looks, be it in memory or written to a file. I think you will see how the order of the For:Loops can reduce number of block retrievals.
Code:
Private Declare Sub CopyMemory Lib "kernel32.dll" Alias "RtlMoveMemory" (ByRef Destination As Any, ByRef Source As Any, ByVal Length As Long)
Private Sub Command1_Click()
Dim x As Long, y As Long, z As Long, w As Long, v As Long
Dim lptr As Long, s As String, e As Long
Dim xyz(1 To 3, 1 To 2, 1 To 2, 1 To 3, 0 To 2) As String
' array elements for this example are v, w, x, y, z
' fill test array
For v = LBound(xyz, 1) To UBound(xyz, 1)
For w = LBound(xyz, 2) To UBound(xyz, 2)
For x = LBound(xyz, 3) To UBound(xyz, 3)
For y = LBound(xyz, 4) To UBound(xyz, 4)
For z = LBound(xyz, 5) To UBound(xyz, 5)
xyz(v, w, x, y, z) = v & "." & w & "." & x & "." & y & "." & z
Next
Next
Next
Next
Next
Debug.Print "How array is stored in memmory"
lptr = VarPtr(xyz(1, 1, 1, 1, 0))
For v = LBound(xyz, 1) To UBound(xyz, 1)
For w = LBound(xyz, 2) To UBound(xyz, 2)
For x = LBound(xyz, 3) To UBound(xyz, 3)
For y = LBound(xyz, 4) To UBound(xyz, 4)
For z = LBound(xyz, 5) To UBound(xyz, 5)
CopyMemory ByVal VarPtr(s), ByVal lptr, 4&
lptr = lptr + 4
Debug.Print s; " ";
Next
Debug.Print "elements"; e; "to"; e + 4 - 1
e = e + 4
Next
Next
Next
Next
CopyMemory ByVal VarPtr(s), 0&, 4&
Debug.Print "Looping from left to right - jumping around array happens for every record retrieved"
For v = LBound(xyz, 1) To LBound(xyz, 1)
For w = LBound(xyz, 2) To LBound(xyz, 2)
For x = LBound(xyz, 3) To LBound(xyz, 3)
For y = LBound(xyz, 4) To LBound(xyz, 4)
For z = LBound(xyz, 5) To UBound(xyz, 5)
Debug.Print xyz(v, w, x, y, z); " ";
Next
Debug.Print ""
Next
Next
Next
Next
Debug.Print "Looping from right to left - retrieves records in same order as in memory; consecutively"
For z = LBound(xyz, 5) To LBound(xyz, 5)
For y = LBound(xyz, 4) To LBound(xyz, 4)
For x = LBound(xyz, 3) To LBound(xyz, 3)
For w = LBound(xyz, 2) To LBound(xyz, 2)
For v = LBound(xyz, 1) To UBound(xyz, 1)
Debug.Print xyz(v, w, x, y, z); " ";
Next
Next
Next
Next
Next
End Sub
FYI: To calculate the memory position for any element in a multi-dimensional array
formula: from left to right array elements (E)
(E1-LBound(Dim1) +
(E2-LBound(Dim2))*(NrElemOfDim1) +
(E3-LBound(Dim3))*(NrElemOfDim1*nrElemOfDim2) +
(E4-LBound(Dim4))*(NrElemOfDim1*nrElemOfDim2*nrElemOfDim3)
(E5-LBound(Dim5))*(NrElemOfDim1*nrElemOfDim2*nrElemOfDim3*nrElemOfDim4) ) * element length in bytes
Whether this is more efficienct for actual memory vs file, haven't tested it.
Re: How can we hold an array has 20 MILLION PARAMETERS in VB?
again YOU ARE MARVELLOUS FRIENDS
thanks a milion to each of you
it seems to be resolved but i want to test to be sure
Re: How can we hold an array has 20 MILLION PARAMETERS in VB?
Keep in mind that this function in my routine
Code:
Private Function Location(x1 As Integer, x2 As Integer, x3 As Integer) As Long
' 1000 x 500 x 200 is the array size
' 1st double is for 0,0,0
' 2nd double is for 0,0,1
' 201st double is for 0,0,200
' 202nd double is for 0,1,0
' 203rd double is for 0,1,1
' 402nd double is for 0,1,200
' 403rd double is for 0,2,0
' .
' .
' 201 for each x3
' repeated 501 times for each x2
' 201 * 501 = 100701 spots x1 = 0
'
' 100702 spot is for 1,0,0
Location = (x1 * ((b2 + 1) * (b3 + 1))) + (x2 * (b3 + 1)) + x3 + 1
End Function
determines the location of the value.
I've always felt that the right-most array slot should contain values that are most closely related. As the left-most array slot changes "large" scale differences are being related.
This LOCATION function is written with that in mind.
It can easily be changed to work any way you like.
Re: How can we hold an array has 20 MILLION PARAMETERS in VB?
specially to szlamany and lavolpe
i test the code sent it's working
i could not do full test in my program, there is something to arrange from me to adopt code to program and this will take time for a while
before this i have some questions
first the user type UDT_Array isa double type, how can we change to string type? as we could this with from "WA(999) as Double" to "WA(999) as String" will it enough or need something? because i want to reduce some dimension and to store some parameters within a string variable spaced with a space character.
second while program is runing, writing and reading to file of array is ok, but i could not see the values in the file when i look it with a text editor. After program run and ArraySpace.tmp file is created, i could not read values directly from the file, it is shown all to "0". Why?
third can we do a class module for this type array? because i will use three array and must be saved with a differrent file names
ok?
Re: How can we hold an array has 20 MILLION PARAMETERS in VB?
I thought you were dealing with DOUBLE values? Why do you want to change it to a string array? Please give very specific details for this.
As for the "visual" appearance of the ArraySpace.tmp file...
We mentioned in several posts about DOUBLES - and I think the point is being missed. I will try to be much more clear about this.
A double appears as a "number" when displayed in VB. It appears to work as a number when it's used in a calculation. But in reality - in memory - all doubles are stored as a a binary-representation that takes 8 bytes of memory. You might see 123.456 or .0123123123e6 or whatever - but that value takes 8 bytes of memory to be stored.
It's the memory we are storing. We have that WA() array of 1000 items. Each item is 8 bytes. That's 8000 bytes of storage.
The perfection of this approach is that there is no need to handle converting the number to a string for storing - or the fact that each number "appears" as a different length in VB when looked at numerically.
No delimiters are required in the TMP file.
That is required to maintain some level of speed here. If you had to "convert" 1000 elements to strings with delimeters for storage this concept would fail miserably.
That's why I asked for specific reasons for why you want to change the datatype of the array.
Re: How can we hold an array has 20 MILLION PARAMETERS in VB?
it is just reduce the number of dimension
Also i want to know whether it is possible to change datatype or not
do you have any answer for my 2nd and 3rd question?
Re: How can we hold an array has 20 MILLION PARAMETERS in VB?
Strings are variable length.
So no you cannot store strings with this method.
The point of storing the doubles is that each is fixed length. That is what allows this "paging" of array data to disk to work.
There is no reason to reduce the number of dimensions - as this routine turns them into a single-line array position.
Again - I'm more then willing to help if you give me very specific information regarding your requirements.
1 Attachment(s)
Re: [RESOLVED] How can we hold an array has 20 MILLION PARAMETERS in VB?
Yes this is the working program sample for array on disk or FileArray (named by me) maybe someone could call differently
i want to express that it is really handling very must parameters that i expected aproximately 700-800 million records that i tested maybe much more canbe handled
Just try it
Re: [RESOLVED] How can we hold an array has 20 MILLION PARAMETERS in VB?
Bluerose
I'm glad this worked for you!
If you don't mind I will create a codebank thread with both my explanation of how this works and you code sample.
Is that ok?
Re: [RESOLVED] How can we hold an array has 20 MILLION PARAMETERS in VB?
Sure ok?
Almostly it is yours
Re: [RESOLVED] How can we hold an array has 20 MILLION PARAMETERS in VB?
