Storing a huge array on disk - paging in and out of memory

We had a thread back a couple of weeks ago where someone wanted to store a truly large array - one too large for memory and stack space in VB.

This code shows an example of how to store an ARRAY of DOUBLE's on disk. It cannot be used for string arrays - as the len of each string is not fixed - while the length of a DOUBLE is always 8 bytes in memory (for instance).

This is the variable declarations

Code:

Public b1 As Long, b2 As Long, b3 As Long ' Use these to retain the "bounds" of our array Private Type UDT_Array WA(999) As Double End Type ' This is our work array - we will page into ' memory from the disk 1000 doubles at a time Private dblPage As UDT_Array ' This is our current page Private CurWorkRecord As Long ' This will retain the "current" page in memory

This is the function that "dimensions" the array. Basically writes a "zero-value" into each array spot and writes those pages to disk.

Code:

Private Function DimArray(x1 As Integer, x2 As Integer, x3 As Integer) Dim i As Long b1 = x1 b2 = x2 b3 = x3 ' Retain the bounds that were passed Open "C:\ArraySpace.tmp" For Random As #1 Len = LenB(dblPage) ' Create the RANDOM access disk file with page size to match our work array For i = 1 To (b1 * b2 * b3) / 1000 Put #1, i, dblPage Next i ' Initialize the disk file to all zeroes CurWorkRecord = -1 ' Mark as "no current work record" paged End Function

This is the actual array function itself - you can both "read" from this function or "write" to this function.

Code:

Private Function MyArray(x1 As Integer, x2 As Integer, x3 As Integer _ , booSet As Boolean, dblValue As Double) As Double ' booSet as false will return a value from x1, x2, x3 ' booSet as true will set the value at x1, x2, x3 to the dblValue passed Dim WorkSlot As Long, WorkRecord As Long WorkSlot = Location(x1, x2, x3) - 1 ' Turn the x1, x2, x3 into a single-bound location WorkRecord = WorkSlot \ 1000 ' Divide by 1000 to find out what page that slot is in WorkRecord = WorkRecord + 1 ' Add 1 since the division will return 0 for the first 1000 slots If CurWorkRecord <> WorkRecord Then ' We need a fresh read from disk If CurWorkRecord <> -1 Then ' We need to write record back to disk Put #1, CurWorkRecord, dblPage End If Get #1, WorkRecord, dblPage ' Now get the page we need CurWorkRecord = WorkRecord ' And remember what page is in memory End If If booSet Then ' Set the value dblPage.WA(WorkSlot - ((WorkRecord - 1) * 1000)) = dblValue Debug.Print "Let's change the values at location "; x1; " "; x2; " "; x3; " to "; dblValue Debug.Print "Really at slot "; WorkSlot - ((WorkRecord - 1) * 1000); " in record "; CurWorkRecord Else ' Return the value Debug.Print "Let's return the value from "; x1; " "; x2; " "; x3 Debug.Print "Really at slot "; WorkSlot - ((WorkRecord - 1) * 1000); " in record "; CurWorkRecord MyArray = dblPage.WA(WorkSlot - ((WorkRecord - 1) * 1000)) End If End Function

This function simply turns the 3 dimension array location into a single line location.

Code:

Private Function Location(x1 As Integer, x2 As Integer, x3 As Integer) As Long ' 1000 x 500 x 200 is the array size ' 1st double is for 0,0,0 ' 2nd double is for 0,0,1 ' 201st double is for 0,0,200 ' 202nd double is for 0,1,0 ' 203rd double is for 0,1,1 ' 402nd double is for 0,1,200 ' 403rd double is for 0,2,0 ' . ' . ' 201 for each x3 ' repeated 501 times for each x2 ' 201 * 501 = 100701 spots x1 = 0 ' ' 100702 spot is for 1,0,0 Location = (x1 * ((b2 + 1) * (b3 + 1))) + (x2 * (b3 + 1)) + x3 + 1 End Function

Here is a test of the process itself

Code:

Private Sub Form_Load() Dim SomeDouble As Double Call DimArray(100, 500, 200) ' This dimensions our array on disk Call MyArray(1, 2, 3, True, 999.111) ' Set a value SomeDouble = MyArray(1, 2, 3, False, 0) ' Return a value Debug.Print SomeDouble Debug.Print "This just tested setting and returning a value at 1,2,3" Debug.Print "" Call MyArray(100, 500, 200, True, 888.222) ' Set a value SomeDouble = MyArray(100, 500, 200, False, 0) ' Return a value Debug.Print SomeDouble Debug.Print "This just tested setting and returning a value at 100,500,200" Debug.Print "" SomeDouble = MyArray(1, 2, 3, False, 0) ' Return a value Debug.Print SomeDouble Debug.Print "This just tested regetting the value from 1,2,3" Debug.Print "This proves that we are properly paging the work array" End Sub

When it's run this is the output in the immediate window testing it

Code:

Let's change the values at location 1 2 3 to 999.111 Really at slot 106 in record 102 Let's return the value from 1 2 3 Really at slot 106 in record 102 999.111 This just tested setting and returning a value at 1,2,3 Let's change the values at location 100 500 200 to 888.222 Really at slot 800 in record 10171 Let's return the value from 100 500 200 Really at slot 800 in record 10171 888.222 This just tested setting and returning a value at 100,500,200 Let's return the value from 1 2 3 Really at slot 106 in record 102 999.111 This just tested regetting the value from 1,2,3 This proves that we are properly paging the work array

Bluerose took these examples and put together a nice class that uses these concepts. Here is the zip file.

Here is the thread post where Bluerose uploaded the zip - if you wanted to see the rest of this

http://www.vbforums.com/showpost.php...4&postcount=49