Storing a huge array on disk - paging in and out of memory

[szlamany]

We had a thread back a couple of weeks ago where someone wanted to store a truly large array - one too large for memory and stack space in VB.

This code shows an example of how to store an ARRAY of DOUBLE's on disk. It cannot be used for string arrays - as the len of each string is not fixed - while the length of a DOUBLE is always 8 bytes in memory (for instance).

This is the variable declarations

Public b1 As Long, b2 As Long, b3 As Long
' Use these to retain the "bounds" of our array

Private Type UDT_Array
    WA(999) As Double
End Type
' This is our work array - we will page into
' memory from the disk 1000 doubles at a time

Private dblPage As UDT_Array
' This is our current page

Private CurWorkRecord As Long
' This will retain the "current" page in memory

This is the function that "dimensions" the array. Basically writes a "zero-value" into each array spot and writes those pages to disk.

Private Function DimArray(x1 As Integer, x2 As Integer, x3 As Integer)

Dim i As Long

b1 = x1
b2 = x2
b3 = x3
' Retain the bounds that were passed

Open "C:\ArraySpace.tmp" For Random As #1 Len = LenB(dblPage)
' Create the RANDOM access disk file with page size to match our work array

For i = 1 To (b1 * b2 * b3) / 1000
    Put #1, i, dblPage
Next i
' Initialize the disk file to all zeroes

CurWorkRecord = -1
' Mark as "no current work record" paged

End Function

This is the actual array function itself - you can both "read" from this function or "write" to this function.

Private Function MyArray(x1 As Integer, x2 As Integer, x3 As Integer _
                                        , booSet As Boolean, dblValue As Double) As Double

' booSet as false will return a value from x1, x2, x3
' booSet as true will set the value at x1, x2, x3 to the dblValue passed

Dim WorkSlot As Long, WorkRecord As Long

WorkSlot = Location(x1, x2, x3) - 1
' Turn the x1, x2, x3 into a single-bound location

WorkRecord = WorkSlot \ 1000
' Divide by 1000 to find out what page that slot is in

WorkRecord = WorkRecord + 1
' Add 1 since the division will return 0 for the first 1000 slots

If CurWorkRecord <> WorkRecord Then     ' We need a fresh read from disk
    If CurWorkRecord <> -1 Then         ' We need to write record back to disk
        Put #1, CurWorkRecord, dblPage
    End If
    Get #1, WorkRecord, dblPage         ' Now get the page we need
    CurWorkRecord = WorkRecord          ' And remember what page is in memory
End If

If booSet Then                          ' Set the value
    dblPage.WA(WorkSlot - ((WorkRecord - 1) * 1000)) = dblValue
    Debug.Print "Let's change the values at location "; x1; " "; x2; " "; x3; " to "; dblValue
    Debug.Print "Really at slot "; WorkSlot - ((WorkRecord - 1) * 1000); " in record "; CurWorkRecord
Else                                    ' Return the value
    Debug.Print "Let's return the value from "; x1; " "; x2; " "; x3
    Debug.Print "Really at slot "; WorkSlot - ((WorkRecord - 1) * 1000); " in record "; CurWorkRecord
    MyArray = dblPage.WA(WorkSlot - ((WorkRecord - 1) * 1000))
End If

End Function

This function simply turns the 3 dimension array location into a single line location.

Private Function Location(x1 As Integer, x2 As Integer, x3 As Integer) As Long

' 1000 x 500 x 200 is the array size

' 1st double is for 0,0,0
' 2nd double is for 0,0,1
' 201st double is for 0,0,200

' 202nd double is for 0,1,0
' 203rd double is for 0,1,1
' 402nd double is for 0,1,200

' 403rd double is for 0,2,0
' .
' .
' 201 for each x3
' repeated 501 times for each x2
' 201 * 501 = 100701 spots x1 = 0
'
' 100702 spot is for 1,0,0

Location = (x1 * ((b2 + 1) * (b3 + 1))) + (x2 * (b3 + 1)) + x3 + 1

End Function

Here is a test of the process itself

Private Sub Form_Load()

Dim SomeDouble As Double

Call DimArray(100, 500, 200)
' This dimensions our array on disk

Call MyArray(1, 2, 3, True, 999.111)        ' Set a value

SomeDouble = MyArray(1, 2, 3, False, 0)     ' Return a value

Debug.Print SomeDouble

Debug.Print "This just tested setting and returning a value at 1,2,3"
Debug.Print ""

Call MyArray(100, 500, 200, True, 888.222)      ' Set a value

SomeDouble = MyArray(100, 500, 200, False, 0)   ' Return a value

Debug.Print SomeDouble

Debug.Print "This just tested setting and returning a value at 100,500,200"
Debug.Print ""

SomeDouble = MyArray(1, 2, 3, False, 0)         ' Return a value

Debug.Print SomeDouble

Debug.Print "This just tested regetting the value from 1,2,3"
Debug.Print "This proves that we are properly paging the work array"

End Sub

When it's run this is the output in the immediate window testing it

Let's change the values at location  1   2   3  to  999.111 
Really at slot  106  in record  102 
Let's return the value from  1   2   3 
Really at slot  106  in record  102 
 999.111 
This just tested setting and returning a value at 1,2,3

Let's change the values at location  100   500   200  to  888.222 
Really at slot  800  in record  10171 
Let's return the value from  100   500   200 
Really at slot  800  in record  10171 
 888.222 
This just tested setting and returning a value at 100,500,200

Let's return the value from  1   2   3 
Really at slot  106  in record  102 
 999.111 
This just tested regetting the value from 1,2,3
This proves that we are properly paging the work array

Bluerose took these examples and put together a nice class that uses these concepts. Here is the zip file.

Here is the thread post where Bluerose uploaded the zip - if you wanted to see the rest of this

http://www.vbforums.com/showpost.php...4&postcount=49