VB6 - Treat a file like an array without reading to memory.

Here's a little snippet of code I posted over on CodeGuru that I thought people over here might be able to make use of. It creates a memory map of a file, meaning that instead of loading the file into memory, it creates a pointer to the file's physical location on the disk. So, you can treat the file as a variable, and any changes you make to the variables are also changing the target file. While useful in and of itself, it's a little limiting without a nice convienent way to index into it (are you thinking array? I am).

The following sample code opens a memory mapped file, creates a byte array, and then modifies the data pointer in the array to point at the file. So, you can do 'reads' with bVar = bByte(x), and 'writes' with bByte(x) = bVar.

It wouldn't be hard to convert from a byte array to another type of array (string excluded, unless the strings were fixed length). All you need to do in that case would be to change the cbElements of the array structure to the size of the chunk of data you want--i.e., if you wanted to treat the file as an array of Longs, cbElements would be 4, a UDT would be LenB(uType), etc. The second type (SafeBound) holds the LBound and UBound information. I wouldn't change the LBound, but you'll want to manually calculate the UBound based on the length of the file and the size of the elements. So, if the file was 1000 bytes and you wanted to treat it as an array of Longs, you would set uTemp.cbElements to 4 and set uTemp.rgsabound.cElements to 249.

Couple of things you might want to do would be to add error checking for the API calls and re-build the original array (mainly just point it back to what it was pointing to before you hacked it) after you remove the memory map to make sure that the memory for the original array data gets de-allocated when it's garbage collected.

Here's the code. This example opens a file and changes the first byte to "=".

VB Code:

'API DECs
Private Declare Sub CopyMemory Lib "kernel32" Alias "RtlMoveMemory" (pDst As Any, pSrc As Any, _
                                                                     ByVal ByteLen As Long)
Private Declare Function CreateFile Lib "kernel32" Alias "CreateFileA" (ByVal lpFileName As String, _
                                         ByVal dwDesiredAccess As Long, ByVal dwShareMode As Long, _
                                         ByVal lpSecurityAttributes As Long, ByVal dwCreationDisposition As Long, _
                                         ByVal dwFlagsAndAttributes As Long, ByVal hTemplateFile As Long) As Long
Private Declare Function CloseHandle Lib "kernel32" (ByVal hObject As Long) As Long
 
'MEMORY MAPPING APIs
Private Declare Function MapViewOfFile Lib "kernel32.dll" (ByVal hFile As Long, ByVal dwDesiredAccess As Long, _
                                         ByVal dwFileOffsetHigh As Long, ByVal dwFileOffsetLow As Long, _
                                         ByVal dwNumberOfBytesToMap As Long) As Long
Private Declare Function CreateFileMapping Lib "kernel32.dll" Alias "CreateFileMappingA" (ByVal hFile As Long, _
                                         ByVal lpAttributes As Long, ByVal flProtect As Long, _
                                         ByVal dwMaximumSizeHigh As Long, ByVal dwMaximumSizeLow As Long, _
                                         ByVal lpName As String) As Long
Private Declare Function UnmapViewOfFile Lib "kernel32.dll" (ByVal lpBaseAddress As Long) As Boolean
 
'STRUCTS FOR THE SAFEARRAY:
Private Type SafeBound
    cElements As Long
    lLbound As Long
End Type
 
Private Type SafeArray
    cDim As Integer
    fFeature As Integer
    cbElements As Long
    cLocks As Long
    pvData As Long
    rgsabound As SafeBound
End Type
 
'MISC CONSTs
Private Const VT_BY_REF = &H4000&
Private Const FILE_SHARE_READ = &H1
Private Const FILE_SHARE_WRITE = &H2
Private Const MOVEFILE_REPLACE_EXISTING = &H1
Private Const FILE_ATTRIBUTE_TEMPORARY = &H100
Private Const FILE_ATTRIBUTE_NORMAL = &H80
Private Const FILE_BEGIN = 0
Private Const CREATE_NEW = 1
Private Const OPEN_EXISTING = 3
Private Const OPEN_ALWAYS = 4
Private Const GENERIC_READ = &H80000000
Private Const GENERIC_WRITE = &H40000000
Private Const PAGE_READWRITE = 4
Private Const FILE_MAP_WRITE = &H2
Private Const FILE_MAP_READ = &H4
Private Const FADF_FIXEDSIZE = &H10
 
Private Sub MapFileMemory()
 
    Dim hFile As Long, sFile As String, lPointer As Long, hFileMap As Long, lFileLen As Long, uTemp As SafeArray
    Dim bBytes() As Byte
 
    sFile = "D:\Test2.txt"                              'Set the filename.
                                                        'Map it.
    hFile = CreateFile(sFile, GENERIC_READ Or GENERIC_WRITE, 0, 0, OPEN_ALWAYS, FILE_ATTRIBUTE_NORMAL, 0)
    hFileMap = CreateFileMapping(hFile, 0, PAGE_READWRITE, 0, 0, "MySharedMapping")
    lPointer = MapViewOfFile(hFileMap, FILE_MAP_WRITE, 0, 0, 0)
 
    bBytes = StrConv("TestTestTest", vbFromUnicode)     'Create an array.
    
    lFileLen = FileLen(sFile)                           'Find the length of the target file.
    
    If GetArrayInfo(bBytes, uTemp) Then                 'Load the UDT with the array info.
        uTemp.cbElements = 1                            'Set element size to a byte.
        uTemp.rgsabound.cElements = lFileLen            'Set the UBound of the array.
        uTemp.fFeature = uTemp.fFeature And FADF_FIXEDSIZE  'Set the "Fixed size" flag, SHOULD MAKE REDIM FAIL!
        uTemp.pvData = lPointer                         'Point it to the memory mapped file as it's data.
        Call AlterArray(bBytes, uTemp)                  'Write the UDT over the old array.
    End If
 
    For lFileLen = LBound(bBytes) To UBound(bBytes)     'This should be reading from the file.
        Debug.Print bBytes(lFileLen)
    Next lFileLen
 
    bBytes(0) = 61                                      'Change the first char to a "=" to see if it worked.
    
    UnmapViewOfFile lPointer                            'Release the memory map.
    CloseHandle hFile                                   'Close the opened file.
 
End Sub
 
Private Function GetArrayInfo(vArray As Variant, uInfo As SafeArray) As Boolean
    
    'NOTE, the array is passed as a variant so we can get it's absolute memory address.  This function
    'loads a copy of the SafeArray structure into the UDT.
    
    Dim lPointer As Long, iVType As Integer
    
    If Not IsArray(vArray) Then Exit Function               'Need to work with a safearray here.
 
    With uInfo
        CopyMemory iVType, vArray, 2                        'First 2 bytes are the subtype.
        CopyMemory lPointer, ByVal VarPtr(vArray) + 8, 4    'Get the pointer.
 
        If (iVType And VT_BY_REF) <> 0 Then                 'Test for subtype "pointer"
            CopyMemory lPointer, ByVal lPointer, 4          'Get the real address.
        End If
        
        CopyMemory uInfo.cDim, ByVal lPointer, 16           'Write the safearray to the passed UDT.
        
        If uInfo.cDim = 1 Then                              'Can't do multi-dimensional
            CopyMemory .rgsabound, ByVal lPointer + 16, LenB(.rgsabound)
            GetArrayInfo = True
        End If
    End With
 
End Function
 
Private Function AlterArray(vArray As Variant, uInfo As SafeArray) As Boolean
    
    'NOTE, the array is passed as a variant so we can get it's absolute memory address.  This function
    'writes the SafeArray UDT information into the actual memory address of the passed array.
    
    Dim lPointer As Long, iVType As Integer
 
    If Not IsArray(vArray) Then Exit Function
 
    With uInfo
        CopyMemory iVType, vArray, 2                        'Get the variant subtype
        CopyMemory lPointer, ByVal VarPtr(vArray) + 8, 4    'Get the pointer.
 
        If (iVType And VT_BY_REF) <> 0 Then                 'Test for subtype "pointer"
            CopyMemory lPointer, ByVal lPointer, 4          'Get the real address.
        End If
 
        CopyMemory ByVal lPointer, uInfo.cDim, 16           'Overwrite the array with the UDT.
 
        If uInfo.cDim = 1 Then                              'Multi-dimensions might wipe out other memory.
            CopyMemory ByVal lPointer + 16, .rgsabound, LenB(.rgsabound)
            AlterArray = True
        End If
 
    End With
 
End Function

That's a very nice code there, i'm sure I'll use it in some of my applications soon.

But, is it possible to do the opposite ?

I mean, to have a memory buffer/pointer and map it on the hard-drive as a file ?

Edit, After reading my own question, I realized you might be asking this: Can't you just write the data to a file ?
Well, no.... I don't want the file to exist on the hard-drive, but I want other applications to open the file as if it's on the hard-dive, and read/write to it.
When that other app is done, then my app will process the data in memory further.

Quote:

---

Originally Posted by CVMichael

That's a very nice code there, i'm sure I'll use it in some of my applications soon.

But, is it possible to do the opposite ?

I mean, to have a memory buffer/pointer and map it on the hard-drive as a file ?

---

I'm not quite sure. You can get a virtual "file handle" to a read-write named pipe instance. That should return a file handle that is compatible with memory mapping. See the CreateNamedPipe API. If you open it in duplex mode, it should act more or less like a file in memory. Let me play around with it for a bit and see if I can create something useable out of it in VB.

I'm trying to grasp this code in what it actually does. I have a problem where I have a large text file approximately 1gb that I'm trying to read and Parse information from and was looking to create a memory map file to it. As trying a StreamReader take way to long to process as reading from disk is very labor intensive, and I need a quicker process.

From what I understand from this code all it does is create a byte array from the word TestTestTest and modifys the first T in the Test string to an "=" and doesn't actually read from the file location.

I have another process that creates our large file and I need a VB application to open and process that file quickly looking for and parsing out certain information out of the file based on different text strings. Would this code work for me or am I just chasing my tail?

Welcome to VBForums :wave:

This code won't work for you as it is for VB6 and earlier, and you are using VB2008 (VB.Net) which is significantly different.

There maybe a way to convert the code to VB.Net, but it would be better to use the methods provided instead... try searching our VB.Net CodeBank forum, or asking a question in our VB.Net forum.

Quote:

---

Originally Posted by ScottCli

From what I understand from this code all it does is create a byte array from the word TestTestTest and modifys the first T in the Test string to an "=" and doesn't actually read from the file location.

---

That's kind of correct... The "TestTestTest" string is actually in a file. The code maps the file to memory and places an array structure onto it. This allows the file to treated like an array in memory. It does read from the file location, but doesn't load it into memory.

Quote:

---

Originally Posted by ScottCli

I have another process that creates our large file and I need a VB application to open and process that file quickly looking for and parsing out certain information out of the file based on different text strings. Would this code work for me or am I just chasing my tail?

---

This is probably not the best approach for this application. If you need to traverse the entire file, you'll have to do file accesses on the entire thing anyway. A memory map would most likely be slower, as you don't get to take advantage of block reads, etc. It's more useful for sharing memory space between 2 applications.

Sorry for bringing up an old thread, but I've been doing a lot of research into this for a project at work, and have some answers to some of these questions.

VB.Net makes this available with System.IO.MemoryMappedFiles. I don't know if the VB.net version takes care of the semaphors and mutexes needed to prevent the race-conditions and deadlocks that can occur using MMFs, it would not surprise me too much if it did but don't assume it does.

When you map a file into memory, it works exactly the same as the system's page-file. The file gets mapped, but nothing is actually loaded into memory until it is referenced. The pagefile is actually a system managed memory-mapped file itself. This makes it very memory efficient for cases where the file does not need to be loaded all at once, but can be loaded incrementally.

Depending on how you impliment it, MMFs can be perfect for accessing extremely large files without consuming much RAM, because Windows will only load up the referenced portion of the file in 4kb chunks (exactly the same way it deals with Virtual Memory).

The primary danger for a single process is in attempting to access more of the file at one time than is available for addressing - if you have 1gb of ram available and you attempt to access 1.5gb of a file at one time you will thrash the hard drive. For 32bit systems this means there is a hard limit of 4gb and a practical limit of about 3gb for accessible data at one time. There is no limit to the overall file size, however, so if you are careful to release pages when they aren't needed the sky is the limit. Pretty cool huh?

Because the file is mapped to memory, another process can access the file in memory by requesting the pointer for the file itself. In this way multiple processes can read, write, or execute the data in memory without needing to open their own copy of a file and multiply the amount of memory used.

Also, unless you specify no caching when you map the file to memory, all IO operations occur in memory and nothing is written to disk until pages are released from memory. This makes MMF operations extremely fast, using the same techniqes as Virtual Memory to maximize the read/write times. For multiple reads/writes, using an MMF instead of streamreading/writing can be orders of magnitude faster.

Things to look out for are race conditions and deadlocks - if either of these occur you get a page fault. Also, you should not use the streamread/write functions on a memory mapped file, it does NOT update the data currently in memory, so if any of the data in memory is changed on disk it will simply get re-written when the page in memory is eventually released. The converse is also true, the data in memory may have changed but the file on disk will not be updated until the page is released. You may be able to get away with this if you dump the pages in memory before opening and reading/writing the file, but that may take away the advantages of using MMFs.

Last but not least, you can actually use the CreateFileMapping function with a file handle of -1 and a defined length to create a block of shared system memory for VB6 applications without creating a file on disk. Windows allocates a portion of the pagefile for the file mapping, and allocates RAM dynamically. This behaves identically to other file mappings, it simply has no permanent file associated with it.

I forgot to mention it, but the necessity of Comintern's code should make it clear that VB6 cannot directly manipulate the mapped file via pointers - you need to dump the data into a variable of some sort (like the SafeArray structure) before you can manipulate it and send it back.

However, this just gives you a few more hoops to jump through, and if you are careful with your code you can maintain all the benefits of shared memory and blazing IO that MMFs give you.

My project is saving acquisition data to a file. The data as: 900 bytes/sample, 500 samples/second. I receive each data sample from a data acquisition board. This sample will be checked and if it okay, my program will write it to a file. When I press a button (on the my program), the program will close the file.

I had a look the example above. The data stay in memory and only write to HDD after released pages. This method will faster a normal method in my program (I think that).

But in my program, I can write the data continue to end posision of file. But in this example, I must declare a fix array before mapping file. Can you find a any way to do with a file without knowing len?

Creating mapped files are useful for reading if you don't want to create a interim array. If you need to just write the file, no need to map it, simply use WriteFile API and pass your 900 bytes as the buffer.

Ex assuming you created a file:

Code:

---

' sample code
Dim lWrote As Long
WriteFile hFile, buffer900bytes(0), 900, lWrote, ByVal 0&
If lWrote = 900 Then ' all is ok, else not all bytes were written

End If

---

' WriteFile declaration

Code:

---

Private Declare Function WriteFile Lib "kernel32" (ByVal hFile As Long, lpBuffer As Any, ByVal nNumberOfBytesToWrite As Long, lpNumberOfBytesWritten As Long, lpOverlapped As Any) As Long

---

Edited: WriteFile API has two advantages over VB's Put # statement, regarding arrays & can also support unicode filenames/paths.
1) You don't have to write the entire array because you tell the API how many bytes to write
2) You don't have to start with the 1st element of the array because you tell the API from where to start writing.
If these advantages don't apply, why not just use VB's Input/Output functions instead.

http://msdn.microsoft.com/en-us/libr...(v=vs.85).aspx

Quote:

---

Originally Posted by Comintern

The "TestTestTest" string is actually in a file... It does read from the file location, but doesn't load it into memory.

---

Sorry to dredge up a very old thread, but VB6 is a very old language, and this posting helped me get going on mapped files.

Even though Comintern's code is great (though I would not regard his suggestion that the SafeArray structure be restored before the bBytes array goes out of scope as optional; gawdnose what VB6's memory manager might do if you end up freeing the same block twice, and, at minimum, you will probably have a memory leak from the original allocation), I believe neither of his statements above is true. The "TestTestTest" string is never in a file. It serves only to return a newly allocated String so there will be something in the bBytes array prior to modifying its SafeArray structure to point to the mapped memory. You actually don't need to do this. When you declare the array, you can give it a dimension if you want to ("Dim bBytes(24) as Byte"), though I doubt even this is really necessary. Regardless, whatever is in the bBytes array before altering it to poing at the mapped memory is never in a file.

Whatever is in the file ("D:\Test2.txt", in Comintern's examples) does get loaded into memory when MapViewOfFile is called. You can map just a chunk or, as Comintern has done, the whole thing. It's a great way to poke around in a file, though, utimately, if all you want to do is to read and write the file, this is a complicated way to do it. Easier would be to read the whole file into a byte array with ordinary file i/o calls, do your work, then write the whole array back again.

Now, the power of Comintern's code is, I would say, in the fact that, once it has been mapped, any other process can also map the same region of memory. Where his code calls "CreateFileMapping," other processes would call "OpenFileMapping" (check out MSDN doc on this, as the calling sequences are surprisingly dissimilar). After both processes have mapped the file and created views, and then altered their bBytes arrays (or whatever they call them), both are then sharing the contents of that array with each other. It becomes one array shared by two (or more) processes. Other IPC mechanisms would be needed to synchronize communications (one might use, say, sockets to alert one process that the other has changed the contents of the shared memory), but this (according to MSDN) the only way for 32-bit Windows processes to share memory.

With a few lines of changed code, I've made Comintern's program into two programs, one that creates the mapping and one that opens it. Both can read and write the same array, with each seeing the other's changes. Now, I have no idea if memory caching could defeat this (that is, if one process caches the memory it writes to, does that mean the other process won't read the changes until after the cache is written through to main memory?). If so, one would have to use some kind of memory fencing to avoid it. But, the problem doesn't come up in my test programs, so maybe mapped memory always writes through from cache to main storage. (If anyone can find a reference that says so, please let me know!)

Great sample program. Thanks for posting it.

UPDATE: According to MSDN, mapped views of the same file are always coherent: no need for memory fences.