Results 1 to 11 of 11

Thread: VB6 - Treat a file like an array without reading to memory.

  1. #1

    Thread Starter
    Fanatic Member Comintern's Avatar
    Join Date
    Nov 2004
    Location
    Lincoln, NE
    Posts
    826

    VB6 - Treat a file like an array without reading to memory.

    Here's a little snippet of code I posted over on CodeGuru that I thought people over here might be able to make use of. It creates a memory map of a file, meaning that instead of loading the file into memory, it creates a pointer to the file's physical location on the disk. So, you can treat the file as a variable, and any changes you make to the variables are also changing the target file. While useful in and of itself, it's a little limiting without a nice convienent way to index into it (are you thinking array? I am).

    The following sample code opens a memory mapped file, creates a byte array, and then modifies the data pointer in the array to point at the file. So, you can do 'reads' with bVar = bByte(x), and 'writes' with bByte(x) = bVar.

    It wouldn't be hard to convert from a byte array to another type of array (string excluded, unless the strings were fixed length). All you need to do in that case would be to change the cbElements of the array structure to the size of the chunk of data you want--i.e., if you wanted to treat the file as an array of Longs, cbElements would be 4, a UDT would be LenB(uType), etc. The second type (SafeBound) holds the LBound and UBound information. I wouldn't change the LBound, but you'll want to manually calculate the UBound based on the length of the file and the size of the elements. So, if the file was 1000 bytes and you wanted to treat it as an array of Longs, you would set uTemp.cbElements to 4 and set uTemp.rgsabound.cElements to 249.

    Couple of things you might want to do would be to add error checking for the API calls and re-build the original array (mainly just point it back to what it was pointing to before you hacked it) after you remove the memory map to make sure that the memory for the original array data gets de-allocated when it's garbage collected.

    Here's the code. This example opens a file and changes the first byte to "=".
    VB Code:
    1. 'API DECs
    2. Private Declare Sub CopyMemory Lib "kernel32" Alias "RtlMoveMemory" (pDst As Any, pSrc As Any, _
    3.                                                                      ByVal ByteLen As Long)
    4. Private Declare Function CreateFile Lib "kernel32" Alias "CreateFileA" (ByVal lpFileName As String, _
    5.                                          ByVal dwDesiredAccess As Long, ByVal dwShareMode As Long, _
    6.                                          ByVal lpSecurityAttributes As Long, ByVal dwCreationDisposition As Long, _
    7.                                          ByVal dwFlagsAndAttributes As Long, ByVal hTemplateFile As Long) As Long
    8. Private Declare Function CloseHandle Lib "kernel32" (ByVal hObject As Long) As Long
    9.  
    10. 'MEMORY MAPPING APIs
    11. Private Declare Function MapViewOfFile Lib "kernel32.dll" (ByVal hFile As Long, ByVal dwDesiredAccess As Long, _
    12.                                          ByVal dwFileOffsetHigh As Long, ByVal dwFileOffsetLow As Long, _
    13.                                          ByVal dwNumberOfBytesToMap As Long) As Long
    14. Private Declare Function CreateFileMapping Lib "kernel32.dll" Alias "CreateFileMappingA" (ByVal hFile As Long, _
    15.                                          ByVal lpAttributes As Long, ByVal flProtect As Long, _
    16.                                          ByVal dwMaximumSizeHigh As Long, ByVal dwMaximumSizeLow As Long, _
    17.                                          ByVal lpName As String) As Long
    18. Private Declare Function UnmapViewOfFile Lib "kernel32.dll" (ByVal lpBaseAddress As Long) As Boolean
    19.  
    20. 'STRUCTS FOR THE SAFEARRAY:
    21. Private Type SafeBound
    22.     cElements As Long
    23.     lLbound As Long
    24. End Type
    25.  
    26. Private Type SafeArray
    27.     cDim As Integer
    28.     fFeature As Integer
    29.     cbElements As Long
    30.     cLocks As Long
    31.     pvData As Long
    32.     rgsabound As SafeBound
    33. End Type
    34.  
    35. 'MISC CONSTs
    36. Private Const VT_BY_REF = &H4000&
    37. Private Const FILE_SHARE_READ = &H1
    38. Private Const FILE_SHARE_WRITE = &H2
    39. Private Const MOVEFILE_REPLACE_EXISTING = &H1
    40. Private Const FILE_ATTRIBUTE_TEMPORARY = &H100
    41. Private Const FILE_ATTRIBUTE_NORMAL = &H80
    42. Private Const FILE_BEGIN = 0
    43. Private Const CREATE_NEW = 1
    44. Private Const OPEN_EXISTING = 3
    45. Private Const OPEN_ALWAYS = 4
    46. Private Const GENERIC_READ = &H80000000
    47. Private Const GENERIC_WRITE = &H40000000
    48. Private Const PAGE_READWRITE = 4
    49. Private Const FILE_MAP_WRITE = &H2
    50. Private Const FILE_MAP_READ = &H4
    51. Private Const FADF_FIXEDSIZE = &H10
    52.  
    53. Private Sub MapFileMemory()
    54.  
    55.     Dim hFile As Long, sFile As String, lPointer As Long, hFileMap As Long, lFileLen As Long, uTemp As SafeArray
    56.     Dim bBytes() As Byte
    57.  
    58.     sFile = "D:\Test2.txt"                              'Set the filename.
    59.                                                         'Map it.
    60.     hFile = CreateFile(sFile, GENERIC_READ Or GENERIC_WRITE, 0, 0, OPEN_ALWAYS, FILE_ATTRIBUTE_NORMAL, 0)
    61.     hFileMap = CreateFileMapping(hFile, 0, PAGE_READWRITE, 0, 0, "MySharedMapping")
    62.     lPointer = MapViewOfFile(hFileMap, FILE_MAP_WRITE, 0, 0, 0)
    63.  
    64.     bBytes = StrConv("TestTestTest", vbFromUnicode)     'Create an array.
    65.    
    66.     lFileLen = FileLen(sFile)                           'Find the length of the target file.
    67.    
    68.     If GetArrayInfo(bBytes, uTemp) Then                 'Load the UDT with the array info.
    69.         uTemp.cbElements = 1                            'Set element size to a byte.
    70.         uTemp.rgsabound.cElements = lFileLen            'Set the UBound of the array.
    71.         uTemp.fFeature = uTemp.fFeature And FADF_FIXEDSIZE  'Set the "Fixed size" flag, SHOULD MAKE REDIM FAIL!
    72.         uTemp.pvData = lPointer                         'Point it to the memory mapped file as it's data.
    73.         Call AlterArray(bBytes, uTemp)                  'Write the UDT over the old array.
    74.     End If
    75.  
    76.     For lFileLen = LBound(bBytes) To UBound(bBytes)     'This should be reading from the file.
    77.         Debug.Print bBytes(lFileLen)
    78.     Next lFileLen
    79.  
    80.     bBytes(0) = 61                                      'Change the first char to a "=" to see if it worked.
    81.    
    82.     UnmapViewOfFile lPointer                            'Release the memory map.
    83.     CloseHandle hFile                                   'Close the opened file.
    84.  
    85. End Sub
    86.  
    87. Private Function GetArrayInfo(vArray As Variant, uInfo As SafeArray) As Boolean
    88.    
    89.     'NOTE, the array is passed as a variant so we can get it's absolute memory address.  This function
    90.     'loads a copy of the SafeArray structure into the UDT.
    91.    
    92.     Dim lPointer As Long, iVType As Integer
    93.    
    94.     If Not IsArray(vArray) Then Exit Function               'Need to work with a safearray here.
    95.  
    96.     With uInfo
    97.         CopyMemory iVType, vArray, 2                        'First 2 bytes are the subtype.
    98.         CopyMemory lPointer, ByVal VarPtr(vArray) + 8, 4    'Get the pointer.
    99.  
    100.         If (iVType And VT_BY_REF) <> 0 Then                 'Test for subtype "pointer"
    101.             CopyMemory lPointer, ByVal lPointer, 4          'Get the real address.
    102.         End If
    103.        
    104.         CopyMemory uInfo.cDim, ByVal lPointer, 16           'Write the safearray to the passed UDT.
    105.        
    106.         If uInfo.cDim = 1 Then                              'Can't do multi-dimensional
    107.             CopyMemory .rgsabound, ByVal lPointer + 16, LenB(.rgsabound)
    108.             GetArrayInfo = True
    109.         End If
    110.     End With
    111.  
    112. End Function
    113.  
    114. Private Function AlterArray(vArray As Variant, uInfo As SafeArray) As Boolean
    115.    
    116.     'NOTE, the array is passed as a variant so we can get it's absolute memory address.  This function
    117.     'writes the SafeArray UDT information into the actual memory address of the passed array.
    118.    
    119.     Dim lPointer As Long, iVType As Integer
    120.  
    121.     If Not IsArray(vArray) Then Exit Function
    122.  
    123.     With uInfo
    124.         CopyMemory iVType, vArray, 2                        'Get the variant subtype
    125.         CopyMemory lPointer, ByVal VarPtr(vArray) + 8, 4    'Get the pointer.
    126.  
    127.         If (iVType And VT_BY_REF) <> 0 Then                 'Test for subtype "pointer"
    128.             CopyMemory lPointer, ByVal lPointer, 4          'Get the real address.
    129.         End If
    130.  
    131.         CopyMemory ByVal lPointer, uInfo.cDim, 16           'Overwrite the array with the UDT.
    132.  
    133.         If uInfo.cDim = 1 Then                              'Multi-dimensions might wipe out other memory.
    134.             CopyMemory ByVal lPointer + 16, .rgsabound, LenB(.rgsabound)
    135.             AlterArray = True
    136.         End If
    137.  
    138.     End With
    139.  
    140. End Function
    Last edited by Comintern; May 8th, 2006 at 08:37 AM. Reason: Added "VB6" to the title.

  2. #2
    PowerPoster
    Join Date
    Feb 2002
    Location
    Canada, Toronto
    Posts
    5,802

    Re: VB6 - Treat a file like an array without reading to memory.

    That's a very nice code there, i'm sure I'll use it in some of my applications soon.

    But, is it possible to do the opposite ?

    I mean, to have a memory buffer/pointer and map it on the hard-drive as a file ?

    Edit, After reading my own question, I realized you might be asking this: Can't you just write the data to a file ?
    Well, no.... I don't want the file to exist on the hard-drive, but I want other applications to open the file as if it's on the hard-dive, and read/write to it.
    When that other app is done, then my app will process the data in memory further.
    Last edited by CVMichael; May 8th, 2006 at 11:22 AM.

  3. #3

    Thread Starter
    Fanatic Member Comintern's Avatar
    Join Date
    Nov 2004
    Location
    Lincoln, NE
    Posts
    826

    Re: VB6 - Treat a file like an array without reading to memory.

    Quote Originally Posted by CVMichael
    That's a very nice code there, i'm sure I'll use it in some of my applications soon.

    But, is it possible to do the opposite ?

    I mean, to have a memory buffer/pointer and map it on the hard-drive as a file ?
    I'm not quite sure. You can get a virtual "file handle" to a read-write named pipe instance. That should return a file handle that is compatible with memory mapping. See the CreateNamedPipe API. If you open it in duplex mode, it should act more or less like a file in memory. Let me play around with it for a bit and see if I can create something useable out of it in VB.

  4. #4
    New Member
    Join Date
    Oct 2008
    Posts
    1

    Re: VB6 - Treat a file like an array without reading to memory.

    I'm trying to grasp this code in what it actually does. I have a problem where I have a large text file approximately 1gb that I'm trying to read and Parse information from and was looking to create a memory map file to it. As trying a StreamReader take way to long to process as reading from disk is very labor intensive, and I need a quicker process.

    From what I understand from this code all it does is create a byte array from the word TestTestTest and modifys the first T in the Test string to an "=" and doesn't actually read from the file location.

    I have another process that creates our large file and I need a VB application to open and process that file quickly looking for and parsing out certain information out of the file based on different text strings. Would this code work for me or am I just chasing my tail?

  5. #5
    Super Moderator si_the_geek's Avatar
    Join Date
    Jul 2002
    Location
    Bristol, UK
    Posts
    41,929

    Re: VB6 - Treat a file like an array without reading to memory.

    Welcome to VBForums

    This code won't work for you as it is for VB6 and earlier, and you are using VB2008 (VB.Net) which is significantly different.

    There maybe a way to convert the code to VB.Net, but it would be better to use the methods provided instead... try searching our VB.Net CodeBank forum, or asking a question in our VB.Net forum.

  6. #6

    Thread Starter
    Fanatic Member Comintern's Avatar
    Join Date
    Nov 2004
    Location
    Lincoln, NE
    Posts
    826

    Re: VB6 - Treat a file like an array without reading to memory.

    Quote Originally Posted by ScottCli
    From what I understand from this code all it does is create a byte array from the word TestTestTest and modifys the first T in the Test string to an "=" and doesn't actually read from the file location.
    That's kind of correct... The "TestTestTest" string is actually in a file. The code maps the file to memory and places an array structure onto it. This allows the file to treated like an array in memory. It does read from the file location, but doesn't load it into memory.
    Quote Originally Posted by ScottCli
    I have another process that creates our large file and I need a VB application to open and process that file quickly looking for and parsing out certain information out of the file based on different text strings. Would this code work for me or am I just chasing my tail?
    This is probably not the best approach for this application. If you need to traverse the entire file, you'll have to do file accesses on the entire thing anyway. A memory map would most likely be slower, as you don't get to take advantage of block reads, etc. It's more useful for sharing memory space between 2 applications.

  7. #7
    New Member
    Join Date
    Jul 2009
    Posts
    5

    Re: VB6 - Treat a file like an array without reading to memory.

    Sorry for bringing up an old thread, but I've been doing a lot of research into this for a project at work, and have some answers to some of these questions.

    VB.Net makes this available with System.IO.MemoryMappedFiles. I don't know if the VB.net version takes care of the semaphors and mutexes needed to prevent the race-conditions and deadlocks that can occur using MMFs, it would not surprise me too much if it did but don't assume it does.

    When you map a file into memory, it works exactly the same as the system's page-file. The file gets mapped, but nothing is actually loaded into memory until it is referenced. The pagefile is actually a system managed memory-mapped file itself. This makes it very memory efficient for cases where the file does not need to be loaded all at once, but can be loaded incrementally.

    Depending on how you impliment it, MMFs can be perfect for accessing extremely large files without consuming much RAM, because Windows will only load up the referenced portion of the file in 4kb chunks (exactly the same way it deals with Virtual Memory).

    The primary danger for a single process is in attempting to access more of the file at one time than is available for addressing - if you have 1gb of ram available and you attempt to access 1.5gb of a file at one time you will thrash the hard drive. For 32bit systems this means there is a hard limit of 4gb and a practical limit of about 3gb for accessible data at one time. There is no limit to the overall file size, however, so if you are careful to release pages when they aren't needed the sky is the limit. Pretty cool huh?

    Because the file is mapped to memory, another process can access the file in memory by requesting the pointer for the file itself. In this way multiple processes can read, write, or execute the data in memory without needing to open their own copy of a file and multiply the amount of memory used.

    Also, unless you specify no caching when you map the file to memory, all IO operations occur in memory and nothing is written to disk until pages are released from memory. This makes MMF operations extremely fast, using the same techniqes as Virtual Memory to maximize the read/write times. For multiple reads/writes, using an MMF instead of streamreading/writing can be orders of magnitude faster.

    Things to look out for are race conditions and deadlocks - if either of these occur you get a page fault. Also, you should not use the streamread/write functions on a memory mapped file, it does NOT update the data currently in memory, so if any of the data in memory is changed on disk it will simply get re-written when the page in memory is eventually released. The converse is also true, the data in memory may have changed but the file on disk will not be updated until the page is released. You may be able to get away with this if you dump the pages in memory before opening and reading/writing the file, but that may take away the advantages of using MMFs.

    Last but not least, you can actually use the CreateFileMapping function with a file handle of -1 and a defined length to create a block of shared system memory for VB6 applications without creating a file on disk. Windows allocates a portion of the pagefile for the file mapping, and allocates RAM dynamically. This behaves identically to other file mappings, it simply has no permanent file associated with it.

  8. #8
    New Member
    Join Date
    Jul 2009
    Posts
    5

    Re: VB6 - Treat a file like an array without reading to memory.

    I forgot to mention it, but the necessity of Comintern's code should make it clear that VB6 cannot directly manipulate the mapped file via pointers - you need to dump the data into a variable of some sort (like the SafeArray structure) before you can manipulate it and send it back.

    However, this just gives you a few more hoops to jump through, and if you are careful with your code you can maintain all the benefits of shared memory and blazing IO that MMFs give you.

  9. #9
    New Member
    Join Date
    Dec 2009
    Posts
    8

    Re: VB6 - Treat a file like an array without reading to memory.

    My project is saving acquisition data to a file. The data as: 900 bytes/sample, 500 samples/second. I receive each data sample from a data acquisition board. This sample will be checked and if it okay, my program will write it to a file. When I press a button (on the my program), the program will close the file.

    I had a look the example above. The data stay in memory and only write to HDD after released pages. This method will faster a normal method in my program (I think that).

    But in my program, I can write the data continue to end posision of file. But in this example, I must declare a fix array before mapping file. Can you find a any way to do with a file without knowing len?

  10. #10
    VB-aholic & Lovin' It LaVolpe's Avatar
    Join Date
    Oct 2007
    Location
    Beside Waldo
    Posts
    19,541

    Re: VB6 - Treat a file like an array without reading to memory.

    Creating mapped files are useful for reading if you don't want to create a interim array. If you need to just write the file, no need to map it, simply use WriteFile API and pass your 900 bytes as the buffer.

    Ex assuming you created a file:
    Code:
    ' sample code
    Dim lWrote As Long
    WriteFile hFile, buffer900bytes(0), 900, lWrote, ByVal 0&
    If lWrote = 900 Then ' all is ok, else not all bytes were written
    
    End If
    ' WriteFile declaration
    Code:
    Private Declare Function WriteFile Lib "kernel32" (ByVal hFile As Long, lpBuffer As Any, ByVal nNumberOfBytesToWrite As Long, lpNumberOfBytesWritten As Long, lpOverlapped As Any) As Long
    Edited: WriteFile API has two advantages over VB's Put # statement, regarding arrays & can also support unicode filenames/paths.
    1) You don't have to write the entire array because you tell the API how many bytes to write
    2) You don't have to start with the 1st element of the array because you tell the API from where to start writing.
    If these advantages don't apply, why not just use VB's Input/Output functions instead.
    Last edited by LaVolpe; Dec 30th, 2009 at 12:17 PM.
    Insomnia is just a byproduct of, "It can't be done"

    Classics Enthusiast? Here's my 1969 Mustang Mach I Fastback. Her sister '67 Coupe has been adopted

    Newbie? Novice? Bored? Spend a few minutes browsing the FAQ section of the forum.
    Read the HitchHiker's Guide to Getting Help on the Forums.
    Here is the list of TAGs you can use to format your posts
    Here are VB6 Help Files online


    {Alpha Image Control} {Memory Leak FAQ} {Unicode Open/Save Dialog} {Resource Image Viewer/Extractor}
    {VB and DPI Tutorial} {Manifest Creator} {UserControl Button Template} {stdPicture Render Usage}

  11. #11
    New Member
    Join Date
    Mar 2013
    Location
    Northern Virginia, USA
    Posts
    1

    Re: VB6 - Treat a file like an array without reading to memory.

    http://msdn.microsoft.com/en-us/libr...(v=vs.85).aspx
    Quote Originally Posted by Comintern View Post
    The "TestTestTest" string is actually in a file... It does read from the file location, but doesn't load it into memory.
    Sorry to dredge up a very old thread, but VB6 is a very old language, and this posting helped me get going on mapped files.

    Even though Comintern's code is great (though I would not regard his suggestion that the SafeArray structure be restored before the bBytes array goes out of scope as optional; gawdnose what VB6's memory manager might do if you end up freeing the same block twice, and, at minimum, you will probably have a memory leak from the original allocation), I believe neither of his statements above is true. The "TestTestTest" string is never in a file. It serves only to return a newly allocated String so there will be something in the bBytes array prior to modifying its SafeArray structure to point to the mapped memory. You actually don't need to do this. When you declare the array, you can give it a dimension if you want to ("Dim bBytes(24) as Byte"), though I doubt even this is really necessary. Regardless, whatever is in the bBytes array before altering it to poing at the mapped memory is never in a file.

    Whatever is in the file ("D:\Test2.txt", in Comintern's examples) does get loaded into memory when MapViewOfFile is called. You can map just a chunk or, as Comintern has done, the whole thing. It's a great way to poke around in a file, though, utimately, if all you want to do is to read and write the file, this is a complicated way to do it. Easier would be to read the whole file into a byte array with ordinary file i/o calls, do your work, then write the whole array back again.

    Now, the power of Comintern's code is, I would say, in the fact that, once it has been mapped, any other process can also map the same region of memory. Where his code calls "CreateFileMapping," other processes would call "OpenFileMapping" (check out MSDN doc on this, as the calling sequences are surprisingly dissimilar). After both processes have mapped the file and created views, and then altered their bBytes arrays (or whatever they call them), both are then sharing the contents of that array with each other. It becomes one array shared by two (or more) processes. Other IPC mechanisms would be needed to synchronize communications (one might use, say, sockets to alert one process that the other has changed the contents of the shared memory), but this (according to MSDN) the only way for 32-bit Windows processes to share memory.

    With a few lines of changed code, I've made Comintern's program into two programs, one that creates the mapping and one that opens it. Both can read and write the same array, with each seeing the other's changes. Now, I have no idea if memory caching could defeat this (that is, if one process caches the memory it writes to, does that mean the other process won't read the changes until after the cache is written through to main memory?). If so, one would have to use some kind of memory fencing to avoid it. But, the problem doesn't come up in my test programs, so maybe mapped memory always writes through from cache to main storage. (If anyone can find a reference that says so, please let me know!)

    Great sample program. Thanks for posting it.

    UPDATE: According to MSDN, mapped views of the same file are always coherent: no need for memory fences.
    Last edited by stevensrmiller; Mar 15th, 2013 at 09:24 AM.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width