Results 1 to 18 of 18

Thread: How much memory is taken up by an array?

  1. #1

    Thread Starter
    Fanatic Member
    Join Date
    Mar 2010
    Posts
    762

    How much memory is taken up by an array?

    I guess an array like this:
    Code:
       Dim L_Arr()      as Long
    Takes up 4 bytes for each item.
    For example if I have:
    Code:
       ReDim L_Arr(99) as Long
    Then it occupies 4 * 100 = 400 bytes.
    Am I right?

    What if I populate SOME but not all of those 100 items?
    Will it still occupy 400 bytes of memory?

    How about an array of strings:
    Code:
       Dim S_Arr()      as String
       ReDim S_Arr(99) as String
    How many bytes of memory will this occupy?

    Please advise.
    Thanks.

  2. #2
    PowerPoster Arnoutdv's Avatar
    Join Date
    Oct 2013
    Posts
    5,872

    Re: How much memory is taken up by an array?

    The value 0 takes also space, so yes
    For an array of strings the size is unknown until there are elements in the array.

    A long is 4 bytes, a dynamic string has a dynamic size.

  3. #3
    PowerPoster
    Join Date
    Feb 2006
    Posts
    24,482

    Re: How much memory is taken up by an array?

    Don't forget the array descriptor for dynamic arrays:

    NOTE: Arrays of any data type require 20 bytes of memory plus 4 bytes for each array dimension plus the number of bytes occupied by the data itself. The memory occupied by the data can be calculated by multiplying the number of data elements by the size of each element. For example, the data in a single-dimension array consisting of 4 Integer data elements of 2 bytes each occupies 8 bytes. The 8 bytes required for the data plus the 24 bytes of overhead brings the total memory requirement for the array to 32 bytes.

    A Variant containing an array requires 12 bytes more than the array alone.

  4. #4

    Thread Starter
    Fanatic Member
    Join Date
    Mar 2010
    Posts
    762

    Re: How much memory is taken up by an array?

    Quote Originally Posted by dilettante View Post
    Don't forget the array descriptor for dynamic arrays:
    NOTE: Arrays of any data type require 20 bytes of memory plus 4 bytes for each array dimension plus the number of bytes occupied by the data itself. The memory occupied by the data can be calculated by multiplying the number of data elements by the size of each element. For example, the data in a single-dimension array consisting of 4 Integer data elements of 2 bytes each occupies 8 bytes. The 8 bytes required for the data plus the 24 bytes of overhead brings the total memory requirement for the array to 32 bytes.

    A Variant containing an array requires 12 bytes more than the array alone.
    Thanks a lot for the explanation.
    But then, how about an array of strings?
    How do I calculate the size of memory taken up by an array of strings (for example a one-dimentional array of strings)?

    There is an overhead of 24 bytes.
    But then How about each element?
    How do I calculate that?

    Please advise.
    Thanks.

  5. #5
    PowerPoster
    Join Date
    Feb 2006
    Posts
    24,482

    Re: How much memory is taken up by an array?

    See https://www.aivosto.com/articles/str...l#memorylayout for they structure of each String or element of a String array.

    To get some background, let's see how strings are stored in RAM. VB6 stores strings in Unicode format. In COM terminology, a VB String is a BSTR. A String requires six overhead bytes plus 2 bytes for each character. Thus, you spend 6 + Len(string)*2 bytes for each string.

    The string starts with a 4-byte length prefix for the size of the string. It's not the character length, though. This 32-bit integer namely counts the number of bytes in the string (not counting the terminating 2 zero bytes). After the length prefix comes the actual text data, 2 bytes for each character. The last 2 bytes are zeros, denoting a NULL terminator (a Unicode null character).
    One thing left out of that is that a String can also hold ANSI characters or even arbitrary bytes, in which case only one byte per character is needed. But only do that if you know what you are doing. Most people don't.

  6. #6
    PowerPoster wqweto's Avatar
    Join Date
    May 2011
    Location
    Sofia, Bulgaria
    Posts
    5,120

    Re: How much memory is taken up by an array?

    Quote Originally Posted by dilettante View Post
    See https://www.aivosto.com/articles/str...l#memorylayout for they structure of each String or element of a String array.



    One thing left out of that is that a String can also hold ANSI characters or even arbitrary bytes, in which case only one byte per character is needed. But only do that if you know what you are doing. Most people don't.
    So the count in the 4-byte prefix is for instance 15 (odd number) but does this string terminate with one zero (it's ANSI string, isn't it), two zeroes (this is what BSTR specification says) or three zeroes (why not pad it to even number so the buffer can be traversed using 16-bit register and LODSW instruction)?

    Another thing to consider is that COM string allocator uses some space for book-keeping (every allocator does) so there is even more overhead not calculated here as close as 16 bytes per allocation (but small strings are cached in some arena allocator probably, probably even more complicated).

    cheers,
    </wqw>

  7. #7

    Thread Starter
    Fanatic Member
    Join Date
    Mar 2010
    Posts
    762

    Re: How much memory is taken up by an array?

    Quote Originally Posted by dilettante View Post
    One thing left out of that is that a String can also hold ANSI characters or even arbitrary bytes, in which case only one byte per character is needed. But only do that if you know what you are doing. Most people don't.
    Thanks.
    How do I know whether each character is stored as ANSI (one byte) or unicode (two bytes)?

    For example lets say I have this:
    Code:
       S_Arr(5) = "abcdefgh"
    Are these characters stored as ANSI (one byte per character) or Unicode (two bytes per character)?

    How about this:
    Code:
       S_Arr(5) = "AΣBωCμ"
    Are the first, third and fifth characters stored as ANSI (one byte each) and second, fourth and sixth characters stored as unicode (two bytes each)?

    Please advise.
    Thanks.

  8. #8
    PowerPoster Arnoutdv's Avatar
    Join Date
    Oct 2013
    Posts
    5,872

    Re: How much memory is taken up by an array?

    Try the Len() and LenB() methods and check the differences

  9. #9
    PowerPoster
    Join Date
    Feb 2006
    Posts
    24,482

    Re: How much memory is taken up by an array?

    Something like:

    Code:
    S = "ABC"
    Here the source code text is ANSI, but it gets stored as a Unicode literal which when run copies the Unicode text as S. Doesn't matter what is between the quotes, you always end up with Unicode. It starts out as ANSI because VB6 source code files are ANSI.

    Such a String always has 16-bit characters because all ANSI characters in literals should map to the normal range of characters.

    It is also possible to stuff 32-bit UTF-16LE surrogate pair characters into a VB6 String. These will be functional where they are supported, however stuff like Len() will misreport values because it was designed for a 16-bit fixed character length world.


    So you can stuff in anything you want, but nothing is going to read your mind about the encoding of those contents. That part is in the eye of the beholder.

    VB6, related controls, etc. all try hard to dumb things down. They expect to only see 16-bit Unicode characters and try to do implicit conversions to/from ANSI as needed to maintain the illusion of 7-bit ASCII extended to 8 bits using additional characters from different ANSI encodings (based on locale).

    It's a little like Jurassic Park where you are meant to keep your car on those rails. Get off the rails and you may find yourself eaten.

  10. #10
    PowerPoster wqweto's Avatar
    Join Date
    May 2011
    Location
    Sofia, Bulgaria
    Posts
    5,120

    Re: How much memory is taken up by an array?

    Quote Originally Posted by dilettante View Post
    Something like:

    Code:
    S = "ABC"
    Here the source code text is ANSI, but it gets stored as a Unicode literal which when run copies the Unicode text as S. Doesn't matter what is between the quotes, you always end up with Unicode. It starts out as ANSI because VB6 source code files are ANSI.
    So "ABC" in the sources is ANSI but it gets compiled to a Unicode string and is length prefixed so in the binary there is literally a byte sequence that starts with 03 00 00 00 (length) and then 41 00 (for A) 42 00 (for B) 43 00 (for C) 00 00 (for terminating zero) and this byte sequence in stored in the executable and loaded as part of the executable and it's address can be retrieved with StrPtr("ABC").

    What happens on assignment S = "ABC" is that a new chunk of memory is allocated from the strings heap and assigned to S and then from the literal address the byte sequence is copied to this new string i.e. S is cloned from the literal as stored in the executable but S does not point to the literal as it's a new dynamic chunk of memory. The literal in the executable is not a regular VB6 String as it is not allocated with SysAllocString and cannot be released with SysAllocFree but it's a correct BSTR struct as a static read-only byte sequence so it can be used with SysStringLen for instance or can be cloned with SysAllocString etc.

    cheers,
    </wqw>

  11. #11
    PowerPoster Arnoutdv's Avatar
    Join Date
    Oct 2013
    Posts
    5,872

    Re: How much memory is taken up by an array?

    But then, how about an array of strings?
    How do I calculate the size of memory taken up by an array of strings (for example a one-dimentional array of strings)?

    There is an overhead of 24 bytes.
    But then How about each element?
    How do I calculate that?
    Is there specific need to know exactly how much memory is in use by the arrays in your application?
    What is your goal?

  12. #12
    Super Moderator Shaggy Hiker's Avatar
    Join Date
    Aug 2002
    Location
    Idaho
    Posts
    38,989

    Re: How much memory is taken up by an array?

    I was wondering that, as well. In embedded systems, memory use can be crucial, but in modern computers, and increasingly in modern embedded systems, memory isn't a real concern. However, just because it isn't a real concern, that doesn't mean that it isn't a concern. You just have to be clear on why you care such that you know whether or not it is worth worrying about.
    My usual boring signature: Nothing

  13. #13

    Thread Starter
    Fanatic Member
    Join Date
    Mar 2010
    Posts
    762

    Re: How much memory is taken up by an array?

    Quote Originally Posted by Arnoutdv View Post
    Is there specific need to know exactly how much memory is in use by the arrays in your application?
    What is your goal?
    I am developing an application, and there are a number of string arrays that get allocated and populated at run time (using the ReDim statement).
    I am wondering if this can run out of memory, because my arrays can potentially be very large (around 100,000 records each), and each element of the array can sometimes be small and sometimes large (probably won't exceed 128 characters).
    That is why, I decided to find out how much memory is occupied by string arrays.

    It is unlikely that based on the above numbers, the system will run out of memory, but I think it is always reasonable to be cautious and at least ask some more experienced programmers to see if they have any real concerns.
    I certainly believe that always asking something (when in doubt) is more reasonable than not asking it.

    Thanks.
    Ilia

  14. #14
    PowerPoster Arnoutdv's Avatar
    Join Date
    Oct 2013
    Posts
    5,872

    Re: How much memory is taken up by an array?

    Valid point.
    Then using the given information you can quite simple do a sum on the size of the strings in the string array adding the the overhead to have quite a reasonable estimation. If you combine this with actual memory used by your process then you have something to work with.
    Out of memory can also occur when the process uses to much memory or when the free memory is to fragmented for huge arrays.

  15. #15

    Thread Starter
    Fanatic Member
    Join Date
    Mar 2010
    Posts
    762

    Re: How much memory is taken up by an array?

    Thanks for your help.

    Quote Originally Posted by Arnoutdv View Post
    If you combine this with actual memory used by your process then you have something to work with.
    How do I find out the actual amount of memory used by my process?

    Out of memory can also occur when the process uses to much memory or when the free memory is to fragmented for huge arrays.
    Is there a way to know that memory fragmentation has happened to a large degree?
    For example is there any method of calculating memory fragmentation?
    And then comparing it to a maximum acceptable value?

    And is there a way to de-fragment the memory (short of rebooting the computer)?

    Please advise.
    Thanks.

  16. #16
    PowerPoster wqweto's Avatar
    Join Date
    May 2011
    Location
    Sofia, Bulgaria
    Posts
    5,120

    Re: How much memory is taken up by an array?

    Quote Originally Posted by IliaPreston View Post
    And is there a way to de-fragment the memory (short of rebooting the computer)?
    The heap fragmentation discussed here is per process so restarting the process usually "levels the field".

    On startup each process is granted a 4GB flat virtual address space which has some parts (pages) mapped to actual physical memory but most are unmapped (unallocated). OS provides VirtualAlloc function to grab chunks from the green field (and map these to physical memory) and it is inside these chunks that "software" memory heaps are implemented entirely in code. There are a lot of "software" memory allocators provided by the OS and by COM and also in C/C++ they have custom application implemented allocators they use. In VB6 land we use COM provided allocators only (e.g. CoTaskMemAlloc and SysAllocString) for almost everything. Every ReDim MyArray and every MyString = String(1000, 0) is a call to one of these allocators behind the scenes.

    cheers,
    </wqw>

  17. #17

    Thread Starter
    Fanatic Member
    Join Date
    Mar 2010
    Posts
    762

    Re: How much memory is taken up by an array?

    Quote Originally Posted by wqweto View Post
    The heap fragmentation discussed here is per process so restarting the process usually "levels the field".
    ......</wqw>
    Thanks.
    So, what I understand from your statement above is that you mean restarting the computer is not needed. Rather, restarting just the application should be enough.
    Is that what you mean?

    Also, in your previous post you said:
    If you combine this with actual memory used by your process then you have something to work with
    How do I find out the actual amount of memory used by my process?

    Thanks for your help and advice.

  18. #18
    PowerPoster wqweto's Avatar
    Join Date
    May 2011
    Location
    Sofia, Bulgaria
    Posts
    5,120

    Re: How much memory is taken up by an array?

    There is no strict definition of "memory used by a process" but the closes is so called WorkingSet size (or private bytes in Task Manager) which can be retrieved like this

    Code:
    Option Explicit
    
    Private Declare Sub CopyMemory Lib "kernel32" Alias "RtlMoveMemory" (Destination As Any, Source As Any, ByVal Length As Long)
    Private Declare Function GetCurrentProcess Lib "kernel32" () As Long
    Private Declare Function GetProcessMemoryInfo Lib "psapi" (ByVal hProcess As Long, ppsmemCounters As PROCESS_MEMORY_COUNTERS, ByVal cb As Long) As Long
    
    Private Type PROCESS_MEMORY_COUNTERS
        cb                      As Long
        PageFaultCount          As Long
        PeakWorkingSetSize      As Long
        WorkingSetSize          As Long
        QuotaPeakPagedPoolUsage As Long
        QuotaPagedPoolUsage     As Long
        QuotaPeakNonPagedPoolUsage As Long
        QuotaNonPagedPoolUsage  As Long
        PagefileUsage           As Long
        PeakPagefileUsage       As Long
        PrivateUsage            As Long
    End Type
    
    Private Function pvGetWorkingSetSizeInMB() As Currency
        Dim uCounters       As PROCESS_MEMORY_COUNTERS
    
        If GetProcessMemoryInfo(GetCurrentProcess(), uCounters, Len(uCounters)) <> 0 Then
            Call CopyMemory(pvGetWorkingSetSizeInMB, uCounters.WorkingSetSize, 4)
            pvGetWorkingSetSizeInMB = pvGetWorkingSetSizeInMB * 10000@ / 1024@ / 1024@
        End If
    End Function
    
    Private Sub Form_Load()
        Debug.Print pvGetWorkingSetSizeInMB
    End Sub
    cheers,
    </wqw>

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width