[RESOLVED] Split array into multiple arrays? :)

[Jim Davis]

Hey!

Im looking for some ideas how can i split a huge array (for example Array(0 to 25000)) into 2 or more arrays without cycling trhu the elements of it one by one. Im thinking on some apis like RtlMoveMemory but how can i set the begin and last element to split it partially?

eg:
Array1(0 to 10000) then Array2(10001 to 25000) from Array(0 to 25000)

[LaVolpe]

You are right on. RtlMoveMemory (aka CopyMemory) is the prime candidate.
But first, one would have to ask why? You are not necessarily gaining anything. Is there another reason? Maybe that question should also be asked by you.

WARNING: Using CopyMemory, it is very easy to crash your project and possibly the entire system. It must be used correctly and the amount of bytes being copied cannot be overestimated/overcalculated.

First you need to determine which variation of that APIs declaration you want to use. Here is some sample code and the declaration included. You didn't mention what datatype the array is (Long, Integer, etc). It matters big time. Assuming they are long.... 250001 items, huh?

ReDim Array1(0 to 10000) As Long
Redim Array2(10001 to 25000) As Long
CopyMemory Array1(0), Array(0), 10001 * 4 ' the 4 is = 4 bytes = 1 Long
CopyMemory Array2(10001), Array(10001), 15000 * 4

** Strings are done differently. Arrays of UDTs may not be done, depending on the UDT. Integers, Dates, Doubles, Singles, Bytes, etc are multiplied by how many bytes their datatype uses.

One version of this API's declaration, probably the most common.
Private Declare Sub CopyMemory Lib "kernel32.dll" Alias "RtlMoveMemory" _
(ByRef Destination As Any, ByRef Source As Any, ByVal Length As Long)

[Jim Davis]

Thanks! It looks great!

But if im want to copy the elements of a string array, i must copy its elements one by one into the another array? Or whats the difference?

[LaVolpe]

Well, strings in a dynamic array (static arrays are different) are really pointers to another memory location. Pointers always have to be handled differently whether you are copying or moving (i.e., move = copy & then delete original immediately). Why? If the same pointer exists in two array items, when one array gets erased, VB will erase the memory location that those pointers pointed to. Then when VB erases the 2nd array, it tries to erase those same memory locations, but they were already erased and may be in use by other things -- crash for sure. To make a long story short, copying/splitting arrays of Strings, objects, and some other exceptions requires a real copy: same values, two different pointers. Moving is far easier, because you can copy the pointers, then zero out the other pointers so VB thinks they are already released. Copying pointers = difficult, copying values = easy.

To Move, requires another API: ZeroMemory
-- The array items will be 4 bytes each, a Long, which a pointer is
-- CopyMemory the array items similar to below
-- ZeroMemory the old array items immediately; fool VB in thinking pointers are released

To Copy, you already know that: Array1()=Array(). But that isn't what you want, so we go an extra step. Example is using Array() As Strings. ZeroMemory used too for this split example.

ReDim tempArray(0 to 25000) As String
ReDim Array1(0 to 10000) As String ' all pointers are zero, vbNullStrings
Redim Array2(10001 to 25000) As String ' all pointers are zero, vbNullStrings
tempArray()=Array() ' copy the entire array; each copied string has own pointer
CopyMemory ByVal VarPtr(Array1(0)), ByVal VarPtr(tempArray(0)), 10001 * 4
CopyMemory ByVal VarPtr(Array2(10001)), ByVal VarPtr(tempArray(10001)), 15000 * 4
ZeroMemory ByVal VarPtr(tempArray(0)), 250001 * 4 ' clear pointers in old tempArray
Erase tempArray ' no longer needed

Edited: Oops. Forgot to mention using ByVal VarPtr(). We must provide copymemory with memory location of where the pointer is. There is another spin on this technique, but regardless, both are a little touchy & must be exact. My unedited portion was from scratch. But I tested what I provided and immediately crashed so I came back to ensure you know that when copying/moving pointers, you must ref pointers in CopyMemory.

[Jim Davis]

Ok! It really works!

Ive did some benchs with 500,000 * 5 string elements, and find out that the technique above is took around 180% more time to get the same result, compared to copying the elements one by one. And ive got a bonus crash at the end of the module, at some "read" state.

So it looks like its better to use the good old copy one by one method.

anyway.. thanks for the detailed help and answers!

Sub SplitArray_1b1(ByRef SourceArray() As String, ByRef Dest1() As String, ByRef Dest2() As String, ByVal CountMax As Long)
Dim C As Long, lB As Long, uB As Long, Temp As String
lB = LBound(SourceArray)
uB = UBound(SourceArray)
ReDim Dest1(0 To CountMax)
ReDim Dest2(0 To uB - CountMax)

C = 0

For C = 0 To CountMax - 1
  Dest1(C) = SourceArray(C)
Next

For C = CountMax To uB
  Dest2(C - CountMax) = SourceArray(C)
Next C

End Sub

[Lord Orwell]

you also have to add one to the length of string arrays because each string is terminated by a null. VB takes this into account but API doesn't.

[Jim Davis]

you also have to add one to the length of string arrays because each string is terminated by a null. VB takes this into account but API doesn't.

Umm.. but this method copies the pointers of strings not the contained strings. so im not sure where i should increase any values?!

[Lord Orwell]

i was referring to static arrays.

[LaVolpe]

Well the extra time is my fault - apologize. Delete line: Redim tempArray(). There is no reason to do this because when tempArray()=Array(), the array is then redimensioned again. Sorry. The crash is your fault. You must calculate the number of bytes to transfer correctly.
Running the two methods separately, here is what I returned:
2 tests In IDE: 1st entry is copymemory version, 2nd is simple loop
0.59412500000326 0.750125000005937
0.64100000000326 0.71912500000326
Notice that there is not a huge difference here because we had to copy the strings. You will see much greater difference if they were to be moved as shown below. Similar test, but not needing to copy the array:
1.62500000005821E-02 0.719624999997905
1.58749999973224E-02 0.71875

CopyMemory code is below.

Dim tempArray() As String
Dim Array1() As String
Dim Array2() As String
Dim ArrayS() As String
Dim X As Long
Dim T As Double

ReDim ArrayS(0 To 500000)
For X = 0 To 500000
    ArrayS(X) = "Item " & X
Next

T = Timer

X = 250053
ReDim Array1(0 To X) As String ' all pointers are zero, vbNullStrings
ReDim Array2(X + 1 To UBound(ArrayS)) As String  ' all pointers are zero, vbNullStrings
tempArray() = ArrayS() ' copy the entire array; each copied string has own pointer
CopyMemory ByVal VarPtr(Array1(0)), ByVal VarPtr(tempArray(0)), (X + 1) * 4 ' the 4 is = 4 bytes = 1 Long
CopyMemory ByVal VarPtr(Array2(X + 1)), ByVal VarPtr(tempArray(X + 1)), (UBound(ArrayS) - X + 1) * 4
ZeroMemory ByVal VarPtr(tempArray(0)), (UBound(ArrayS) + 1) * 4 ' clear pointers in old tempArray
Erase tempArray ' no longer needed

Debug.Print Timer - T

Edited: Must admit there is not a big difference in speed when copying strings. Compiled shouldnt' show any signifcance either. Not only that, memory requirements are far greater using CopyMemory. Why? Well we are using 500k array long * 3 between 4 arrays and 500k strings * 2. In the simple loop, you are using 500k longs * 2 between 3 arrays and 500K strings * 2. Is the excess, temporary memory usage worth the extra ms trimmed? I think not. But when copying values (Longs, Bytes, Integers, etc). There is no question that CopyMemory is significantly faster than loops when large numbers of array items are involved. Thanx for the opportunity to see things from a different angle.

[Jim Davis]

In my case, i think i'll get more than a few msec. Im using SQLiteVB that handles a local sql database file. After a query it passes the whole results (with the names of each column) within one single dimension array. So i have to put the values into the right bag to handle it. The result of this handling causes a bit more computing. But this way i can reduce the time.. i hope so

Memory usage? Its temporary, so it will free up the memory after the spliting. But i can split this robust array into multiple arrays (one array for each column) by a few lines. And in less time.

edit:
by using this method ive did a subroutine that helps to pick up any part of an array.

Private Declare Sub CopyMemory Lib "kernel32.dll" Alias "RtlMoveMemory" (ByRef Destination As Any, ByRef Source As Any, ByVal Length As Long)
Private Declare Sub ZeroMemory Lib "kernel32" Alias "RtlZeroMemory" (dst As Any, ByVal iLen&)


Sub ArrayPart(ByRef SourceArray() As String, ByRef OutputArray() As String, ByVal First As Long, Optional ByVal Last As Long = -1, Optional ByVal Lenght As Long = -1)
  Dim uB As Long, lB As Long
  uB = UBound(SourceArray): lB = LBound(SourceArray)
  
  Dim TempArray() As String
  
  If First > uB Then First = uB - 1
  If First < lB Then First = lB
  
  If Lenght = 0 Then Exit Sub
  If Lenght > 0 Then Last = First + Lenght
  
  If Last = -1 Then
    Last = uB
  ElseIf Last < First Then
    Last = First + 1
  End If
  
  ReDim OutputArray(Last - First)
  
  TempArray = SourceArray
  
  CopyMemory ByVal VarPtr(OutputArray(0)), ByVal VarPtr(TempArray(First)), 4 * ((Last - First) + 1)
  ZeroMemory ByVal VarPtr(TempArray(0)), ((uB - lB) + 1) * 4
  
  Erase TempArray
End Sub

'Usage:
'ArrayPart MyArray(),Output(),FirstElement,[Lastelement],[NumberOfElements]

'NumberofElements overrides the lastelement's value if both are set.

Actually i have to do a bit more complex calculations to get to use only one array duplication while splitting up the whole array... But thats not a problem

And again, thanks for to giving me the right directions!
JD

[LaVolpe]

In that case, you might consider moving the data from your passed array directly to the split arrays. Here's the code I used for that second test. There is no tempArray() in it.

X = 250053
ReDim Array1(0 To X) As String ' all pointers are zero, vbNullStrings
ReDim Array2(X + 1 To UBound(ArrayS)) As String  ' all pointers are zero, vbNullStrings
CopyMemory ByVal VarPtr(Array1(0)), ByVal VarPtr(ArrayS(0)), (X + 1) * 4 ' the 4 is = 4 bytes = 1 Long
CopyMemory ByVal VarPtr(Array2(X + 1)), ByVal VarPtr(ArrayS(X + 1)), (UBound(ArrayS) - X + 1) * 4
ZeroMemory ByVal VarPtr(ArrayS(0)), (UBound(ArrayS) + 1) * 4 ' clear pointers in source Array

[Jim Davis]

LOL thanks a lot. Ill give it a try!

But i'm lost at this point. By watching the previous posts i dont get it why have you used the temporary array before?

Ahh nevermind.. i got it.. so its just needed to reserve the source array. but in my case i can drop into the trash after splitting, because the column arrays already holds the elements. so this way i can get much more extra msecs.. cool.. ",

[LaVolpe]

The tempArray example. That was to create a copy of the source array, where one would want two copies of that 500K string array, the source left as is and a new split array. String pointers cannot be duplicated without extreme caution because when memory that pointer is pointing to is deleted, VB better not be allowed to try to delete it again.

The 2nd one is much less headache and much faster because we are just moving pointers from one array to the other, using zeromemory to clean up.

[Jim Davis]

Yup I cant wait to test it..

[Lord Orwell]

most likely to preserve the original array. if you reference a variable in a sub that way and you don't want to mess the original up, you have to copy it before changing it, or any manipulation you do to it is permanent