What is the fastest way to append to a string?

[VaxoP]

string = string & new_data

takes an enormously long time to execute..
it works very well for small strings and small loops but if you have to do a loop 180,000 times appending to a string, it takes FOREVER using the above code.

is there a faster method i dont know about?

[Ellis Dee]

Redimension a byte array with as many elements as there will be characters. For each new character to append, write its Asc() code to the next unused array element. When you are finished "appending" characters, convert the byte array to a string:

strResult = StrConv(bytArray, vbUnicode)

[VaxoP]

Redimension a byte array with as many elements as there will be characters. For each new character to append, write its Asc() code to the next unused array element. When you are finished "appending" characters, convert the byte array to a string:

strResult = StrConv(bytArray, vbUnicode)

hmm
I dont know how many characters there will be though..

[Ellis Dee]

Then redimennsion blocks at a time. If you're expecting 180k, I'd start it off at 200k and make subsequent blocks 10k each.

Don't forget to size it back down to the exact length before converting it to a string.

[DigiRev]

string = string & new_data

takes an enormously long time to execute..
it works very well for small strings and small loops but if you have to do a loop 180,000 times appending to a string, it takes FOREVER using the above code.

is there a faster method i dont know about?

This class was made just for this.

http://www.vbforums.com/attachment.p...6&d=1175915331

Dim myString As New clsConcat
 
myString.SConcat "this "
myString.SConcat "is "
myString.SConcat "a "
myString.SConcat "test."
 
MsgBox myString.GetString
Set myString = Nothing

It's a huge improvement over normal string concatenation and speeds large string operations up a lot.

If speed is really an issue I usually try to rewrite the code to work with byte arrays.

[VaxoP]

This class was made just for this.

http://www.vbforums.com/attachment.p...6&d=1175915331

Dim myString As New clsConcat
 
myString.SConcat "this "
myString.SConcat "is "
myString.SConcat "a "
myString.SConcat "test."
 
MsgBox myString.GetString
Set myString = Nothing

It's a huge improvement over normal string concatenation and speeds large string operations up a lot.

If speed is really an issue I usually try to rewrite the code to work with byte arrays.

Neat that runs about 16% faster (up to 60% faster for really big strings)
How can you do it with byte arrays? Do you have any sample code i could peek at (ive never used em before)

[Ellis Dee]

I'd be happy to set up a working example for you, but first I need to know where the input text is coming from. Can you set up a shell that skips the concatenation part?

The class isn't much faster because each call forces a redimension. A byte array approach will be like greased lightning in comparison.

[VaxoP]

Well right now the program just goes through an array of 100,000 items, does a simple check on them (if instr(array,something)) and appends the value to a string.

for x = 0 to ubound(array) - 1
if instr(array(x),"something") then myString = myString & array(x) & vbcrlf
next


like that

[techgnome]

Ick! .... it could be the array itself and not the concatenation that's the issue....

try this see if there's any improvement:

Dim MyText As String
for x = 0 to ubound(array) - 1
  MyText = array(x)
if instr(MyText,"something") then myString = myString & MyText & vbcrlf
next

It could also be the Instr that's innefficient.

-tg

[VaxoP]

Ick! .... it could be the array itself and not the concatenation that's the issue....

try this see if there's any improvement:

Dim MyText As String
for x = 0 to ubound(array) - 1
  MyText = array(x)
if instr(MyText,"something") then myString = myString & MyText & vbcrlf
next

It could also be the Instr that's innefficient.

-tg

Thats not it..
I tried removing my concat line altogether and doing something else with the data (adding to listbox for example) and speed improved significantly.

It is also that line that I changed when I was using the class module provided here.. which showed a huge performance increase when switching.

The bottleneck without question is myString = myString & array(x) & vbcrlf

[Logophobic]

This is similar to Ellis Dee's suggestion, but with much less hassle.
Be sure to replace the array variable in this code with the actual array.

Dim strMyString As String ' The string
Dim strArray() As String  ' Stand-in for the array
Dim strBuffer As String   ' The buffer
Dim lngBuffSize As Long   ' Amount of space to buffer
Dim lngPos As Long        ' Position within the buffer
Dim lngLen As Long        ' Lenght of string to copy to buffer
Dim i As Long

' Try to pick a length that is 20% to 60% of the length of the final string.
' Too long wastes memory. Too short slows it down with extra extensions.
lngBuffSize = 500

' Set the initial size of the buffer
lngLen = Len(strMyString)
strBuffer = Space(lngLen + lngBuffSize)

' Copy strMyString to the buffer
Mid(strBuffer, 1, lngLen) = strMyString
lngPos = Len(strMyString) + 1

For i = 0 To UBound(strArray) - 1
  If InStr(strArray(i), "something") > 0 Then
    lngLen = Len(strArray(i))
    
    ' Extend the length of the buffer for more space, if needed.
    If lngPos + lngLen + 2 > Len(strBuffer) Then ' + 2 for vbCrLf
      strBuffer = strBuffer & Space(lngBuffSize + lngLen + 2)
    End If
    
    ' Copy the array element to the buffer.
    Mid(strBuffer, lngPos, lngLen) = strArray(i)
    lngPos = lngPos + lngLen
    
    ' Copy vbCrLf to the buffer.
    Mid(strBuffer, lngPos, 2) = vbCrLf
    lngPos = lngPos + 2
  End If
Next i

' Copy the buffer to strMyString
strMyString = Left$(strBuffer, lngPos - 1)

' Clear the buffer
strBuffer = ""

Edit: After looking at the class that DigiRev posted, I see that this code is doing the exact same thing as the class. This inline code will run faster, though, due to the overhead of repeatedly calling a function and the general nature of classes.

Regarding the class, there is a bug. The SetString method needs to update lOffset. Add the line lOffset = Len(Source).

[Ellis Dee]

Well right now the program just goes through an array of 100,000 items, does a simple check on them (if instr(array,something)) and appends the value to a string.

This changes the ballgame entirely. (And you're right about the concatenation being the bottleneck.) The magic bullet for speeding character concatenation is to use byte arrays. But you aren't concatenating characters, you're concatenating strings. The magic bullet for that is the Join() function.

Here's what you want to do:

• Start by creating a second (empty) string array the same size as the original
• Maintain a "current position" counter variable identifying the next available slot in the new array
• Iterate the original array; each time you find a hit, copy it to the next available slot in the new array, updating your counter
• When you are all done, set your destination string equal to Join(NewArray, "")

It'll be as fast as it gets.

You can ignore the extraneous array elements that don't end up getting used for simplicity. If you're as anal as I am, you'll probably want to ReDim Preserve the second array to make it only hold the number of elements actually used. (I suspect this might actually slow it down, though.)

EDIT: Logo's method will be very fast, but I believe this will be significantly faster than that.

[Ellis Dee]

Also, you may be able to do it as a one-liner using the Filter() function.

EDIT: I'm setting up a benchmark to test how fast each method is. I'll post the code and results in a bit.

[Ellis Dee]

Logo's buffer method is extremely fast, but it has a bug somewhere in it where it's copying too much text. Not sure what the deal is. (EDIT: I figured it out and corrected; see next post.) Other than that, the main problem is that you have to specify a correct buffer size to get the best results, which is problematic for a generic implementation.

I took his idea and ran with it, but improved it to not require a buffer size at all. Instead, I run through the original array and add up the sizes of all the matches, create the output string once with the correct size, then run through the array again pasting each match into the output string. I use a secondary array of booleans to track which array elements were matches so I don't have to repeat the criteria.

Both these methods are faster than anything else, but not by much. My Join() method is pretty peppy and quite flexible, but it's not as good as either buffer method in any way. The native Filter() command is by far the easiest to implement if your criteria boils down to a simple InStr() call. Try this:

OutputString = Join(Filter(StringArray, SearchString), "")

One line of code and quite fast, so this is probably the way you want to go. For completeness, here's the benchmark program and the results. You should be able to figure out how to use any of the methods you need should your criteria expand beyond a simple InStr() call.

To run the benchmark, create a project, add a multi-line textbox and a command button to the form, and paste this code (reply with quote and copy it from the reply window) into the form's module:

Option Explicit
 
Private Const ArraySize = 99999
Private Const Delimiter = vbNewLine
 
Public Function FilterArray(pstrArray() As String, pstrFilter As String, pstrDelimiter As String)
    Dim strDest() As String
    Dim lngPos As Long
    Dim i As Long
    Dim iMin As Long
    Dim iMax As Long
    
    iMin = LBound(pstrArray)
    iMax = UBound(pstrArray)
    ReDim strDest(iMin To iMax)
    lngPos = iMin
    For i = iMin To iMax
        If InStr(pstrArray(i), pstrFilter) <> 0 Then
            strDest(lngPos) = pstrArray(i)
            lngPos = lngPos + 1
        End If
    Next
    ReDim Preserve strDest(iMin To lngPos - 1)
    FilterArray = Join(strDest, pstrDelimiter)
    Erase strDest
End Function
 
Public Function EllisBufferMethod(pstrArray() As String, pstrFilter As String, pstrDelimiter As String)
    Dim strBuffer As String
    Dim lngLen As Long
    Dim lngPos As Long
    Dim blnMatch() As Boolean
    Dim i As Long
    Dim iMin As Long
    Dim iMax As Long
    
    iMin = LBound(pstrArray)
    iMax = UBound(pstrArray)
    ReDim blnMatch(iMin To iMax)
    For i = iMin To iMax
        If InStr(pstrArray(i), pstrFilter) Then
            lngLen = lngLen + Len(pstrArray(i)) + Len(pstrDelimiter)
            blnMatch(i) = True
        End If
    Next
    strBuffer = Space(lngLen + 1)
    lngPos = 1
    For i = iMin To iMax
        If blnMatch(i) Then
            Mid$(strBuffer, lngPos, Len(pstrArray(i))) = pstrArray(i)
            lngPos = lngPos + Len(pstrArray(i))
            Mid$(strBuffer, lngPos, Len(pstrDelimiter)) = pstrDelimiter
            lngPos = lngPos + Len(pstrDelimiter)
        End If
    Next
    EllisBufferMethod = Left$(strBuffer, lngLen - Len(pstrDelimiter))
    strBuffer = ""
    Erase blnMatch
End Function
 
Public Function LogoBufferMethod(pstrArray() As String, pstrFilter As String, pstrDelimiter As String)
    Dim strBuffer As String   ' The buffer
    Dim lngBuffSize As Long   ' Amount of space to buffer
    Dim lngPos As Long        ' Position within the buffer
    Dim lngLen As Long        ' Lenght of string to copy to buffer
    Dim i As Long
    Dim lngDelimiterLen As Long
    
    ' Try to pick a length that is 20% to 60% of the length of the final string.
    ' Too long wastes memory. Too short slows it down with extra extensions.
    lngBuffSize = 80000 ' ~40%
    
    ' Set the initial size of the buffer
    strBuffer = Space(lngBuffSize)
    
    lngPos = 1
    lngDelimiterLen = Len(pstrDelimiter)
    
    For i = 0 To UBound(pstrArray) - 1
        If InStr(pstrArray(i), pstrFilter) > 0 Then
            lngLen = Len(pstrArray(i))
            
            ' Extend the length of the buffer for more space, if needed.
            If lngPos + lngLen + lngDelimiterLen > Len(strBuffer) Then
                strBuffer = strBuffer & Space(lngBuffSize + lngLen + lngDelimiterLen)
            End If
            
            ' Copy the array element to the buffer.
            Mid(strBuffer, lngPos, lngLen) = pstrArray(i)
            lngPos = lngPos + lngLen
            ' Copy delimiter to the buffer.
            Mid(strBuffer, lngPos, lngDelimiterLen) = pstrDelimiter
            lngPos = lngPos + lngDelimiterLen
        End If
    Next i
    
    ' Copy the buffer to strMyString
    LogoBufferMethod = Left$(strBuffer, lngPos - lngDelimiterLen - 1)
    
    ' Clear the buffer
    strBuffer = ""
End Function
 
Private Sub Command1_Click()
    Me.MousePointer = vbHourglass
    Me.Text1.Text = Sample()
    Me.MousePointer = vbNormal
End Sub
 
Public Function Sample() As String
    Dim strArray(ArraySize) As String
    Dim i As Long
    Dim strOutput As String
    Dim sngStart As Single
    Dim strMessage As String
    
    For i = 0 To ArraySize
        strArray(i) = i
    Next
    ' String concatenation
    sngStart = Timer
    For i = 0 To ArraySize
        If InStr(strArray(i), "1") <> 0 Then strOutput = strOutput & strArray(i) & Delimiter
    Next
    strMessage = strMessage & "Concatenation: " & Format(Timer - sngStart, "0.000") & " seconds (" & Len(strOutput) & " characters)" & vbCrLf
    strOutput = ""
    ' Logo's string buffer method
    sngStart = Timer
    strOutput = LogoBufferMethod(strArray, "1", Delimiter)
    strMessage = strMessage & "Logo's buffer method: " & Format(Timer - sngStart, "0.000") & " seconds (" & Len(strOutput) & " characters)" & vbCrLf
    strOutput = ""
    ' Ellis's string buffer method
    sngStart = Timer
    strOutput = EllisBufferMethod(strArray, "1", Delimiter)
    strMessage = strMessage & "Ellis's buffer method: " & Format(Timer - sngStart, "0.000") & " seconds (" & Len(strOutput) & " characters)" & vbCrLf
    strOutput = ""
    ' Join
    sngStart = Timer
    strOutput = FilterArray(strArray, "1", Delimiter)
    strMessage = strMessage & "Join(): " & Format(Timer - sngStart, "0.000") & " seconds (" & Len(strOutput) & " characters)" & vbCrLf
    strOutput = ""
    ' FIlter
    sngStart = Timer
    strOutput = Join(Filter(strArray, "1"), Delimiter)
    strMessage = strMessage & "Filter(): " & Format(Timer - sngStart, "0.000") & " seconds (" & Len(strOutput) & " characters)" & vbCrLf
    Sample = strMessage
End Function

You should run it from a compiled exe to get realistic results. On my old, slow machine, the results are:

Concatenation: 253.289 seconds (282927 characters)
Logo's buffer method: 0.070 seconds (282925 characters)
Ellis's buffer method: 0.086 seconds (282925 characters)
Join(): 0.203 seconds (282925 characters)
Filter(): 0.227 seconds (282925 characters)

[Ellis Dee]

I figured out the "bug" in Logo's code, and it isn't a bug at all. (I'm a fool for doubting him.) He's adding a delimiter between each string, which makes a ton of sense.

I'll update the code above to handle this, as well as the results. Shouldn't take long.

EDIT: I finished the changes. You can specify an empty string as the delimiter if you don't want to use one. While Logo's basic method is the fastest, it requires prior knowledge of the results.

[leinad31]

Why isn't the data stored in a database table?

[Ellis Dee]

On an unrelated note, I'd like to point out that if there was ever compelling evidence that you never need to store the results of Len() to a variable for speed reasons, my buffer method code is it.

[Logophobic]

My PC is a bit faster, so I used a high-resolution timer.
The percentages are buffer size relative to total output length.
The times are an average over 10 calls.

Output string length: 282,925 characters
Logo:
 100%   11.45 ms
  53%   11.83 ms
  35%   11.74 ms
  27%   12.03 ms
  21%   12.23 ms
  17%   13.02 ms
  11%   27.64 ms

Ellis:  12.57 ms

I also increased the array size by a factor of 10.

Output string length: 3,703,789 characters
Logo:
 100%  132.62 ms
  54%  143.39 ms
  34%  155.08 ms
  27%  157.05 ms
  20%  160.27 ms
  17%  170.40 ms
  11%  364.39 ms

Ellis: 144.46 ms

As you can see, the difference between Ellis Dee's pre-pass method and a good estimate of the space needed is negligible. However, a low estimate causes a significant loss of efficiency, so your best bet would be to use his code.

[VaxoP]

neat thanks
so is the only drawback to using a large space the memory usage?

[Ellis Dee]

neat thanks
so is the only drawback to using a large space the memory usage?

Pretty much, yeah. Most of that extra space can be removed easily enough by using the return value directly instead of using a local string. That means that there is no extra space being used for a temporary string, but it still requires the overhead of the boolean array. I've renamed it to the more generic FilterArrayToString():

Public Function FilterArrayToString(pstrArray() As String, pstrFilter As String, pstrDelimiter As String) As String
    Dim lngLen As Long
    Dim lngPos As Long
    Dim blnMatch() As Boolean
    Dim i As Long
    Dim iMin As Long
    Dim iMax As Long
    
    iMin = LBound(pstrArray)
    iMax = UBound(pstrArray)
    ReDim blnMatch(iMin To iMax)
    For i = iMin To iMax
        If InStr(pstrArray(i), pstrFilter) <> 0 Then
            lngLen = lngLen + Len(pstrArray(i)) + Len(pstrDelimiter)
            blnMatch(i) = True
        End If
    Next
    FilterArrayToString = Space(lngLen + 1)
    lngPos = 1
    For i = iMin To iMax
        If blnMatch(i) Then
            Mid$(FilterArrayToString, lngPos, Len(pstrArray(i))) = pstrArray(i)
            lngPos = lngPos + Len(pstrArray(i))
            Mid$(FilterArrayToString, lngPos, Len(pstrDelimiter)) = pstrDelimiter
            lngPos = lngPos + Len(pstrDelimiter)
        End If
    Next
    FilterArrayToString = Left$(FilterArrayToString, lngLen - Len(pstrDelimiter))
    Erase blnMatch
End Function

This approach is nice because it is very customizable, but I'm still a fan of the one-liner approach.