[RESOLVED] how to get faster way from Array

[esugiawan]

I create program to get data from socket server.
General format for each transmitted record ( Header| Date |Time | Sequence #|RecordType|DATA |Checksum |CR LF
Each item/field/column in the record is separated by pipe / ”|” /
format Header always "IDX"
-----
First, i create procedure with socket to receive all data and save it to array ( let said df(array) )
Second , i try to process df(array) using BackgroundWorker and save the valid data to sql server
after all, the program running success full , but if i running from 9 AM to 16 PM, the difference time between first procedure and second procedure until 5 minute

My question is, how to create faster array to solve that program ? thx

------
Sample Data that i got from socket server

        df(0) = "142300|001023095120|001022461737|??
IDX|20180123|145628|01605548|5|PGAS                 |RG  |00000002470.00|00000002670.00|0000"
        df(1) = "0002490.00|00000002660.00|+00190.00|000277433200|0000724698055000|0017562|000000886.6667|00024241508196|00000002500.00|0000000265"
        df(2) = "0.00|000002386300|00000002660.00|000004706600|00000002612.00|0|?7
IDX|20180123|145628|01605549|5|ICBP                 |RG  |0000"

-------
first procedure

   
 Dim maxArr As Integer = 2000

 Private Sub AppendOutput(message As String)

        If message.Substring(0, 7) = "Welcome" Then
            Dim buffer() As Byte = System.Text.Encoding.ASCII.GetBytes("password" & vbCrLf & vbCrLf)
            _Connection.Stream.Write(buffer, 0, buffer.Length)
        End If
       df(dfCounter) = message

       If dfCounter >= maxArr - 1 Then  dfCounter = 0   Else  dfCounter +=  1
    End Sub

Second procedure

  Private Sub BackgroundWorker1_DoWork(sender As Object, e As DoWorkEventArgs) Handles BackgroundWorker1.DoWork
        Dim i As Integer = 0
        Dim oldCounter As Integer = 0
        Dim JmlArr As Integer = 0
        Dim TempString As String = ""
        Dim z As Integer = 0
ulang:

        If statusOut = True Then
            TempString = ""
            JmlArr = 0 : oldCounter = 0 : z = 0
            Exit Sub
        End If
        If TempDfCounter = dfCounter Then GoTo ulang Else TempDfCounter = dfCounter
        If oldCounter > TempDfCounter Then JmlArr = maxArr Else JmlArr = TempDfCounter


        For i = oldCounter To JmlArr
            TempString += df(i)
        Next

        pstr = oldCounter & " to " & JmlArr
        OutputMsg(pstr)

        Dim HasilSplit() As String = Split(TempString, "IDX|")
        For z = 0 To HasilSplit.Length - 1
            TempString = InsertToOrder("IDX|" & HasilSplit(z))
            OutputMsg("z:" & z & " to " & HasilSplit.Length - 1 & " hasil & " & HasilSplit(z))
        Next

        If oldCounter >= maxArr Then oldCounter = 0 Else oldCounter = JmlArr + 1
        GoTo ulang
    End Sub

Function InsertToOrder(msg As String) As String
        Try
            Dim MsgSplit() As String
            MsgSplit = Split(msg, "|")
            If MsgSplit.Length < 10 Then Exit Function
            Select Case MsgSplit(4) ' kode datafeed
                Case Is = "1" 'order
                    sqlstr = "dbo.spUpdate1"
                    Using cmd As SqlClient.SqlCommand = New SqlClient.SqlCommand(sqlstr, con)
                        cmd.CommandType = CommandType.StoredProcedure
                        cmd.Parameters.Add("@SecCode", SqlDbType.NVarChar).Value = MsgSplit(7).Trim
                        cmd.Parameters.Add("@board", SqlDbType.NVarChar).Value = MsgSplit(8).Trim
                        cmd.Parameters.Add("@bid", SqlDbType.Int).Value = Convert.ToDouble(MsgSplit(15).Trim)
                        cmd.Parameters.Add("@qtybid", SqlDbType.Money).Value = Convert.ToDouble(MsgSplit(16).Trim)
                        cmd.Parameters.Add("@offer", SqlDbType.Int).Value = Convert.ToDouble(MsgSplit(17).Trim)
                        cmd.Parameters.Add("@qtyOffer", SqlDbType.Money).Value = Convert.ToDouble(MsgSplit(18).Trim)
                        cmd.Connection.Open()
                        cmd.ExecuteNonQuery()
                        cmd.Connection.Close()
                    End Using

                Case Is = "2" ' Trade
                    sqlstr = "dbo.spUpdate2"
                    Using cmd As SqlClient.SqlCommand = New SqlClient.SqlCommand(sqlstr, con)
                        cmd.CommandType = CommandType.StoredProcedure
                        cmd.Parameters.Add("@SecCode", SqlDbType.NVarChar).Value = MsgSplit(7).Trim
                        cmd.Parameters.Add("@board", SqlDbType.NVarChar).Value = MsgSplit(8).Trim
                        cmd.Parameters.Add("@LastPrice", SqlDbType.Int).Value = Convert.ToDouble(MsgSplit(10))
                        cmd.Parameters.Add("@bid", SqlDbType.Int).Value = Convert.ToDouble(MsgSplit(16))
                        cmd.Parameters.Add("@qtybid", SqlDbType.Money).Value = Convert.ToDouble(MsgSplit(17))
                        cmd.Parameters.Add("@offer", SqlDbType.Int).Value = Convert.ToDouble(MsgSplit(18))
                        cmd.Parameters.Add("@qtyOffer", SqlDbType.Money).Value = Convert.ToDouble(MsgSplit(19))
                        cmd.Connection.Open()
                        cmd.ExecuteNonQuery()
                        cmd.Connection.Close()
                    End Using
         
                Case = "0" Or "3" Or "4" Or "6" Or "7" Or "8" Or "9"
                    InsertToOrder = ""
                Case Else
                    InsertToOrder = msg
            End Select
        Catch ex As Exception
            InsertToOrder = msg
        End Try

    End Function

[si_the_geek]

Welcome to VBForums

There are a few things about your code that are odd, and one or more of them could be causing the problem. One is this:

                Case = "0" Or "3" Or "4" Or "6" Or "7" Or "8" Or "9"

This code is actually invalid, but if you do not have Option Strict On (it is much better to have it On) then it will compile and run, but it won't do what you think... it will actually do bit-manipulation, which means it would be exactly the same as this:

                Case = "15"

What you should do instead is this:

                Case "0", "3", "4", "6", "7", "8", "9"

The next thing that stands out is your use of GoTo, which is usually a bad idea because it makes the code hard to read. If can often be re-written using more readable alternatives, and in this case you could change it from this:

        Dim z As Integer = 0
ulang:

        If statusOut = True Then
            TempString = ""
            JmlArr = 0 : oldCounter = 0 : z = 0
            Exit Sub
        End If
        If TempDfCounter = dfCounter Then GoTo ulang Else TempDfCounter = dfCounter
        If oldCounter > TempDfCounter Then JmlArr = maxArr Else JmlArr = TempDfCounter

        ...

        If oldCounter >= maxArr Then oldCounter = 0 Else oldCounter = JmlArr + 1
        GoTo ulang
    End Sub

...to this:

        Dim z As Integer = 0
        Do
            Do
                If statusOut = True Then
                    TempString = ""
                    JmlArr = 0 : oldCounter = 0 : z = 0
                    Exit Sub
                End If
            Loop While TempDfCounter = dfCounter 
            TempDfCounter = dfCounter
            If oldCounter > TempDfCounter Then JmlArr = maxArr Else JmlArr = TempDfCounter

            ...

            If oldCounter >= maxArr Then oldCounter = 0 Else oldCounter = JmlArr + 1
        Loop
    End Sub

After this change, it is easy to see that you have an infinite-loop, which is often a bad sign (because it tend to cause problems, including the kind of thing that is happening here).

I would recommend removing the infinite-loop, and instead let the BackgroundWorker finish when all the data is dealt with. You can re-start it when new data is available.

[esugiawan]

hello si_the_geek and others , thx for your reply , i try to following your instruction and result more faster than before.


i have one more problem. my program is slow because i use

" Dim message As String = System.Text.Encoding.ASCII.GetString(info._Buffer)"

i thing i should receiving data until i can find char CR LF from transmitted record then save it to text string , it will be faster to process, but until now i not find solution to get char CR LF from streaming

note : General format for each transmitted record ( Header| Date |Time | Sequence #|RecordType|DATA |Checksum |CR LF

Below is complete code to get data from socket server

     
        Public ReadOnly Property Stream As NetworkStream
            Get
                Return _Stream
            End Get
        End Property

        Private _LastReadLength As Integer
        Public ReadOnly Property LastReadLength As Integer
            Get
                Return _LastReadLength
            End Get
        End Property

        Private _Buffer(128) As Byte

   Public Sub AwaitData()
            _Stream.BeginRead(_Buffer, 0, _Buffer.Length, AddressOf DoReadData, Me)
        End Sub

        Private Sub DoReadData(result As IAsyncResult)

            Dim info As ConnectionInfo = CType(result.AsyncState, ConnectionInfo)
            Try
                If info._Stream IsNot Nothing AndAlso info._Stream.CanRead Then
                    info._LastReadLength = info._Stream.EndRead(result)
                    If info._LastReadLength > 0 Then
                        Dim message As String = System.Text.Encoding.ASCII.GetString(info._Buffer)
                        info._AppendMethod(message)
                    End If
                    info.AwaitData()
                End If
            Catch ex As Exception
                info._LastReadLength = -1
                info._AppendMethod(ex.Message)
            End Try
        End Sub
    End Class

[Niya]

Would you be averse to rewriting all that?

I have a lot of experience in writing communication protocols and I can show you much cleaner methods of processing received messages for your protocol.

My tried and true method is two have two threads. One for receiving the message and one for processing it. Both threads operate on a common buffer which is synchronized by a lock.

[esugiawan]

I love too and will do it if you have better solution , thx

[jmcilhinney]

Both threads operate on a common buffer which is synchronized by a lock.

A ConcurrentQueue(Of T) should be all you need in that regard.

[esugiawan]

Would you be averse to rewriting all that?

I have a lot of experience in writing communication protocols and I can show you much cleaner methods of processing received messages for your protocol.

My tried and true method is two have two threads. One for receiving the message and one for processing it. Both threads operate on a common buffer which is synchronized by a lock.

Yes, i will wait for your program, until now i still confused with protocol sockets
how to split the message with cr lf and then union the rest with new incoming message

[Niya]

A ConcurrentQueue(Of T) should be all you need in that regard.

Hmm...never used that class. I'll check it out.

Yes, i will wait for your program, until now i still confused with protocol sockets
how to split the message with cr lf and then union the rest with new incoming message

Is there a specific reason you are using a text protocol? A binary protocol would be far easier to implement. I'll try to find some time today to put together you an example for you.

[esugiawan]

Hi niya, I Just try to solve the problem, event with text protocol, i stll not find solution , if you have any suggestions with some program how to get cr lf from socket streaming, i will try to learn and try to implement, for your help

[esugiawan]

after all, i have 1 solution, make it faster, but not solve my program how to receive the rest of data split end of message but not with end of Lf and union it with new data incoming, I believe i should split when it still in byte , not in string ,to make it more faster

what i do is ,
1. Private _eolChar As Char = CChar(vbLf)
2 add some code in red color

        Private Sub DoReadData(result As IAsyncResult)


            Dim info As ConnectionInfo = CType(result.AsyncState, ConnectionInfo)
            Try
                If info._Stream IsNot Nothing AndAlso info._Stream.CanRead Then
                    info._LastReadLength = info._Stream.EndRead(result)
                    If info._LastReadLength > 0 Then
                        Dim message As String = System.Text.Encoding.ASCII.GetString(info._Buffer)
                       Dim sep() As Char = {EOLChar}
                        Dim z As Int32
                        Dim HasilSplit() As String = message.Split(sep)
                        For z = 0 To HasilSplit.Length - 1
                            info._AppendMethod(HasilSplit(z))
                        Next
                    End If
                    info.AwaitData()
                End If
            Catch ex As Exception
                info._LastReadLength = -1
                info._AppendMethod("DoReadData" & ex.Message)
            End Try


        End Sub

[esugiawan]

SOLVED using Parallel.For when looping
------------------

  Private Sub doRead(ByVal ar As System.IAsyncResult)
        Dim totalRead As Integer
        Try
            totalRead = client.GetStream.EndRead(ar) 'Ends the reading and returns the number of bytes read.
        Catch ex As Exception
            'The underlying socket have probably been closed OR an error has occured whilst trying to access it, either way, this is where you should remove close all eventuall connections
            'to this client and remove it from the list of connected clients.
        End Try

        If totalRead > 0 Then
            'the readBuffer array will contain everything read from the client.
            ' Dim receivedString As String = System.Text.Encoding.UTF8.GetString(readBuffer, 0, totalRead)
            Dim receivedString As String = Encoding.ASCII.GetString(readBuffer, 0, totalRead)
            'messageReceived(receivedString)  asli nya
            ''
            If (receivedString.IndexOf(MESSAGE_DELIMITER) > -1) Then
                Dim subMessages() As String = receivedString.Split(MESSAGE_DELIMITER)
                'The first element in the subMessages string array must be the last part of the current message.
                'So we append it to the StringBuilder and raise the dataReceived event
                sBuilder.Append(subMessages(0))
                messageReceived(subMessages(0))
                sBuilder = New System.Text.StringBuilder
                'If there are only 2 elements in the array, we know that the second one is an incomplete message,
                'though if there are more then two then every element inbetween the first and the last are complete messages:
                If subMessages.Length = 2 Then
                    sBuilder.Append(subMessages(1))
                    messageReceived(subMessages(1))
                Else
                    Parallel.For(0, subMessages.GetUpperBound(0) - 1,
                           Sub(index As Integer)
                               messageReceived(subMessages(index))
                           End Sub)

                    sBuilder.Append(subMessages(subMessages.GetUpperBound(0)))
                End If
            Else
                'MESSAGE_DELIMITER was not found in the message, so we just append everything to the stringbuilder.
                sBuilder.Append(receivedString)
                messageReceived(receivedString)
            End If
            ''

        End If

        Try
            client.GetStream.BeginRead(readBuffer, 0, BYTES_TO_READ, AddressOf doRead, Nothing) 'Begin the reading again.
        Catch ex As Exception
            'The underlying socket have probably been closed OR an error has occured whilst trying to access it, either way, this is where you should remove close all eventuall connections
            'to this client and remove it from the list of connected clients.
        End Try

[Niya]

Oh nice.

Sorry I didn't get back to you sooner. I got caught up in another discussion and spent a lot of time sussing out some things with regards to it. Nice you know you got it working. I hadn't forgotten about you. I tend to get carried way and distracted by my own passions frequently. Sorry about that.

[esugiawan]

Hi Niya, thx for your care, right now my code has solved my problem and is faster than before, but i see your code using winsock Demo.zip, perhaps you could help me to convert my private doread using ThreadPool.QueueUserWorkItem(Sub() ReceiveData() , because i thing your code , maybe faster than me

[esugiawan]

I have a new idea, and it more faster
the idea is streaming socket come from byte , so if i want to faster split it, i must split when the data still in byte not in string

        If totalRead > 0 Then
            For i = 0 To totalRead - 1
                If bytes(i) = EofLf Or bytes(i) = EofCr Then
                    If bytes(i) = EofLf Then
                        receivedString = Encoding.ASCII.GetString(bytes, start, i - start)
                        start = i
                    Else
                        receivedString = Encoding.ASCII.GetString(bytes, start, i - start + 1)
                        start = i + 1
                    End If
                    InsertToOrder(receivedString)
                End If
            Next
        End If