dcsimg
Results 1 to 14 of 14

Thread: [RESOLVED] how to get faster way from Array

  1. #1

    Thread Starter
    Junior Member
    Join Date
    Jan 2018
    Posts
    18

    Resolved [RESOLVED] how to get faster way from Array

    I create program to get data from socket server.
    General format for each transmitted record ( Header| Date |Time | Sequence #|RecordType|DATA |Checksum |CR LF
    Each item/field/column in the record is separated by pipe / ”|” /
    format Header always "IDX"
    -----
    First, i create procedure with socket to receive all data and save it to array ( let said df(array) )
    Second , i try to process df(array) using BackgroundWorker and save the valid data to sql server
    after all, the program running success full , but if i running from 9 AM to 16 PM, the difference time between first procedure and second procedure until 5 minute

    My question is, how to create faster array to solve that program ? thx

    ------
    Sample Data that i got from socket server
    Code:
            df(0) = "142300|001023095120|001022461737|??
    IDX|20180123|145628|01605548|5|PGAS                 |RG  |00000002470.00|00000002670.00|0000"
            df(1) = "0002490.00|00000002660.00|+00190.00|000277433200|0000724698055000|0017562|000000886.6667|00024241508196|00000002500.00|0000000265"
            df(2) = "0.00|000002386300|00000002660.00|000004706600|00000002612.00|0|?7
    IDX|20180123|145628|01605549|5|ICBP                 |RG  |0000"


    -------
    first procedure
    Code:
       
     Dim maxArr As Integer = 2000
    
     Private Sub AppendOutput(message As String)
    
            If message.Substring(0, 7) = "Welcome" Then
                Dim buffer() As Byte = System.Text.Encoding.ASCII.GetBytes("password" & vbCrLf & vbCrLf)
                _Connection.Stream.Write(buffer, 0, buffer.Length)
            End If
           df(dfCounter) = message
    
           If dfCounter >= maxArr - 1 Then  dfCounter = 0   Else  dfCounter +=  1
        End Sub
    Second procedure
    Code:
      Private Sub BackgroundWorker1_DoWork(sender As Object, e As DoWorkEventArgs) Handles BackgroundWorker1.DoWork
            Dim i As Integer = 0
            Dim oldCounter As Integer = 0
            Dim JmlArr As Integer = 0
            Dim TempString As String = ""
            Dim z As Integer = 0
    ulang:
    
            If statusOut = True Then
                TempString = ""
                JmlArr = 0 : oldCounter = 0 : z = 0
                Exit Sub
            End If
            If TempDfCounter = dfCounter Then GoTo ulang Else TempDfCounter = dfCounter
            If oldCounter > TempDfCounter Then JmlArr = maxArr Else JmlArr = TempDfCounter
    
    
            For i = oldCounter To JmlArr
                TempString += df(i)
            Next
    
            pstr = oldCounter & " to " & JmlArr
            OutputMsg(pstr)
    
            Dim HasilSplit() As String = Split(TempString, "IDX|")
            For z = 0 To HasilSplit.Length - 1
                TempString = InsertToOrder("IDX|" & HasilSplit(z))
                OutputMsg("z:" & z & " to " & HasilSplit.Length - 1 & " hasil & " & HasilSplit(z))
            Next
    
            If oldCounter >= maxArr Then oldCounter = 0 Else oldCounter = JmlArr + 1
            GoTo ulang
        End Sub
    Code:
    Function InsertToOrder(msg As String) As String
            Try
                Dim MsgSplit() As String
                MsgSplit = Split(msg, "|")
                If MsgSplit.Length < 10 Then Exit Function
                Select Case MsgSplit(4) ' kode datafeed
                    Case Is = "1" 'order
                        sqlstr = "dbo.spUpdate1"
                        Using cmd As SqlClient.SqlCommand = New SqlClient.SqlCommand(sqlstr, con)
                            cmd.CommandType = CommandType.StoredProcedure
                            cmd.Parameters.Add("@SecCode", SqlDbType.NVarChar).Value = MsgSplit(7).Trim
                            cmd.Parameters.Add("@board", SqlDbType.NVarChar).Value = MsgSplit(8).Trim
                            cmd.Parameters.Add("@bid", SqlDbType.Int).Value = Convert.ToDouble(MsgSplit(15).Trim)
                            cmd.Parameters.Add("@qtybid", SqlDbType.Money).Value = Convert.ToDouble(MsgSplit(16).Trim)
                            cmd.Parameters.Add("@offer", SqlDbType.Int).Value = Convert.ToDouble(MsgSplit(17).Trim)
                            cmd.Parameters.Add("@qtyOffer", SqlDbType.Money).Value = Convert.ToDouble(MsgSplit(18).Trim)
                            cmd.Connection.Open()
                            cmd.ExecuteNonQuery()
                            cmd.Connection.Close()
                        End Using
    
                    Case Is = "2" ' Trade
                        sqlstr = "dbo.spUpdate2"
                        Using cmd As SqlClient.SqlCommand = New SqlClient.SqlCommand(sqlstr, con)
                            cmd.CommandType = CommandType.StoredProcedure
                            cmd.Parameters.Add("@SecCode", SqlDbType.NVarChar).Value = MsgSplit(7).Trim
                            cmd.Parameters.Add("@board", SqlDbType.NVarChar).Value = MsgSplit(8).Trim
                            cmd.Parameters.Add("@LastPrice", SqlDbType.Int).Value = Convert.ToDouble(MsgSplit(10))
                            cmd.Parameters.Add("@bid", SqlDbType.Int).Value = Convert.ToDouble(MsgSplit(16))
                            cmd.Parameters.Add("@qtybid", SqlDbType.Money).Value = Convert.ToDouble(MsgSplit(17))
                            cmd.Parameters.Add("@offer", SqlDbType.Int).Value = Convert.ToDouble(MsgSplit(18))
                            cmd.Parameters.Add("@qtyOffer", SqlDbType.Money).Value = Convert.ToDouble(MsgSplit(19))
                            cmd.Connection.Open()
                            cmd.ExecuteNonQuery()
                            cmd.Connection.Close()
                        End Using
             
                    Case = "0" Or "3" Or "4" Or "6" Or "7" Or "8" Or "9"
                        InsertToOrder = ""
                    Case Else
                        InsertToOrder = msg
                End Select
            Catch ex As Exception
                InsertToOrder = msg
            End Try
    
        End Function
    Last edited by esugiawan; Feb 1st, 2018 at 03:48 AM. Reason: solve

  2. #2
    Super Moderator si_the_geek's Avatar
    Join Date
    Jul 2002
    Location
    Bristol, UK
    Posts
    40,213

    Re: how to get faster way from Array

    Welcome to VBForums

    There are a few things about your code that are odd, and one or more of them could be causing the problem. One is this:
    Code:
                    Case = "0" Or "3" Or "4" Or "6" Or "7" Or "8" Or "9"
    This code is actually invalid, but if you do not have Option Strict On (it is much better to have it On) then it will compile and run, but it won't do what you think... it will actually do bit-manipulation, which means it would be exactly the same as this:
    Code:
                    Case = "15"
    What you should do instead is this:
    Code:
                    Case "0", "3", "4", "6", "7", "8", "9"

    The next thing that stands out is your use of GoTo, which is usually a bad idea because it makes the code hard to read. If can often be re-written using more readable alternatives, and in this case you could change it from this:
    Code:
            Dim z As Integer = 0
    ulang:
    
            If statusOut = True Then
                TempString = ""
                JmlArr = 0 : oldCounter = 0 : z = 0
                Exit Sub
            End If
            If TempDfCounter = dfCounter Then GoTo ulang Else TempDfCounter = dfCounter
            If oldCounter > TempDfCounter Then JmlArr = maxArr Else JmlArr = TempDfCounter
    
            ...
    
            If oldCounter >= maxArr Then oldCounter = 0 Else oldCounter = JmlArr + 1
            GoTo ulang
        End Sub
    ...to this:
    Code:
            Dim z As Integer = 0
            Do
                Do
                    If statusOut = True Then
                        TempString = ""
                        JmlArr = 0 : oldCounter = 0 : z = 0
                        Exit Sub
                    End If
                Loop While TempDfCounter = dfCounter 
                TempDfCounter = dfCounter
                If oldCounter > TempDfCounter Then JmlArr = maxArr Else JmlArr = TempDfCounter
    
                ...
    
                If oldCounter >= maxArr Then oldCounter = 0 Else oldCounter = JmlArr + 1
            Loop
        End Sub
    After this change, it is easy to see that you have an infinite-loop, which is often a bad sign (because it tend to cause problems, including the kind of thing that is happening here).

    I would recommend removing the infinite-loop, and instead let the BackgroundWorker finish when all the data is dealt with. You can re-start it when new data is available.

  3. #3

    Thread Starter
    Junior Member
    Join Date
    Jan 2018
    Posts
    18

    Re: how to get faster way from Array

    hello si_the_geek and others , thx for your reply , i try to following your instruction and result more faster than before.


    i have one more problem. my program is slow because i use
    Code:
    " Dim message As String = System.Text.Encoding.ASCII.GetString(info._Buffer)"
    i thing i should receiving data until i can find char CR LF from transmitted record then save it to text string , it will be faster to process, but until now i not find solution to get char CR LF from streaming

    note : General format for each transmitted record ( Header| Date |Time | Sequence #|RecordType|DATA |Checksum |CR LF

    Below is complete code to get data from socket server
    Code:
         
            Public ReadOnly Property Stream As NetworkStream
                Get
                    Return _Stream
                End Get
            End Property
    
            Private _LastReadLength As Integer
            Public ReadOnly Property LastReadLength As Integer
                Get
                    Return _LastReadLength
                End Get
            End Property
    
            Private _Buffer(128) As Byte
    
       Public Sub AwaitData()
                _Stream.BeginRead(_Buffer, 0, _Buffer.Length, AddressOf DoReadData, Me)
            End Sub
    
            Private Sub DoReadData(result As IAsyncResult)
    
                Dim info As ConnectionInfo = CType(result.AsyncState, ConnectionInfo)
                Try
                    If info._Stream IsNot Nothing AndAlso info._Stream.CanRead Then
                        info._LastReadLength = info._Stream.EndRead(result)
                        If info._LastReadLength > 0 Then
                            Dim message As String = System.Text.Encoding.ASCII.GetString(info._Buffer)
                            info._AppendMethod(message)
                        End If
                        info.AwaitData()
                    End If
                Catch ex As Exception
                    info._LastReadLength = -1
                    info._AppendMethod(ex.Message)
                End Try
            End Sub
        End Class
    Last edited by esugiawan; Jan 24th, 2018 at 06:33 AM.

  4. #4
    Angel of Code Niya's Avatar
    Join Date
    Nov 2011
    Posts
    5,626

    Re: how to get faster way from Array

    Would you be averse to rewriting all that?

    I have a lot of experience in writing communication protocols and I can show you much cleaner methods of processing received messages for your protocol.

    My tried and true method is two have two threads. One for receiving the message and one for processing it. Both threads operate on a common buffer which is synchronized by a lock.
    Treeview with NodeAdded/NodesRemoved events | BlinkLabel control | Calculate Permutations | Object Enums | ComboBox with centered items | .Net Internals article(not mine) | Wizard Control | Understanding Multi-Threading | Simple file compression | Demon Arena


    C++ programmers will dismiss you as a cretinous simpleton for your inability to keep track of pointers chained 6 levels deep and Java programmers will pillory you for buying into the evils of Microsoft. Meanwhile C# programmers will get paid just a little bit more than you for writing exactly the same code and VB6 programmers will continue to whitter on about "footprints". - FunkyDexter

    There's just no reason to use garbage like InputBox. -jmcilhinney

  5. #5

    Thread Starter
    Junior Member
    Join Date
    Jan 2018
    Posts
    18

    Re: how to get faster way from Array

    I love too and will do it if you have better solution , thx

  6. #6
    .NUT jmcilhinney's Avatar
    Join Date
    May 2005
    Location
    Sydney, Australia
    Posts
    100,048

    Re: how to get faster way from Array

    Quote Originally Posted by Niya View Post
    Both threads operate on a common buffer which is synchronized by a lock.
    A ConcurrentQueue(Of T) should be all you need in that regard.
    Why is my data not saved to my database? | MSDN Data Walkthroughs
    VBForums Database Development FAQ
    My CodeBank Submissions: VB | C#
    My Blog: Data Among Multiple Forms (3 parts)
    Beginner Tutorials: VB | C# | SQL

  7. #7

    Thread Starter
    Junior Member
    Join Date
    Jan 2018
    Posts
    18

    Re: how to get faster way from Array

    Quote Originally Posted by Niya View Post
    Would you be averse to rewriting all that?

    I have a lot of experience in writing communication protocols and I can show you much cleaner methods of processing received messages for your protocol.

    My tried and true method is two have two threads. One for receiving the message and one for processing it. Both threads operate on a common buffer which is synchronized by a lock.
    Yes, i will wait for your program, until now i still confused with protocol sockets
    how to split the message with cr lf and then union the rest with new incoming message

  8. #8
    Angel of Code Niya's Avatar
    Join Date
    Nov 2011
    Posts
    5,626

    Re: how to get faster way from Array

    Quote Originally Posted by jmcilhinney View Post
    A ConcurrentQueue(Of T) should be all you need in that regard.
    Hmm...never used that class. I'll check it out.

    Quote Originally Posted by esugiawan View Post
    Yes, i will wait for your program, until now i still confused with protocol sockets
    how to split the message with cr lf and then union the rest with new incoming message
    Is there a specific reason you are using a text protocol? A binary protocol would be far easier to implement. I'll try to find some time today to put together you an example for you.
    Treeview with NodeAdded/NodesRemoved events | BlinkLabel control | Calculate Permutations | Object Enums | ComboBox with centered items | .Net Internals article(not mine) | Wizard Control | Understanding Multi-Threading | Simple file compression | Demon Arena


    C++ programmers will dismiss you as a cretinous simpleton for your inability to keep track of pointers chained 6 levels deep and Java programmers will pillory you for buying into the evils of Microsoft. Meanwhile C# programmers will get paid just a little bit more than you for writing exactly the same code and VB6 programmers will continue to whitter on about "footprints". - FunkyDexter

    There's just no reason to use garbage like InputBox. -jmcilhinney

  9. #9

    Thread Starter
    Junior Member
    Join Date
    Jan 2018
    Posts
    18

    Re: how to get faster way from Array

    Hi niya, I Just try to solve the problem, event with text protocol, i stll not find solution , if you have any suggestions with some program how to get cr lf from socket streaming, i will try to learn and try to implement, for your help

  10. #10

    Thread Starter
    Junior Member
    Join Date
    Jan 2018
    Posts
    18

    Re: how to get faster way from Array

    after all, i have 1 solution, make it faster, but not solve my program how to receive the rest of data split end of message but not with end of Lf and union it with new data incoming, I believe i should split when it still in byte , not in string ,to make it more faster

    what i do is ,
    1. Private _eolChar As Char = CChar(vbLf)
    2 add some code in red color
    Code:
            Private Sub DoReadData(result As IAsyncResult)
    
    
                Dim info As ConnectionInfo = CType(result.AsyncState, ConnectionInfo)
                Try
                    If info._Stream IsNot Nothing AndAlso info._Stream.CanRead Then
                        info._LastReadLength = info._Stream.EndRead(result)
                        If info._LastReadLength > 0 Then
                            Dim message As String = System.Text.Encoding.ASCII.GetString(info._Buffer)
                           Dim sep() As Char = {EOLChar}
                            Dim z As Int32
                            Dim HasilSplit() As String = message.Split(sep)
                            For z = 0 To HasilSplit.Length - 1
                                info._AppendMethod(HasilSplit(z))
                            Next
                        End If
                        info.AwaitData()
                    End If
                Catch ex As Exception
                    info._LastReadLength = -1
                    info._AppendMethod("DoReadData" & ex.Message)
                End Try
    
    
            End Sub

  11. #11

    Thread Starter
    Junior Member
    Join Date
    Jan 2018
    Posts
    18

    Re: how to get faster way from Array

    SOLVED using Parallel.For when looping
    ------------------

    Code:
      Private Sub doRead(ByVal ar As System.IAsyncResult)
            Dim totalRead As Integer
            Try
                totalRead = client.GetStream.EndRead(ar) 'Ends the reading and returns the number of bytes read.
            Catch ex As Exception
                'The underlying socket have probably been closed OR an error has occured whilst trying to access it, either way, this is where you should remove close all eventuall connections
                'to this client and remove it from the list of connected clients.
            End Try
    
            If totalRead > 0 Then
                'the readBuffer array will contain everything read from the client.
                ' Dim receivedString As String = System.Text.Encoding.UTF8.GetString(readBuffer, 0, totalRead)
                Dim receivedString As String = Encoding.ASCII.GetString(readBuffer, 0, totalRead)
                'messageReceived(receivedString)  asli nya
                ''
                If (receivedString.IndexOf(MESSAGE_DELIMITER) > -1) Then
                    Dim subMessages() As String = receivedString.Split(MESSAGE_DELIMITER)
                    'The first element in the subMessages string array must be the last part of the current message.
                    'So we append it to the StringBuilder and raise the dataReceived event
                    sBuilder.Append(subMessages(0))
                    messageReceived(subMessages(0))
                    sBuilder = New System.Text.StringBuilder
                    'If there are only 2 elements in the array, we know that the second one is an incomplete message,
                    'though if there are more then two then every element inbetween the first and the last are complete messages:
                    If subMessages.Length = 2 Then
                        sBuilder.Append(subMessages(1))
                        messageReceived(subMessages(1))
                    Else
                        Parallel.For(0, subMessages.GetUpperBound(0) - 1,
                               Sub(index As Integer)
                                   messageReceived(subMessages(index))
                               End Sub)
    
                        sBuilder.Append(subMessages(subMessages.GetUpperBound(0)))
                    End If
                Else
                    'MESSAGE_DELIMITER was not found in the message, so we just append everything to the stringbuilder.
                    sBuilder.Append(receivedString)
                    messageReceived(receivedString)
                End If
                ''
    
            End If
    
            Try
                client.GetStream.BeginRead(readBuffer, 0, BYTES_TO_READ, AddressOf doRead, Nothing) 'Begin the reading again.
            Catch ex As Exception
                'The underlying socket have probably been closed OR an error has occured whilst trying to access it, either way, this is where you should remove close all eventuall connections
                'to this client and remove it from the list of connected clients.
            End Try

  12. #12
    Angel of Code Niya's Avatar
    Join Date
    Nov 2011
    Posts
    5,626

    Re: how to get faster way from Array

    Oh nice.

    Sorry I didn't get back to you sooner. I got caught up in another discussion and spent a lot of time sussing out some things with regards to it. Nice you know you got it working. I hadn't forgotten about you. I tend to get carried way and distracted by my own passions frequently. Sorry about that.
    Treeview with NodeAdded/NodesRemoved events | BlinkLabel control | Calculate Permutations | Object Enums | ComboBox with centered items | .Net Internals article(not mine) | Wizard Control | Understanding Multi-Threading | Simple file compression | Demon Arena


    C++ programmers will dismiss you as a cretinous simpleton for your inability to keep track of pointers chained 6 levels deep and Java programmers will pillory you for buying into the evils of Microsoft. Meanwhile C# programmers will get paid just a little bit more than you for writing exactly the same code and VB6 programmers will continue to whitter on about "footprints". - FunkyDexter

    There's just no reason to use garbage like InputBox. -jmcilhinney

  13. #13

    Thread Starter
    Junior Member
    Join Date
    Jan 2018
    Posts
    18

    Re: how to get faster way from Array

    Hi Niya, thx for your care, right now my code has solved my problem and is faster than before, but i see your code using winsock Demo.zip, perhaps you could help me to convert my private doread using ThreadPool.QueueUserWorkItem(Sub() ReceiveData() , because i thing your code , maybe faster than me

  14. #14

    Thread Starter
    Junior Member
    Join Date
    Jan 2018
    Posts
    18

    SOLVED Re: how to get faster way from Array

    I have a new idea, and it more faster
    the idea is streaming socket come from byte , so if i want to faster split it, i must split when the data still in byte not in string

    Code:
            If totalRead > 0 Then
                For i = 0 To totalRead - 1
                    If bytes(i) = EofLf Or bytes(i) = EofCr Then
                        If bytes(i) = EofLf Then
                            receivedString = Encoding.ASCII.GetString(bytes, start, i - start)
                            start = i
                        Else
                            receivedString = Encoding.ASCII.GetString(bytes, start, i - start + 1)
                            start = i + 1
                        End If
                        InsertToOrder(receivedString)
                    End If
                Next
            End If

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Featured


Click Here to Expand Forum to Full Width