Read MultiLine Data via Serial

[joko markono]

Hi All,

here is my code from sender terminal:

Dim send1 = String.Format(finaltitleTemplate1, titleheader1(0), titleheader1(1), titleheader1(2), titleheader1(3), titleheader1(4))
Dim send2 = String.Format(finaldataTemplate1, datavalue1(0), datavalue1(1), datavalue1(2), datavalue1(3), datavalue1(4))
Dim send3 = String.Format(finaltitleTemplate2, titleheader2(0), titleheader2(1), titleheader2(2), titleheader2(3), titleheader2(4))
Dim send4 = String.Format(finaldataTemplate2, datavalue2(0), datavalue2(1), datavalue2(2), datavalue2(3), datavalue2(4))

        CompiledBox1.Text = Chr(&H15) & vbNewLine & send1 & vbNewLine & Chr(&H14) & send2 & vbNewLine & send3 & vbNewLine & send4

        'to send data to serial port 1
        If SerialPort1.IsOpen Then
            Serial1OpenButton.BackColor = Color.LimeGreen
            SerialPort1.Write(Chr(&H15) & vbNewLine & send1 & vbNewLine & Chr(&H14) & send2 & vbNewLine & send3 & vbNewLine & send4)
        End If

and here is the code on the receiver terminal:

Private ReadSerialPortThread As Threading.Thread
    Private Sub SerialPort1_DataReceived(sender As Object, e As IO.Ports.SerialDataReceivedEventArgs) Handles SerialPort1.DataReceived
        'this is it
        ReadWait.Set() 'cause ReadSerialPort to run
    End Sub

    Private ReadWait As New Threading.AutoResetEvent(False)

    Private Sub ReadSerialPort()
        'this is run as a Background task
        Dim data As New System.Text.StringBuilder(1024 * 1024) 'accumulate SerialPort data here
        Dim mess As New System.Text.StringBuilder              'individual message
        Dim foundMess As Boolean = False                       'found message

        Do
            ReadWait.WaitOne() 'wait for SerialPort1.DataReceived
            Do While SerialPort1.BytesToRead > 0 'accumulate all available bytes
                data.Append(SerialPort1.ReadExisting)
            Loop
            Do 'process message, if one available

                mess.Length = 0 'reset message
                foundMess = False 'if message not found,

                Dim idx As Integer = 0
                For idx = 0 To data.Length - 1 'look for CR
                    If data(idx) = ControlChars.Cr Then
                        'note that the CR will NOT be in mess
                        foundMess = True 'found message
                        Exit For

                    End If
                    mess.Append(data(idx))
                Next
                If foundMess Then
                    data.Remove(0, idx + 1) 'remove found message from data
                    'process your message here <<<<<<<<<<<<
                    'how to interact with the UI

                    Me.Invoke(Sub()
                                  datareceive_label.Text = (mess.ToString)
                              End Sub)

                    'end example
                End If


                'look for more messages in data?
            Loop While foundMess AndAlso data.Length > 0
        Loop
    End Sub

the problem is that I can only receive the first line of the data only.
How to show all lines of sent data with above code?

TIA

[.paul.]

You should try splitting data on vbnewline…

Dim lined() As String = data.Split(New String() {vbNewLine}, StringSplitOptions.None)

[pourkascheff]

Hi joko markono,

Why not using

        SerialPort1.Read(Buffer, 0, Buffer.Length - 1) 'To recieving and_
        SerialPort1.Write(Buffer, 0, Buffer.Length - 1) '_to transmitting

at first place? All USART based communication frames are "byte" dedicated, so no more strings and their textual capabilities therefore a more practical response in return.

(It is a personal perception, if your code works sort-of-fine, do not get yourself into more troubles)

[passel]

If you're sending your data as lines, then I think it would be easiest to read them as lines.
What I've done when reading serial data that is transmitted as lines of text, is simply start a background thread containing code along these lines. Also, since you have this background thread reading serial data a line at a time, you don't need a DataReceived Event handler.

Private Sub ReadThread()
  Do Until Leaving
    Try
      Dim line as String = comPort.ReadLine()
      Me.Invoke(SerialLineIn, line)
    Catch ex As Exception
    End Try
  Loop
End Sub

The thread will wait on the ReadLine call until a line is received, it will then invoke a Sub on the GUI thread to process that line, and then loop back and wait on the ReadLine for the next line.

When the program is going to close the port, it sets the boolean value "Leaving" to True first and then closes the port.
Closing the port will cause an exception on the ReadLine call, which will be handled (ignored), and the loop will exit because of the Leaving flag being True and the thread will end.

[joko markono]

If you're sending your data as lines, then I think it would be easiest to read them as lines.
What I've done when reading serial data that is transmitted as lines of text, is simply start a background thread containing code along these lines. Also, since you have this background thread reading serial data a line at a time, you don't need a DataReceived Event handler.

Private Sub ReadThread()
  Do Until Leaving
    Try
      Dim line as String = comPort.ReadLine()
      Me.Invoke(SerialLineIn, line)
    Catch ex As Exception
    End Try
  Loop
End Sub

The thread will wait on the ReadLine call until a line is received, it will then invoke a Sub on the GUI thread to process that line, and then loop back and wait on the ReadLine for the next line.

When the program is going to close the port, it sets the boolean value "Leaving" to True first and then closes the port.
Closing the port will cause an exception on the ReadLine call, which will be handled (ignored), and the loop will exit because of the Leaving flag being True and the thread will end.

definitely no problem with single line data and I can split them with no issue.

[joko markono]

You should try splitting data on vbnewline…

Dim lined() As String = data.Split(New String() {vbNewLine}, StringSplitOptions.None)

you mean on the receiving side?

[dbasnett]

Try changing the sender to

        Dim send1 = String.Format(finaltitleTemplate1, titleheader1(0), titleheader1(1), titleheader1(2), titleheader1(3), titleheader1(4))
        Dim send2 = String.Format(finaldataTemplate1, datavalue1(0), datavalue1(1), datavalue1(2), datavalue1(3), datavalue1(4))
        Dim send3 = String.Format(finaltitleTemplate2, titleheader2(0), titleheader2(1), titleheader2(2), titleheader2(3), titleheader2(4))
        Dim send4 = String.Format(finaldataTemplate2, datavalue2(0), datavalue2(1), datavalue2(2), datavalue2(3), datavalue2(4))

        Dim sendText As String
        sendText = String.Format("{1}{0}{3}{2}{4}{0}{5}{0}{6}{0}", ControlChars.Cr,
                                                                    Chr(&H15),
                                                                    Chr(&H14),
                                                                    send1,
                                                                    send2,
                                                                    send3,
                                                                    send4)

        CompiledBox1.Text = sendText

        'to send data to serial port 1
        If SerialPort1.IsOpen Then
            Serial1OpenButton.BackColor = Color.LimeGreen
            SerialPort1.Write(sendText)
        End If

Based on this in the receiver you will only see one message.

    datareceive_label.Text = (mess.ToString)

[passel]

definitely no problem with single line data and I can split them with no issue.

That is fine, but you said you were only receiving the first line.

Perhaps you just meant that if you send three lines, or five lines, you would like to know when you've received all the lines so you can parse out all the lines at once.
If you don't have a "flag" of some sort to look for to indicate the end of your transmission and then process the collected buffer or a "header" to indicate how many lines follow, then I've used a watchdog type counter, or a time tag to identify when a line has been receive, i.e. when a line of data is received, the counter is set to a value to be counted down by a separate timer or periodic loop, or a time tag is set to the current time.

The point being in the periodic loop, it can check if perhaps a tenth of a second has passed since the last line was received and if so, process the collected lines.

As for using ReadLine, the reason I use it is if I know that the serial data I'm receiving are lines of text, then I don't have to collect and process multiple portions of parts of a line, and manually manage the buffer to extract lines from the buffered data. The ReadLine will take care of that, and just give you each line of text as received (minus the line termination character(s)). Much less code and processing that my code has to do.

Of course, for pretty much any other type of data, then accumulating the bytes into a buffer and processing them manually is the correct choice. Since .Net provides concurrent queues to make accessing data between threads easier, then that is my usual choice to accumulate the received data in.

[dbasnett]

...Of course, for pretty much any other type of data, then accumulating the bytes into a buffer and processing them manually is the correct choice. Since .Net provides concurrent queues to make accessing data between threads easier, then that is my usual choice to accumulate the received data in.

The code they are using looks like something I wrote and without a serious re-write ReadLine is not an option. Based on this piece of code,

Me.Invoke(Sub()
                                  datareceive_label.Text = (mess.ToString)
                              End Sub)

they will only see the last message. What might be helpful for them is to change that code to use a RichTextBox.

        Me.Invoke(Sub()
                      RichTextBox1.AppendText(mess.ToString)
                      RichTextBox1.AppendText(ControlChars.Cr)
                  End Sub)

[joko markono]

Try changing the sender to

        Dim send1 = String.Format(finaltitleTemplate1, titleheader1(0), titleheader1(1), titleheader1(2), titleheader1(3), titleheader1(4))
        Dim send2 = String.Format(finaldataTemplate1, datavalue1(0), datavalue1(1), datavalue1(2), datavalue1(3), datavalue1(4))
        Dim send3 = String.Format(finaltitleTemplate2, titleheader2(0), titleheader2(1), titleheader2(2), titleheader2(3), titleheader2(4))
        Dim send4 = String.Format(finaldataTemplate2, datavalue2(0), datavalue2(1), datavalue2(2), datavalue2(3), datavalue2(4))

        Dim sendText As String
        sendText = String.Format("{1}{0}{3}{2}{4}{0}{5}{0}{6}{0}", ControlChars.Cr,
                                                                    Chr(&H15),
                                                                    Chr(&H14),
                                                                    send1,
                                                                    send2,
                                                                    send3,
                                                                    send4)

        CompiledBox1.Text = sendText

        'to send data to serial port 1
        If SerialPort1.IsOpen Then
            Serial1OpenButton.BackColor = Color.LimeGreen
            SerialPort1.Write(sendText)
        End If

Based on this in the receiver you will only see one message.

    datareceive_label.Text = (mess.ToString)

I can only receive last line of message with this method.

[joko markono]

That is fine, but you said you were only receiving the first line.

Perhaps you just meant that if you send three lines, or five lines, you would like to know when you've received all the lines so you can parse out all the lines at once.
If you don't have a "flag" of some sort to look for to indicate the end of your transmission and then process the collected buffer or a "header" to indicate how many lines follow, then I've used a watchdog type counter, or a time tag to identify when a line has been receive, i.e. when a line of data is received, the counter is set to a value to be counted down by a separate timer or periodic loop, or a time tag is set to the current time.

The point being in the periodic loop, it can check if perhaps a tenth of a second has passed since the last line was received and if so, process the collected lines.

As for using ReadLine, the reason I use it is if I know that the serial data I'm receiving are lines of text, then I don't have to collect and process multiple portions of parts of a line, and manually manage the buffer to extract lines from the buffered data. The ReadLine will take care of that, and just give you each line of text as received (minus the line termination character(s)). Much less code and processing that my code has to do.

Of course, for pretty much any other type of data, then accumulating the bytes into a buffer and processing them manually is the correct choice. Since .Net provides concurrent queues to make accessing data between threads easier, then that is my usual choice to accumulate the received data in.

my incoming data should look like this and i want to display it all on a webcam text overlay:
Attachment 187301

[joko markono]

The code they are using looks like something I wrote and without a serious re-write ReadLine is not an option. Based on this piece of code,

Me.Invoke(Sub()
                                  datareceive_label.Text = (mess.ToString)
                              End Sub)

they will only see the last message. What might be helpful for them is to change that code to use a RichTextBox.

        Me.Invoke(Sub()
                      RichTextBox1.AppendText(mess.ToString)
                      RichTextBox1.AppendText(ControlChars.Cr)
                  End Sub)

since I want to overlay the data on a live video, I can't use richtextbox because I need to make it transparent.
Also, I have tried the richtextbox but the data keep on coming without displaying the latest one.

If I do this:

RichTextBox1.AppendText(mess.ToString)
                                  RichTextBox1.AppendText(ControlChars.Cr)
                                  RichTextBox2.Text = (mess.ToString)

then I got the latest data, but again only display one line of the data.

[dbasnett]

I went back to my original code and tried to find an error. I'm re-posting it since it might have changed. This code typically runs on a PC with a loop back plug so that I can test both sender and receiver on the same machine. The code works so my guess is that it is how you are processing messages / a misunderstanding of the flow of data.

When I say the code works I mean that every message sent is received and processed.

Public Class Form1
    Private WithEvents SerialPort1 As New IO.Ports.SerialPort

    Private ReadSerialPortTask As Task
    Private Sub Form1_Shown(sender As Object, e As EventArgs) Handles Me.Shown
        'start the worker task <<<<<<<<<<<<!!!!!
        'this could be in the load event also
        ReadSerialPortTask = Task.Run(Sub() ReadSerialPort())
        ' this is ran on a PC with a hardware serial port
        '  this port has a loop back plug which means what is sent is received on the same PC
        Dim t As Task
        t = Task.Run(Sub()
                         SerialPort1.PortName = "COM1"
                         'OverRun errors occur at speeds > 38400 when using the loop back
                         SerialPort1.BaudRate = 38400 '110, 300, 600, 1200, 2400, 4800, 9600, 14400, 19200, 38400, 57600, 115200 
                         SerialPort1.Open()
                         SerialPort1.DtrEnable = True
                         Dim mess As String
                         Dim mTX As Integer
                         Threading.Thread.Sleep(250)
                         Const sendCT As Integer = 400
                         For mTX = 1 To sendCT
                             mess = String.Format("{1}{0}FOO {1}{0}{1,4} Lorem ipsum dolor sit amet, consectetur adipiscing...{0}",
                                                    ControlChars.Tab, mTX)
                             SerialPort1.Write(mess)
                             SerialPort1.Write(ControlChars.CrLf)
                         Next

                         Threading.Thread.Sleep(sendCT \ 100 * 1000 + 500) 'give a chance for all messages to be processed

                         Me.BeginInvoke(Sub()
                                            Label1.Text = _DataReceivedEvents.ToString("n0")
                                            Label2.Text = _messagesRcvd.ToString("n0")
                                        End Sub)
                     End Sub)
    End Sub

    Private _ErrorReceivedEvents As Long = 0L
    Private Sub SerialPort1_ErrorReceived(sender As Object,
                                          e As IO.Ports.SerialErrorReceivedEventArgs) Handles SerialPort1.ErrorReceived
        Threading.Interlocked.Increment(_ErrorReceivedEvents)
        Dim err As Long = Threading.Interlocked.Read(_ErrorReceivedEvents)
        Debug.Write(err.ToString("n0"))
        Debug.Write("  ERR  ")
        Debug.WriteLine(e.EventType.ToString)
        ' Stop
    End Sub

    Private _DataReceivedEvents As Integer = 0 'typically a lot more _DataReceivedEvents than of _messagesRcvd
    Private ReadWait As New Threading.AutoResetEvent(False)
    Private Sub SerialPort1_DataReceived(sender As Object,
                                         e As IO.Ports.SerialDataReceivedEventArgs) Handles SerialPort1.DataReceived
        'this is the entire DataReceived event handler
        Threading.Interlocked.Increment(_DataReceivedEvents)
        ReadWait.Set() 'cause ReadSerialPort to run
    End Sub

    Private _messagesRcvd As Integer = 0
    Private Sub ReadSerialPort()
        '
        'this is run as a Background task
        '
        Dim data As New System.Text.StringBuilder(1024 * 1024) 'accumulate SerialPort data here
        Dim mess As New System.Text.StringBuilder(1024 * 2) 'individual message
        Dim foundMess As Boolean = False
        Do
            ReadWait.WaitOne() 'wait for SerialPort1.DataReceived

            Do While SerialPort1.BytesToRead > 0 'accumulate all available bytes
                data.Append(SerialPort1.ReadExisting)
            Loop
            Do 'process message, if one available
                '  protocol
                '   in this case looking for a string terminated by CR
                mess.Length = 0 'reset message
                foundMess = False 'not found
                Dim idx As Integer = 0
                For idx = 0 To data.Length - 1
                    'look for CR??????
                    If data(idx) = ControlChars.Cr Then '<<<<<<<<<<<<<<<<<<<<<<<
                        'note that the CR will NOT be in mess
                        foundMess = True 'message found 
                        Threading.Interlocked.Increment(_messagesRcvd)
                        Exit For
                    ElseIf data(idx) = ControlChars.Lf Then '<<<<<<<<<<<<<<<<<<<<<<<
                        Continue For 'skip LF
                    End If
                    mess.Append(data(idx))
                Next
                If foundMess Then
                    data.Remove(0, idx + 1) 'remove found message from data
                    '
                    'process the message here <<<<<<<<<<<<
                    'note how UI interaction is handled in example
                    '

                    'example MY message has parts separated by tab
                    Dim parts() As String
                    parts = mess.ToString.Split(ControlChars.Tab)
                    '  interact with the UI
                    Me.Invoke(Sub()
                                  Label1.Text = parts(0)
                                  If parts.Length >= 3 Then TextBox1.Text = parts(2)
                                  RichTextBox1.AppendText(parts(1))
                                  RichTextBox1.AppendText(ControlChars.Cr)
                                  RichTextBox1.ScrollToCaret()
                              End Sub)
                    'end example
                End If

                'look for more messages in data?
            Loop While foundMess AndAlso data.Length > 0
        Loop
    End Sub
End Class

At the end of this post https://www.vbforums.com/showthread....=1#post3279031 there is another alternative to how this might be done. It may suit your needs better.

[joko markono]

I went back to my original code and tried to find an error. I'm re-posting it since it might have changed. This code typically runs on a PC with a loop back plug so that I can test both sender and receiver on the same machine. The code works so my guess is that it is how you are processing messages / a misunderstanding of the flow of data.

When I say the code works I mean that every message sent is received and processed.

Public Class Form1
    Private WithEvents SerialPort1 As New IO.Ports.SerialPort

    Private ReadSerialPortTask As Task
    Private Sub Form1_Shown(sender As Object, e As EventArgs) Handles Me.Shown
        'start the worker task <<<<<<<<<<<<!!!!!
        'this could be in the load event also
        ReadSerialPortTask = Task.Run(Sub() ReadSerialPort())
        ' this is ran on a PC with a hardware serial port
        '  this port has a loop back plug which means what is sent is received on the same PC
        Dim t As Task
        t = Task.Run(Sub()
                         SerialPort1.PortName = "COM1"
                         'OverRun errors occur at speeds > 38400 when using the loop back
                         SerialPort1.BaudRate = 38400 '110, 300, 600, 1200, 2400, 4800, 9600, 14400, 19200, 38400, 57600, 115200 
                         SerialPort1.Open()
                         SerialPort1.DtrEnable = True
                         Dim mess As String
                         Dim mTX As Integer
                         Threading.Thread.Sleep(250)
                         Const sendCT As Integer = 400
                         For mTX = 1 To sendCT
                             mess = String.Format("{1}{0}FOO {1}{0}{1,4} Lorem ipsum dolor sit amet, consectetur adipiscing...{0}",
                                                    ControlChars.Tab, mTX)
                             SerialPort1.Write(mess)
                             SerialPort1.Write(ControlChars.CrLf)
                         Next

                         Threading.Thread.Sleep(sendCT \ 100 * 1000 + 500) 'give a chance for all messages to be processed

                         Me.BeginInvoke(Sub()
                                            Label1.Text = _DataReceivedEvents.ToString("n0")
                                            Label2.Text = _messagesRcvd.ToString("n0")
                                        End Sub)
                     End Sub)
    End Sub

    Private _ErrorReceivedEvents As Long = 0L
    Private Sub SerialPort1_ErrorReceived(sender As Object,
                                          e As IO.Ports.SerialErrorReceivedEventArgs) Handles SerialPort1.ErrorReceived
        Threading.Interlocked.Increment(_ErrorReceivedEvents)
        Dim err As Long = Threading.Interlocked.Read(_ErrorReceivedEvents)
        Debug.Write(err.ToString("n0"))
        Debug.Write("  ERR  ")
        Debug.WriteLine(e.EventType.ToString)
        ' Stop
    End Sub

    Private _DataReceivedEvents As Integer = 0 'typically a lot more _DataReceivedEvents than of _messagesRcvd
    Private ReadWait As New Threading.AutoResetEvent(False)
    Private Sub SerialPort1_DataReceived(sender As Object,
                                         e As IO.Ports.SerialDataReceivedEventArgs) Handles SerialPort1.DataReceived
        'this is the entire DataReceived event handler
        Threading.Interlocked.Increment(_DataReceivedEvents)
        ReadWait.Set() 'cause ReadSerialPort to run
    End Sub

    Private _messagesRcvd As Integer = 0
    Private Sub ReadSerialPort()
        '
        'this is run as a Background task
        '
        Dim data As New System.Text.StringBuilder(1024 * 1024) 'accumulate SerialPort data here
        Dim mess As New System.Text.StringBuilder(1024 * 2) 'individual message
        Dim foundMess As Boolean = False
        Do
            ReadWait.WaitOne() 'wait for SerialPort1.DataReceived

            Do While SerialPort1.BytesToRead > 0 'accumulate all available bytes
                data.Append(SerialPort1.ReadExisting)
            Loop
            Do 'process message, if one available
                '  protocol
                '   in this case looking for a string terminated by CR
                mess.Length = 0 'reset message
                foundMess = False 'not found
                Dim idx As Integer = 0
                For idx = 0 To data.Length - 1
                    'look for CR??????
                    If data(idx) = ControlChars.Cr Then '<<<<<<<<<<<<<<<<<<<<<<<
                        'note that the CR will NOT be in mess
                        foundMess = True 'message found 
                        Threading.Interlocked.Increment(_messagesRcvd)
                        Exit For
                    ElseIf data(idx) = ControlChars.Lf Then '<<<<<<<<<<<<<<<<<<<<<<<
                        Continue For 'skip LF
                    End If
                    mess.Append(data(idx))
                Next
                If foundMess Then
                    data.Remove(0, idx + 1) 'remove found message from data
                    '
                    'process the message here <<<<<<<<<<<<
                    'note how UI interaction is handled in example
                    '

                    'example MY message has parts separated by tab
                    Dim parts() As String
                    parts = mess.ToString.Split(ControlChars.Tab)
                    '  interact with the UI
                    Me.Invoke(Sub()
                                  Label1.Text = parts(0)
                                  If parts.Length >= 3 Then TextBox1.Text = parts(2)
                                  RichTextBox1.AppendText(parts(1))
                                  RichTextBox1.AppendText(ControlChars.Cr)
                                  RichTextBox1.ScrollToCaret()
                              End Sub)
                    'end example
                End If

                'look for more messages in data?
            Loop While foundMess AndAlso data.Length > 0
        Loop
    End Sub
End Class

At the end of this post https://www.vbforums.com/showthread....=1#post3279031 there is another alternative to how this might be done. It may suit your needs better.

thank you for the brief explaination. I guess I have to do it in a conservative way since too difficult for me. I will just read the single line data and then I rearrange them again into the array I want to display on the video.