[RESOLVED] Determining the "head" and "tail" of a data stream

[Vizier87]

Hi guys, I've been trying to log a hefty number of bytes from a data logger via Xbee to the serial port.

Here's the video of the stream:


Now, the problem with stabilizing the feed is because the number of bytes received per "packet", if you will, is different. This is because the wireless data has some losses.

You can see the number of bytes on one of the labels (beside the cutoff byte ProgressBar.. sorry lousy placement)

I can see that there is some periodicity to the stream but I'm not sure how I can make the profile relatively stable in the graph ie not streaming like this.

Here's my current code for updating the :

Public Sub DataReceived(ByVal sender As Object, ByVal e As SerialDataReceivedEventArgs)
        
        Dim ReceiveBuf(1000) As Byte
        myComPort.Read(ReceiveBuf, 0, 1000)
        UpdateGrid(ReceiveBuf)

    End Sub




Public Sub Lister(ByRef Byte_Array As Byte())

        For Each value In Byte_Array
            If value = Nothing Then Exit For
            List1.Add(CInt(value))
        Next

        Thread.Sleep(90)
    End Sub



    Public Sub UpdateGrid(ByVal bytes_Input As Byte())



        Dim Stopwatch As Stopwatch = Stopwatch.StartNew()
        If Me.InvokeRequired Then
            Me.Invoke(New Action(Of Byte())(AddressOf UpdateGrid), bytes_Input)
        Else

            Select Case cycle
                Case 1

                    label_originalpacket.Text = bytes_Input.Count

                    List1 = New List(Of Integer)

                    Lister(bytes_Input)

                Case 2
                    Lister(bytes_Input)
                Case 3
                    Lister(bytes_Input)
                Case 4
                    Lister(bytes_Input)
                Case 5
                    Lister(bytes_Input)
                Case 6
                    Lister(bytes_Input)
                Case 7
                    Lister(bytes_Input)

                Case 8

                    Lister(bytes_Input)

                    Label3_total.Text = List1.Count



                    cycle = 0




                    If List1.Count > 1700 And List1.Count < 1850 Then

                        '////////////////////////////////////////////////////////////////////
                        'Process list's cutoffs
                        For Each value In List1
                            Dim beforeValue = value
                        Next


                        'end lim





                        Dim y = 1
                        DT1 = CreateDataTable()
                        For Each value In List1
                            dr = DT1.NewRow()
                            dr("Wavelength") = y
                            dr("%") = value
                            DT1.Rows.Add(dr)
                            y += 1
                        Next
                        Charter(Chart1, DT1)




                    End If


            End Select

            cycle += 1


            Stopwatch.Stop()
            Label6.Text = Stopwatch.ElapsedMilliseconds & " ms"



            '//////////////////////////////////////////////////////////////////////////
            '//////////////////////////////////////////////////////////////////////////

            Application.DoEvents()

        End If


    End Sub

I'd appreciate any input on my problem. Thanks!

Vizier87

[dbasnett]

This is code that I use as a starting point for serial port projects. It is very good at reading in bytes. You could add code after
If AllDataQ.Count > 0 Then
The way you've done it the serialport is waiting on the UI. IMO the serial port should drive the UI.

Imports System.IO.Ports

Public Class Form1

    Private WithEvents TheSP As New IO.Ports.SerialPort
    Private HaveData As New Threading.AutoResetEvent(False)
    Private InData As New Concurrent.ConcurrentQueue(Of List(Of Byte))
    Private ProcDataTask As Task

    Private Sub Form1_Load(sender As Object, e As EventArgs) Handles Me.Load
        ProcDataTask = Task.Run(Sub() ProcData()) 'start a background thread to process data
    End Sub

    Private Sub Form1_Shown(sender As Object, e As EventArgs) Handles Me.Shown
        OpenSP()
    End Sub

    Private Sub Form1_FormClosing(sender As Object, e As FormClosingEventArgs) Handles Me.FormClosing
        Try
            If TheSP.IsOpen Then
                TheSP.Close()
            End If
            TheSP.Dispose()
        Catch ex As Exception
            Stop
        End Try
    End Sub

    Private Sub _DataReceived(sender As Object,
                                         e As IO.Ports.SerialDataReceivedEventArgs) Handles TheSP.DataReceived
        'do as little as possible in the handler
        ' .BytesToRead may not be what you expect
        Dim readBuf(TheSP.BytesToRead - 1) As Byte 'buffer for bytes
        Try
            '.Read returns how many read
            Dim br As Integer = TheSP.Read(readBuf, 0, readBuf.Length) 'read bytes
            'queue up the data received
            ' Debug.WriteLine("< " & br)
            InData.Enqueue(readBuf.Take(br).ToList)
            HaveData.Set() ' Signal ProcData
            '
            If br <> readBuf.Length Then
                'todo Didn't read all available
                'never observed this happening
                Stop
            End If
        Catch ex As Exception
            'todo
        End Try
    End Sub

    Private AllDataQ As New List(Of Byte)
    Private Sub ProcData()
        'process the data
        Do
            HaveData.WaitOne() 'wait for HaveData.Set in Handler
            'Debug.WriteLine("")
            'process the queue
            Dim OneBuf As List(Of Byte)
            While Not InData.IsEmpty
                If InData.TryDequeue(OneBuf) Then 'get some data
                    AllDataQ.AddRange(OneBuf)
                Else
                    HaveData.WaitOne(1) 'Dequeue failed
                End If
            End While

            '>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
            ' at this point the protocol of the device is
            '  enforced. 
            '>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
            If AllDataQ.Count > 0 Then
                'whatever processing goes on here
                '  should be emptying AllDataQ
                ' Debug.WriteLine("> " & AllDataQ.Count)
                AllDataQ.Clear()
            End If
        Loop
    End Sub

    Private Sub OpenSP()
        Dim t As Task
        t = Task.Run(Sub()
                         If Not TheSP.IsOpen Then
                             'Modify settings as needed <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                             TheSP.PortName = "COM1"
                             TheSP.BaudRate = 115200 '  19200 '  
                             TheSP.DataBits = 8
                             TheSP.Parity = Parity.None
                             TheSP.StopBits = StopBits.One

                             'other settings as needed - consider speed and max size to determine settings  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                             TheSP.ReceivedBytesThreshold = 1 'one is the default, recommend no change unless absolutely needed
                             TheSP.ReadTimeout = 1000 'default is infinite if not set
                             TheSP.WriteTimeout = 1000 'default is infinite if not set
                             TheSP.ReadBufferSize = 1024 * 4 'Windows-created input buffer 4096 is default, change if needed
                             TheSP.WriteBufferSize = 1024 * 2  'Windows-created output buffer 2048 is default, change if needed

                             'this setting is informational only.  the code only reads bytes  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                             'if the device is sending strings recommend setting this to match encoding device is using  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                             'if the application is writing strings this must match devices encoding                     <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                             TheSP.Encoding = System.Text.Encoding.GetEncoding(28591) 'default is 7 bit ascii, this is an 8 bit flavor of asciii
                             Try
                                 TheSP.Open()
                                 'some devices require the following
                                 TheSP.DtrEnable = True 'only after Open. may cause multiple PinChanged events
                                 'SerialPort Open 
                             Catch ex As Exception
                                 Debug.WriteLine(ex.Message)
                             End Try
                         End If
                     End Sub)
    End Sub

    'Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
    '    Dim b(4095) As Byte
    '    TheSP.Write(b, 0, b.Length)
    'End Sub
End Class

[Vizier87]

Hi dBassNet, thanks for the snippet.

I've done some minor tweaks to the code and added a handler.

What do you think of this:

Imports System
Imports System.IO
Imports System.IO.Ports
Imports System.Threading
Imports System.Text
Imports System.Windows.Forms.DataVisualization.Charting

Public Class Form1
    Private WithEvents TheSP As New IO.Ports.SerialPort
    Private HaveData As New Threading.AutoResetEvent(False)
    Private InData As New Concurrent.ConcurrentQueue(Of List(Of Byte))
    Private ProcDataTask As Task


    Dim DT1 As DataTable
    Dim dr As DataRow


    Private Sub Form1_Load(sender As Object, e As EventArgs) Handles Me.Load
        ProcDataTask = Task.Run(Sub() ProcData()) 'start a background thread to process data
    End Sub

    Private Sub Form1_Shown(sender As Object, e As EventArgs) Handles Me.Shown
        OpenSP()
    End Sub

    Private Sub Form1_FormClosing(sender As Object, e As FormClosingEventArgs) Handles Me.FormClosing
        Try
            If TheSP.IsOpen Then
                TheSP.Close()
            End If
            TheSP.Dispose()
        Catch ex As Exception
            Stop
        End Try
    End Sub

    Private Sub _DataReceived(sender As Object,
                                         e As IO.Ports.SerialDataReceivedEventArgs) Handles TheSP.DataReceived
        'do as little as possible in the handler
        ' .BytesToRead may not be what you expect
        Dim readBuf(TheSP.BytesToRead - 1) As Byte 'buffer for bytes
        Try
            '.Read returns how many read
            Dim br As Integer = TheSP.Read(readBuf, 0, readBuf.Length) 'read bytes
            'queue up the data received
            ' Debug.WriteLine("< " & br)
            InData.Enqueue(readBuf.Take(br).ToList)
            HaveData.Set() ' Signal ProcData

            If br <> readBuf.Length Then
                'todo Didn't read all available
                'never observed this happening
                Stop
            End If
        Catch ex As Exception
            'todo
        End Try
    End Sub




    Private AllDataQ As New List(Of Byte)
    Private Sub ProcData()

        ' On Error Resume Next

        'process the data
        Do
            HaveData.WaitOne() 'wait for HaveData.Set in Handler
            'Debug.WriteLine("")
            'process the queue
            Dim OneBuf As List(Of Byte)
            While Not InData.IsEmpty
                If InData.TryDequeue(OneBuf) Then 'get some data
                    AllDataQ.AddRange(OneBuf)
                Else
                    HaveData.WaitOne(1) 'Dequeue failed
                End If
            End While

            '>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
            ' at this point the protocol of the device is
            '  enforced. 
            '>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
            If AllDataQ.Count > 0 Then
                'whatever processing goes on here
                '  should be emptying AllDataQ
                ' Debug.WriteLine("> " & AllDataQ.Count)
                Dim ReceivedData(OneBuf.Count) As Byte
                For a = 0 To OneBuf.Count - 1
                    ReceivedData(a) = OneBuf(a)
                Next
                UpdateGrid(ReceivedData)

                AllDataQ.Clear()
            End If
        Loop
    End Sub

    Private Sub OpenSP()
        Dim t As Task
        t = Task.Run(Sub()
                         If Not TheSP.IsOpen Then
                             'Modify settings as needed <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                             TheSP.PortName = "COM14"
                             TheSP.BaudRate = 57600 '  19200 '  
                             TheSP.DataBits = 8
                             TheSP.Parity = Parity.None
                             TheSP.StopBits = StopBits.One

                             'other settings as needed - consider speed and max size to determine settings  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                             TheSP.ReceivedBytesThreshold = 1 'one is the default, recommend no change unless absolutely needed
                             TheSP.ReadTimeout = 1000 'default is infinite if not set
                             TheSP.WriteTimeout = 1000 'default is infinite if not set
                             TheSP.ReadBufferSize = 1024 * 4 'Windows-created input buffer 4096 is default, change if needed
                             TheSP.WriteBufferSize = 1024 * 2  'Windows-created output buffer 2048 is default, change if needed

                             'this setting is informational only.  the code only reads bytes  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                             'if the device is sending strings recommend setting this to match encoding device is using  <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                             'if the application is writing strings this must match devices encoding                     <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
                             TheSP.Encoding = System.Text.Encoding.GetEncoding(28591) 'default is 7 bit ascii, this is an 8 bit flavor of asciii
                             Try
                                 TheSP.Open()
                                 'some devices require the following
                                 TheSP.DtrEnable = True 'only after Open. may cause multiple PinChanged events
                                 'SerialPort Open 
                             Catch ex As Exception
                                 Debug.WriteLine(ex.Message)
                             End Try
                         End If
                     End Sub)
    End Sub

    Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
        Dim b(4095) As Byte
        TheSP.Write(b, 0, b.Length)


    End Sub


    Public Function CreateDataTable() As DataTable
        Dim dt As New DataTable
        dt.Columns.Add("Wavelength")
        dt.Columns.Add("%")
        Return dt
    End Function

    Dim cycle = 0
    Public Sub UpdateGrid(ByVal bytes_Input As Byte())

        Dim Stopwatch As Stopwatch = Stopwatch.StartNew()
        If Me.InvokeRequired Then
            Me.Invoke(New Action(Of Byte())(AddressOf UpdateGrid), bytes_Input)
        Else


            Dim y = 1
            DT1 = CreateDataTable()
            For Each value In bytes_Input
                dr = DT1.NewRow()
                dr("Wavelength") = y
                dr("%") = value
                DT1.Rows.Add(dr)
                y += 1
            Next

            DataGridView1.DataSource = DT1

            '//////////////////////////////////////////////////////////////////////////
            '//////////////////////////////////////////////////////////////////////////

            Application.DoEvents()

        End If


    End Sub




End Class

Still, I'm trying to squeeze as much data from my Serial Port and still only receiving around 60 bytes. The transmitted bytes are 4000 by the way.

My previous attempt at squeezing the data out is by reading multiple cycles to obtain as much bytes as I could, but that ends up in the problem summarized in the video I shared in the first post.

Any pointers? I greatly appreciate them. Thanks again.

Vizier87

[dbasnett]

There is a mistake starting at the comment. When you get to
If AllDataQ.Count > 0 Then
AllDataQ will contain all of the bytes received up until that point. OneBuf ONLY has the last item from the queue.

Changes below.

            '>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
            ' at this point the protocol of the device is
            '  enforced. 
            '>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
            If AllDataQ.Count > 0 Then
                'whatever processing goes on here
                '  should be emptying AllDataQ
                ' Debug.WriteLine("> " & AllDataQ.Count)
                Me.BeginInvoke(Sub()
                                   UpdateGrid(AllDataQ.ToArray)
                               End Sub)
                AllDataQ.Clear()
            End If

[Vizier87]

Hi dbasnet,

I've tried your code, and it partially works.

Is there a way I can successfully capture 4000 bytes in a single packet?

(By the way, the ToArray method is super helpful, big thanks for that)

I've only managed to capture 70-100 bytes per cycle.

I've changed the COM port settings to TheSP.ReadBufferSize = 4000 .

And the code for the ProcData is slightly changed because Me.BeginInvoke(Sub() didn't work for me:

Private Sub ProcData()

        ' On Error Resume Next

        'process the data
        Do
            HaveData.WaitOne() 'wait for HaveData.Set in Handler
            'Debug.WriteLine("")
            'process the queue
            Dim OneBuf As List(Of Byte)
            While Not InData.IsEmpty
                If InData.TryDequeue(OneBuf) Then 'get some data
                    AllDataQ.AddRange(OneBuf)
                Else
                    HaveData.WaitOne(1) 'Dequeue failed
                End If
            End While

            If AllDataQ.Count > 0 Then
                UpdateGrid(AllDataQ.ToArray)
                AllDataQ.Clear()
            End If

        Loop
    End Sub

Here's what it looks like for now:



My Transmission side is just a simple Arduino transmission:

for(unsigned int n = 0; n<4000; n++)
  {
    Serial.write(buffer[n]);
    delayMicroseconds(100);
  }

Any suggestions for me to go around this?

Edit: Even when I'm receiving only 20 bytes, which is easier since I can change If AllDataQ.Count > 0 Then to If AllDataQ.Count = 20 Then, there is this "streaming" effect like shown in the first video.

And setting If AllDataQ.Count = 4000 Then is not possible since I couldn't get that many bytes per cycle. Sigh.

Thanks!

[dbasnett]

First of all do NOT change this
TheSP.ReadBufferSize
Take the default.

I don't know what you mean by cycle.

Looking at the code you provided this

If AllDataQ.Count >= 4000 Then

should work.

[passel]

Even with those changes, it seems to me you may still drop data.
Your baud rate is only 57600 and you generally divide baud rate by 10 to get the approximate number of bytes of bandwidth you have for the interface, which would be 5760 bytes per second in this case.
Your Arduino code is setup to output the bytes at a rate of 10000 bytes per second, so you are probably overrunning your serial buffer on the transmit side and loosing data.
If you want to transmit at a 10000 bytes per second rate you'll need to bump your baud rate up to the next level, i.e. 115200, otherwise put in a longer delay in your Arduino loop, i.e. 200us will give you a 5000 byte per second rate which would fit in the bandwidth you have available at 57600 baud.

[Vizier87]

First of all do NOT change this
TheSP.ReadBufferSize
Take the default.

I don't know what you mean by cycle.

Looking at the code you provided this

If AllDataQ.Count >= 4000 Then

should work.

Thanks dbasnet. Your solution worked very well.

A cycle meant the 4 kB data from Buffer[0] to Buffer[3999] essentially from the Arduino side. Each byte is sent with a 250 us delay.

Is there a way I can check for the incoming data for a preamble before starting to save the data? On my Arduino side, it set it to sending 7 bytes of "255". But playing around with OneBuf only allows me to check it in the end rather than the beginning. It seems most of the time, I get the AllDataQ somewhere in the middle of the "cycle", if I may.

Even with those changes, it seems to me you may still drop data.
Your baud rate is only 57600 and you generally divide baud rate by 10 to get the approximate number of bytes of bandwidth you have for the interface, which would be 5760 bytes per second in this case.
Your Arduino code is setup to output the bytes at a rate of 10000 bytes per second, so you are probably overrunning your serial buffer on the transmit side and loosing data.
If you want to transmit at a 10000 bytes per second rate you'll need to bump your baud rate up to the next level, i.e. 115200, otherwise put in a longer delay in your Arduino loop, i.e. 200us will give you a 5000 byte per second rate which would fit in the bandwidth you have available at 57600 baud.

A red-faced me was just looking up on how baud rates and Bytes/second correlates with each other. Thanks, now I see how the dissonance between the devices affect the data quality. I've set the Arduino delay to 250 us while also setting the baud rates to 115200. It works better, but still having the "streaming" issue.

Here's the latest feed, but I've added a "preamble" if you like but I'm still figuring how to sorta "wait" for the preamble so that the data streaming can be stabilized without any "motion". Any pointers?



Thanks!

[dbasnett]

'Is there a way I can check for the incoming data for a preamble before starting to save the data? On my Arduino side, it set it to sending 7 bytes of "255". But playing around with OneBuf only allows me to check it in the end rather than the beginning. It seems most of the time, I get the AllDataQ somewhere in the middle of the "cycle", if I may.'

I think I've said this before, OneBuf shouldn't be used for anything other than as specified!

This code will find the preamble before processing(UpdateGrid). I would suggest sending more than 7 preamble bytes WITHOUT changing the code below.

I have put stops so you can see what is going on.

        'possible 4000 data bytes + 7 preamble bytes
        Const DataSize As Integer =  4000
        Const PreambleSize As Integer = 7
        If AllDataQ.Count >= (DataSize + PreambleSize) Then
            Dim f255 As Integer 'first 255
            f255 = AllDataQ.IndexOf(255)
            If f255 >= 0 Then
                Dim d255 As Integer = f255
                Do While AllDataQ(d255) = 255 'find end of 255's
                    d255 += 1
                Loop
                Stop
                If d255 >= PreambleSize Then
                    'remove bytes before preamble AND preamble
                    AllDataQ.RemoveRange(0, d255)
                    Stop
                    If AllDataQ.Count >= DataSize Then
                        'we have 4000 data bytes
                        'process
                        Dim a() As Byte = AllDataQ.Take(DataSize).ToArray
                        Stop
                        AllDataQ.RemoveRange(0, DataSize)
                        'UpdateGrid(a)
                    Else
                        'do not have 4000 data bytes
                        're-insert the preamble
                        AllDataQ.InsertRange(0, {255, 255, 255, 255, 255, 255, 255})
                    End If
                End If
            End If
        End If

[Vizier87]

Hi dbasnett, you're a lifesaver.

The problem was solved, because by controlling the incoming bytes as per the code you mentioned, I was able to control the stream of data, make heads and tails out of it by buffering it in two lists, then reconstructed the packet frame.

I'll be trying out the preamble code soon too, but many thanks! I'll update soon.

Cheers,
Vizier87