-
Jun 17th, 2022, 03:47 AM
#1
Thread Starter
Lively Member
Last edited by Vizier87; Jun 17th, 2022 at 03:54 AM.
-
Jun 17th, 2022, 08:51 AM
#2
Re: Determining the "head" and "tail" of a data stream
This is code that I use as a starting point for serial port projects. It is very good at reading in bytes. You could add code after
If AllDataQ.Count > 0 Then
The way you've done it the serialport is waiting on the UI. IMO the serial port should drive the UI.
Code:
Imports System.IO.Ports
Public Class Form1
Private WithEvents TheSP As New IO.Ports.SerialPort
Private HaveData As New Threading.AutoResetEvent(False)
Private InData As New Concurrent.ConcurrentQueue(Of List(Of Byte))
Private ProcDataTask As Task
Private Sub Form1_Load(sender As Object, e As EventArgs) Handles Me.Load
ProcDataTask = Task.Run(Sub() ProcData()) 'start a background thread to process data
End Sub
Private Sub Form1_Shown(sender As Object, e As EventArgs) Handles Me.Shown
OpenSP()
End Sub
Private Sub Form1_FormClosing(sender As Object, e As FormClosingEventArgs) Handles Me.FormClosing
Try
If TheSP.IsOpen Then
TheSP.Close()
End If
TheSP.Dispose()
Catch ex As Exception
Stop
End Try
End Sub
Private Sub _DataReceived(sender As Object,
e As IO.Ports.SerialDataReceivedEventArgs) Handles TheSP.DataReceived
'do as little as possible in the handler
' .BytesToRead may not be what you expect
Dim readBuf(TheSP.BytesToRead - 1) As Byte 'buffer for bytes
Try
'.Read returns how many read
Dim br As Integer = TheSP.Read(readBuf, 0, readBuf.Length) 'read bytes
'queue up the data received
' Debug.WriteLine("< " & br)
InData.Enqueue(readBuf.Take(br).ToList)
HaveData.Set() ' Signal ProcData
'
If br <> readBuf.Length Then
'todo Didn't read all available
'never observed this happening
Stop
End If
Catch ex As Exception
'todo
End Try
End Sub
Private AllDataQ As New List(Of Byte)
Private Sub ProcData()
'process the data
Do
HaveData.WaitOne() 'wait for HaveData.Set in Handler
'Debug.WriteLine("")
'process the queue
Dim OneBuf As List(Of Byte)
While Not InData.IsEmpty
If InData.TryDequeue(OneBuf) Then 'get some data
AllDataQ.AddRange(OneBuf)
Else
HaveData.WaitOne(1) 'Dequeue failed
End If
End While
'>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
' at this point the protocol of the device is
' enforced.
'>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
If AllDataQ.Count > 0 Then
'whatever processing goes on here
' should be emptying AllDataQ
' Debug.WriteLine("> " & AllDataQ.Count)
AllDataQ.Clear()
End If
Loop
End Sub
Private Sub OpenSP()
Dim t As Task
t = Task.Run(Sub()
If Not TheSP.IsOpen Then
'Modify settings as needed <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
TheSP.PortName = "COM1"
TheSP.BaudRate = 115200 ' 19200 '
TheSP.DataBits = 8
TheSP.Parity = Parity.None
TheSP.StopBits = StopBits.One
'other settings as needed - consider speed and max size to determine settings <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
TheSP.ReceivedBytesThreshold = 1 'one is the default, recommend no change unless absolutely needed
TheSP.ReadTimeout = 1000 'default is infinite if not set
TheSP.WriteTimeout = 1000 'default is infinite if not set
TheSP.ReadBufferSize = 1024 * 4 'Windows-created input buffer 4096 is default, change if needed
TheSP.WriteBufferSize = 1024 * 2 'Windows-created output buffer 2048 is default, change if needed
'this setting is informational only. the code only reads bytes <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
'if the device is sending strings recommend setting this to match encoding device is using <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
'if the application is writing strings this must match devices encoding <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
TheSP.Encoding = System.Text.Encoding.GetEncoding(28591) 'default is 7 bit ascii, this is an 8 bit flavor of asciii
Try
TheSP.Open()
'some devices require the following
TheSP.DtrEnable = True 'only after Open. may cause multiple PinChanged events
'SerialPort Open
Catch ex As Exception
Debug.WriteLine(ex.Message)
End Try
End If
End Sub)
End Sub
'Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
' Dim b(4095) As Byte
' TheSP.Write(b, 0, b.Length)
'End Sub
End Class
-
Jun 20th, 2022, 04:08 AM
#3
Thread Starter
Lively Member
Re: Determining the "head" and "tail" of a data stream
Hi dBassNet, thanks for the snippet.
I've done some minor tweaks to the code and added a handler.
What do you think of this:
Code:
Imports System
Imports System.IO
Imports System.IO.Ports
Imports System.Threading
Imports System.Text
Imports System.Windows.Forms.DataVisualization.Charting
Public Class Form1
Private WithEvents TheSP As New IO.Ports.SerialPort
Private HaveData As New Threading.AutoResetEvent(False)
Private InData As New Concurrent.ConcurrentQueue(Of List(Of Byte))
Private ProcDataTask As Task
Dim DT1 As DataTable
Dim dr As DataRow
Private Sub Form1_Load(sender As Object, e As EventArgs) Handles Me.Load
ProcDataTask = Task.Run(Sub() ProcData()) 'start a background thread to process data
End Sub
Private Sub Form1_Shown(sender As Object, e As EventArgs) Handles Me.Shown
OpenSP()
End Sub
Private Sub Form1_FormClosing(sender As Object, e As FormClosingEventArgs) Handles Me.FormClosing
Try
If TheSP.IsOpen Then
TheSP.Close()
End If
TheSP.Dispose()
Catch ex As Exception
Stop
End Try
End Sub
Private Sub _DataReceived(sender As Object,
e As IO.Ports.SerialDataReceivedEventArgs) Handles TheSP.DataReceived
'do as little as possible in the handler
' .BytesToRead may not be what you expect
Dim readBuf(TheSP.BytesToRead - 1) As Byte 'buffer for bytes
Try
'.Read returns how many read
Dim br As Integer = TheSP.Read(readBuf, 0, readBuf.Length) 'read bytes
'queue up the data received
' Debug.WriteLine("< " & br)
InData.Enqueue(readBuf.Take(br).ToList)
HaveData.Set() ' Signal ProcData
If br <> readBuf.Length Then
'todo Didn't read all available
'never observed this happening
Stop
End If
Catch ex As Exception
'todo
End Try
End Sub
Private AllDataQ As New List(Of Byte)
Private Sub ProcData()
' On Error Resume Next
'process the data
Do
HaveData.WaitOne() 'wait for HaveData.Set in Handler
'Debug.WriteLine("")
'process the queue
Dim OneBuf As List(Of Byte)
While Not InData.IsEmpty
If InData.TryDequeue(OneBuf) Then 'get some data
AllDataQ.AddRange(OneBuf)
Else
HaveData.WaitOne(1) 'Dequeue failed
End If
End While
'>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
' at this point the protocol of the device is
' enforced.
'>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
If AllDataQ.Count > 0 Then
'whatever processing goes on here
' should be emptying AllDataQ
' Debug.WriteLine("> " & AllDataQ.Count)
Dim ReceivedData(OneBuf.Count) As Byte
For a = 0 To OneBuf.Count - 1
ReceivedData(a) = OneBuf(a)
Next
UpdateGrid(ReceivedData)
AllDataQ.Clear()
End If
Loop
End Sub
Private Sub OpenSP()
Dim t As Task
t = Task.Run(Sub()
If Not TheSP.IsOpen Then
'Modify settings as needed <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
TheSP.PortName = "COM14"
TheSP.BaudRate = 57600 ' 19200 '
TheSP.DataBits = 8
TheSP.Parity = Parity.None
TheSP.StopBits = StopBits.One
'other settings as needed - consider speed and max size to determine settings <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
TheSP.ReceivedBytesThreshold = 1 'one is the default, recommend no change unless absolutely needed
TheSP.ReadTimeout = 1000 'default is infinite if not set
TheSP.WriteTimeout = 1000 'default is infinite if not set
TheSP.ReadBufferSize = 1024 * 4 'Windows-created input buffer 4096 is default, change if needed
TheSP.WriteBufferSize = 1024 * 2 'Windows-created output buffer 2048 is default, change if needed
'this setting is informational only. the code only reads bytes <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
'if the device is sending strings recommend setting this to match encoding device is using <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
'if the application is writing strings this must match devices encoding <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
TheSP.Encoding = System.Text.Encoding.GetEncoding(28591) 'default is 7 bit ascii, this is an 8 bit flavor of asciii
Try
TheSP.Open()
'some devices require the following
TheSP.DtrEnable = True 'only after Open. may cause multiple PinChanged events
'SerialPort Open
Catch ex As Exception
Debug.WriteLine(ex.Message)
End Try
End If
End Sub)
End Sub
Private Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
Dim b(4095) As Byte
TheSP.Write(b, 0, b.Length)
End Sub
Public Function CreateDataTable() As DataTable
Dim dt As New DataTable
dt.Columns.Add("Wavelength")
dt.Columns.Add("%")
Return dt
End Function
Dim cycle = 0
Public Sub UpdateGrid(ByVal bytes_Input As Byte())
Dim Stopwatch As Stopwatch = Stopwatch.StartNew()
If Me.InvokeRequired Then
Me.Invoke(New Action(Of Byte())(AddressOf UpdateGrid), bytes_Input)
Else
Dim y = 1
DT1 = CreateDataTable()
For Each value In bytes_Input
dr = DT1.NewRow()
dr("Wavelength") = y
dr("%") = value
DT1.Rows.Add(dr)
y += 1
Next
DataGridView1.DataSource = DT1
'//////////////////////////////////////////////////////////////////////////
'//////////////////////////////////////////////////////////////////////////
Application.DoEvents()
End If
End Sub
End Class
Still, I'm trying to squeeze as much data from my Serial Port and still only receiving around 60 bytes. The transmitted bytes are 4000 by the way.
My previous attempt at squeezing the data out is by reading multiple cycles to obtain as much bytes as I could, but that ends up in the problem summarized in the video I shared in the first post.
Any pointers? I greatly appreciate them. Thanks again.
Vizier87
-
Jun 20th, 2022, 08:18 AM
#4
Re: Determining the "head" and "tail" of a data stream
There is a mistake starting at the comment. When you get to
If AllDataQ.Count > 0 Then
AllDataQ will contain all of the bytes received up until that point. OneBuf ONLY has the last item from the queue.
Changes below.
Code:
'>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
' at this point the protocol of the device is
' enforced.
'>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
If AllDataQ.Count > 0 Then
'whatever processing goes on here
' should be emptying AllDataQ
' Debug.WriteLine("> " & AllDataQ.Count)
Me.BeginInvoke(Sub()
UpdateGrid(AllDataQ.ToArray)
End Sub)
AllDataQ.Clear()
End If
Last edited by dbasnett; Jun 20th, 2022 at 08:28 AM.
-
Jun 27th, 2022, 03:24 AM
#5
Thread Starter
Lively Member
Re: Determining the "head" and "tail" of a data stream
Hi dbasnet,
I've tried your code, and it partially works.
Is there a way I can successfully capture 4000 bytes in a single packet?
(By the way, the ToArray method is super helpful, big thanks for that)
I've only managed to capture 70-100 bytes per cycle.
I've changed the COM port settings to TheSP.ReadBufferSize = 4000 .
And the code for the ProcData is slightly changed because Me.BeginInvoke(Sub() didn't work for me:
Code:
Private Sub ProcData()
' On Error Resume Next
'process the data
Do
HaveData.WaitOne() 'wait for HaveData.Set in Handler
'Debug.WriteLine("")
'process the queue
Dim OneBuf As List(Of Byte)
While Not InData.IsEmpty
If InData.TryDequeue(OneBuf) Then 'get some data
AllDataQ.AddRange(OneBuf)
Else
HaveData.WaitOne(1) 'Dequeue failed
End If
End While
If AllDataQ.Count > 0 Then
UpdateGrid(AllDataQ.ToArray)
AllDataQ.Clear()
End If
Loop
End Sub
Here's what it looks like for now:
My Transmission side is just a simple Arduino transmission:
Code:
for(unsigned int n = 0; n<4000; n++)
{
Serial.write(buffer[n]);
delayMicroseconds(100);
}
Any suggestions for me to go around this?
Edit: Even when I'm receiving only 20 bytes, which is easier since I can change If AllDataQ.Count > 0 Then to If AllDataQ.Count = 20 Then, there is this "streaming" effect like shown in the first video.
And setting If AllDataQ.Count = 4000 Then is not possible since I couldn't get that many bytes per cycle. Sigh.
Thanks!
Last edited by Vizier87; Jun 27th, 2022 at 03:39 AM.
-
Jun 27th, 2022, 08:26 AM
#6
Re: Determining the "head" and "tail" of a data stream
First of all do NOT change this
TheSP.ReadBufferSize
Take the default.
I don't know what you mean by cycle.
Looking at the code you provided this
Code:
If AllDataQ.Count >= 4000 Then
should work.
-
Jun 27th, 2022, 09:06 AM
#7
Re: Determining the "head" and "tail" of a data stream
Even with those changes, it seems to me you may still drop data.
Your baud rate is only 57600 and you generally divide baud rate by 10 to get the approximate number of bytes of bandwidth you have for the interface, which would be 5760 bytes per second in this case.
Your Arduino code is setup to output the bytes at a rate of 10000 bytes per second, so you are probably overrunning your serial buffer on the transmit side and loosing data.
If you want to transmit at a 10000 bytes per second rate you'll need to bump your baud rate up to the next level, i.e. 115200, otherwise put in a longer delay in your Arduino loop, i.e. 200us will give you a 5000 byte per second rate which would fit in the bandwidth you have available at 57600 baud.
"Anyone can do any amount of work, provided it isn't the work he is supposed to be doing at that moment" Robert Benchley, 1930
-
Jun 27th, 2022, 10:25 PM
#8
Thread Starter
Lively Member
Re: Determining the "head" and "tail" of a data stream
Originally Posted by dbasnett
First of all do NOT change this
TheSP.ReadBufferSize
Take the default.
I don't know what you mean by cycle.
Looking at the code you provided this
Code:
If AllDataQ.Count >= 4000 Then
should work.
Thanks dbasnet. Your solution worked very well.
A cycle meant the 4 kB data from Buffer[0] to Buffer[3999] essentially from the Arduino side. Each byte is sent with a 250 us delay.
Is there a way I can check for the incoming data for a preamble before starting to save the data? On my Arduino side, it set it to sending 7 bytes of "255". But playing around with OneBuf only allows me to check it in the end rather than the beginning. It seems most of the time, I get the AllDataQ somewhere in the middle of the "cycle", if I may.
Originally Posted by passel
Even with those changes, it seems to me you may still drop data.
Your baud rate is only 57600 and you generally divide baud rate by 10 to get the approximate number of bytes of bandwidth you have for the interface, which would be 5760 bytes per second in this case.
Your Arduino code is setup to output the bytes at a rate of 10000 bytes per second, so you are probably overrunning your serial buffer on the transmit side and loosing data.
If you want to transmit at a 10000 bytes per second rate you'll need to bump your baud rate up to the next level, i.e. 115200, otherwise put in a longer delay in your Arduino loop, i.e. 200us will give you a 5000 byte per second rate which would fit in the bandwidth you have available at 57600 baud.
A red-faced me was just looking up on how baud rates and Bytes/second correlates with each other. Thanks, now I see how the dissonance between the devices affect the data quality. I've set the Arduino delay to 250 us while also setting the baud rates to 115200. It works better, but still having the "streaming" issue.
Here's the latest feed, but I've added a "preamble" if you like but I'm still figuring how to sorta "wait" for the preamble so that the data streaming can be stabilized without any "motion". Any pointers?
Thanks!
Last edited by Vizier87; Jun 27th, 2022 at 11:05 PM.
-
Jun 28th, 2022, 09:26 AM
#9
Re: Determining the "head" and "tail" of a data stream
'Is there a way I can check for the incoming data for a preamble before starting to save the data? On my Arduino side, it set it to sending 7 bytes of "255". But playing around with OneBuf only allows me to check it in the end rather than the beginning. It seems most of the time, I get the AllDataQ somewhere in the middle of the "cycle", if I may.'
I think I've said this before, OneBuf shouldn't be used for anything other than as specified!
This code will find the preamble before processing(UpdateGrid). I would suggest sending more than 7 preamble bytes WITHOUT changing the code below.
I have put stops so you can see what is going on.
Code:
'possible 4000 data bytes + 7 preamble bytes
Const DataSize As Integer = 4000
Const PreambleSize As Integer = 7
If AllDataQ.Count >= (DataSize + PreambleSize) Then
Dim f255 As Integer 'first 255
f255 = AllDataQ.IndexOf(255)
If f255 >= 0 Then
Dim d255 As Integer = f255
Do While AllDataQ(d255) = 255 'find end of 255's
d255 += 1
Loop
Stop
If d255 >= PreambleSize Then
'remove bytes before preamble AND preamble
AllDataQ.RemoveRange(0, d255)
Stop
If AllDataQ.Count >= DataSize Then
'we have 4000 data bytes
'process
Dim a() As Byte = AllDataQ.Take(DataSize).ToArray
Stop
AllDataQ.RemoveRange(0, DataSize)
'UpdateGrid(a)
Else
'do not have 4000 data bytes
're-insert the preamble
AllDataQ.InsertRange(0, {255, 255, 255, 255, 255, 255, 255})
End If
End If
End If
End If
Last edited by dbasnett; Jun 28th, 2022 at 09:32 AM.
-
Jun 29th, 2022, 02:24 AM
#10
Thread Starter
Lively Member
Re: Determining the "head" and "tail" of a data stream
Hi dbasnett, you're a lifesaver.
The problem was solved, because by controlling the incoming bytes as per the code you mentioned, I was able to control the stream of data, make heads and tails out of it by buffering it in two lists, then reconstructed the packet frame.
I'll be trying out the preamble code soon too, but many thanks! I'll update soon.
Cheers,
Vizier87
Posting Permissions
- You may not post new threads
- You may not post replies
- You may not post attachments
- You may not edit your posts
-
Forum Rules
|
Click Here to Expand Forum to Full Width
|