Increasing com port reading efficiency

[detlion1643]

So I just completed my first test of reading data coming through a com port (usb gps, using usb->com driver). However, I'd like to pick some minds about the actual process that may be better than what I am doing.

When I connect to my com port (hardcoded for now), I hook the data received event handler. Inside this method, I did a .ReadExisting() with output going to a listbox. I noticed that the com port is sending data very fast which is awesome and expected. I noticed that the data is not sent in complete lines.

I then made a class level stringbuilder and kept appending the .ReadExisting() strings to the stringbuilder. After each append, I attempted to check the new text of the stringbuilder for "GPGGA" and removed it. If already removed once, and found "GPGGA" again, we have a complete line. I realize this is not a very correct way to test for a complete line... After a complete line is found, I parse the data into a latitude/longitude (with some code help from codeproject). The lat/long code I'm using has splits, converstions to doubles, and more conversions back to strings...

After appending, testing, and parsing, I seem to get results about 1/sec. I know the gps puck sends complete lines more often than that. I also know that by setting a breakpoint after a while, the stringbuilder is appending faster than it's tested/parsed.

I've read a couple places that a good way is to read the incoming received data into a byte buffer and test that, having a separate method to actually do the parsing. I don't really want to have a thread constantly running just waiting to parse data. However, a constantly running thread to do the testing and parsing might not be a bad idea.

The only problem, I don't know how to work with bytes, or the .Read() with arguments for a byte,count,etc. Does anyone have any good leeway for a more efficient way of going about this?

[stanav]

You need to know the protocol that your serial device use... Once you have that, you can determine what the 'record' terminator is and go after that.... It could be a fixed number of bytes, it could be a special character such as EOT... You can set you serial port receive threshold to x bytes, and read the bytes into a buffer. Test if you have collected enough bytes in the buffer to form a record and if you do, process that record, remove only the processed byte from the buffer and leaving the incomplete one untouched. You definitely will need to synclock the buffer so that when you read it, the adding has to wait and vice versa...

[SJWhiteley]

The buffer is the most versatile and efficient way to go.

There's nothing wrong with using a thread - remember, the DataRecieve method runs on a separate thread.

[dbasnett]

Here are some ideas. I don't like the string concatenation but since stringbuilder doesn't have an indexof you would need to use .ToString. The DataReceived event reads the data and adds it to the stream. The checker thread looks for complete sentences and adds them to a list that is processed in a timer that runs every .10 secs. The timer is on the UI so updating controls is easy.

How fast is the GPS connection? 9600, 19200...

    Private Sub SerialPort1_DataReceived(sender As Object, _
                                         e As IO.Ports.SerialDataReceivedEventArgs) _
                                     Handles SerialPort1.DataReceived
        Threading.Monitor.Enter(dataStrmLock)
        dataStrm &= SerialPort1.ReadExisting
        Threading.Monitor.Exit(dataStrmLock)
        checkIt.Set()
    End Sub

    Dim checkIt As New Threading.AutoResetEvent(False)
    Dim dataStrm As String
    Dim dataStrmLock As New Object
    Dim sentences As New List(Of String)
    Dim sentenceLock As New Object

    Public Sub checker()
        Dim found As Boolean = False
        Do
            checkIt.WaitOne()
            Do
                found = False
                Threading.Monitor.Enter(dataStrmLock)
                Dim sPos As Integer = dataStrm.IndexOf("$")
                Dim ePos As Integer
                If sPos >= 0 Then
                    ePos = dataStrm.IndexOf(ControlChars.CrLf, sPos)
                    If ePos > sPos Then
                        Threading.Monitor.Enter(sentenceLock)
                        sentences.Add(dataStrm.Substring(sPos, ePos - sPos))
                        Threading.Monitor.Exit(sentenceLock)
                        found = True
                        dataStrm = dataStrm.Remove(0, ePos + 2)
                    End If
                End If
                Threading.Monitor.Exit(dataStrmLock)
            Loop While found
        Loop
    End Sub

    Private Sub Timer1_Tick(sender As Object, e As EventArgs) Handles Timer1.Tick
        Do While sentences.Count > 0
            Threading.Monitor.Enter(sentenceLock)
            Dim aSentence As String = sentences(0)
            sentences.RemoveAt(0)
            Threading.Monitor.Exit(sentenceLock)
            'process the sentence here
            Debug.WriteLine(aSentence)
        Loop
    End Sub

    Private Sub Form1_Shown(sender As Object, e As EventArgs) Handles Me.Shown
        Dim t As New Threading.Thread(AddressOf checker)
        t.IsBackground = True
        t.Start()
        Timer1.Interval = 100
        Timer1.Start()
    End Sub

[detlion1643]

How fast is the GPS connection? 9600, 19200...

I honestly must've overlooked that fact in my testing. I guess by default it was connecting at 9600. I thought for some reason that number was how fast the serial unit updates through the com port (not thinking it could be set via the com port). The unit itself states 4800 - 115200 (1hz - 5hz).

You need to know the protocol that your serial device use

Does this just make it 'seamless' to do. It can be done efficiently without knowing the protocol right?

The buffer is the most versatile and efficient way to go.

As with the above, does the buffer only really work if we know the protocol and the byte amounts? Or is this something we can set and keep checking the incoming and be more efficient than using strings/etc? Remember, i've yet to work with bytes, but started reading up on them...

dbasnett - thanks for the example. Some threading terms I've not yet heard of or used (ie: Threading.Monitor). Doing some research on them. The example seems like how I imagined it should work, maybe I'll give something to that effect a try.

The gps unit is a gm-2 iblue (apparently disco'd for a while). I've had it about 2 years and just started toying with it. Cheers!

[dbasnett]

I honestly must've overlooked that fact in my testing. I guess by default it was connecting at 9600. I thought for some reason that number was how fast the serial unit updates through the com port (not thinking it could be set via the com port). The unit itself states 4800 - 115200 (1hz - 5hz).

Does this just make it 'seamless' to do. It can be done efficiently without knowing the protocol right?

As with the above, does the buffer only really work if we know the protocol and the byte amounts? Or is this something we can set and keep checking the incoming and be more efficient than using strings/etc? Remember, i've yet to work with bytes, but started reading up on them...

dbasnett - thanks for the example. Some threading terms I've not yet heard of or used (ie: Threading.Monitor). Doing some research on them. The example seems like how I imagined it should work, maybe I'll give something to that effect a try.

The gps unit is a gm-2 iblue (apparently disco'd for a while). I've had it about 2 years and just started toying with it. Cheers!

If it is running at those slow speeds your program should not have trouble keeping up.

[detlion1643]

I'm just going to put this out there in case it might help anyone else. This is a snippet of what my gps puck was returning. I eyed the lines to notice that \r\n would be the line ending/beginning.

$GPGSA,A,3,31,13,29,20,32,06,30,16,23,03,,,1.14,0.83,0.79*0B\r\n
$GPGSV,3,1,11,30,80,059,18,16,66,206,33,23,48,308,19,31,44,069,16*72\r\n
$GPGSV,3,2,11,51,34,217,28,20,31,245,19,32,31,215,29,06,22,157,22*75\r\n
$GPGSV,3,3,11,13,21,312,19,29,16,045,15,03,12,175,20*48\r\n
$GPRMC,010337.000,A,4204.300438,N,08006.947251,W,0.19,135.61,080213,,,D*73\r\n
$GPGGA,010338.000,4204.300398,N,08006.947227,W,2,10,0.83,281.632,M,-33.777,M,0000,0000*58\r\n
$GPGSA,A,3,31,13,29,20,32,06,30,16,23,03,,,1.14,0.83,0.79*0B\r\n
$GPGSV,3,1,11,30,80,059,18,16,66,206,33,23,48,308,19,31,44,069,16*72\r\n
$GPGSV,3,2,11,51,34,217,28,20,31,245,19,32,31,215,30,06,22,157,22*7D\r\n
$GPGSV,3,3,11,13,21,312,20,29,16,045,15,03,12,175,21*43\r\n
$GPRMC,010338.000,A,4204.300398,N,08006.947227,W,0.02,135.61,080213,,,D*7A\r\n
$GPGGA,010339.000,4204.300409,N,08006.947231,W,2,10,0.83,281.638,M,-33.777,M,0000,0000*5B\r\n
$GPGSA,A,3,31,13,29,20,32,06,30,16,23,03,,,1.14,0.83,0.79*0B\r\n
$GPGSV,3,1,11,30,80,059,18,16,66,206,33,23,48,308,19,31,44,069,17*73\r\n
$GPGSV,3,2,11,51,34,217,28,20,31,245,20,32,31,215,30,06,22,157,23*76\r\n

Now, I've been looking through some current parsers and what each piece of the lines mean. I still have to figure that part out, but everyone in this thread has been a great help so far.

[detlion1643]

I'm probably just nitpicking now, but does it make any sense to fire an event for each complete line/sentence found? Or is that not efficient with how much it will be firing?

I have set up the parsing part already and fire events based on each 'piece' of information parsed.

[dbasnett]

If you are using something like I posted in post#4 you don't need to fire an event. When you have a sentence the first thing to do is to calculate the checksum and check that it is the same as what is in the sentence.

[detlion1643]

If you are using something like I posted in post#4 you don't need to fire an event.

That's why I mentioned I'm probably just nitpicking now. Timer, thread, event? I guess at the point in which you get a complete sentence, how to begin parsing sounds like it doesn't really matter which method is chosen.

[stanav]

I'm just going to put this out there in case it might help anyone else. This is a snippet of what my gps puck was returning. I eyed the lines to notice that \r\n would be the line ending/beginning.

$GPGSA,A,3,31,13,29,20,32,06,30,16,23,03,,,1.14,0.83,0.79*0B\r\n
$GPGSV,3,1,11,30,80,059,18,16,66,206,33,23,48,308,19,31,44,069,16*72\r\n
$GPGSV,3,2,11,51,34,217,28,20,31,245,19,32,31,215,29,06,22,157,22*75\r\n
$GPGSV,3,3,11,13,21,312,19,29,16,045,15,03,12,175,20*48\r\n
$GPRMC,010337.000,A,4204.300438,N,08006.947251,W,0.19,135.61,080213,,,D*73\r\n
$GPGGA,010338.000,4204.300398,N,08006.947227,W,2,10,0.83,281.632,M,-33.777,M,0000,0000*58\r\n
$GPGSA,A,3,31,13,29,20,32,06,30,16,23,03,,,1.14,0.83,0.79*0B\r\n
$GPGSV,3,1,11,30,80,059,18,16,66,206,33,23,48,308,19,31,44,069,16*72\r\n
$GPGSV,3,2,11,51,34,217,28,20,31,245,19,32,31,215,30,06,22,157,22*7D\r\n
$GPGSV,3,3,11,13,21,312,20,29,16,045,15,03,12,175,21*43\r\n
$GPRMC,010338.000,A,4204.300398,N,08006.947227,W,0.02,135.61,080213,,,D*7A\r\n
$GPGGA,010339.000,4204.300409,N,08006.947231,W,2,10,0.83,281.638,M,-33.777,M,0000,0000*5B\r\n
$GPGSA,A,3,31,13,29,20,32,06,30,16,23,03,,,1.14,0.83,0.79*0B\r\n
$GPGSV,3,1,11,30,80,059,18,16,66,206,33,23,48,308,19,31,44,069,17*73\r\n
$GPGSV,3,2,11,51,34,217,28,20,31,245,20,32,31,215,30,06,22,157,23*76\r\n

Now, I've been looking through some current parsers and what each piece of the lines mean. I still have to figure that part out, but everyone in this thread has been a great help so far.

Collecting data is one thing and making that data useful is another. To accurately parse a "sentence", you need to know what each piece of information means. You can only guess so much, and if you use these values for further calculations, it won't take much to steer your results way off the real target. If possible, I suggest to contact the device manufacturer to see if they can provide any documentation for you...

[detlion1643]

NMEA is an industry standard, is it not? The amount of info in each 'piece' might differ, but isn't the layout of each sentence the same.

IE: Split sentence at comma, each result is a certain piece of info. Result(1) is always latitude, Result(2) is always longitude, and so forth, no matter how much is contained within those Results. I know 1 and 2 aren't lat/long, it's an example.

I've been reading up on the contents of NMEA strings and it seems to point to this at least.

[dbasnett]

NMEA is an industry standard, is it not? The amount of info in each 'piece' might differ, but isn't the layout of each sentence the same.

IE: Split sentence at comma, each result is a certain piece of info. Result(1) is always latitude, Result(2) is always longitude, and so forth, no matter how much is contained within those Results. I know 1 and 2 aren't lat/long, it's an example.

I've been reading up on the contents of NMEA strings and it seems to point to this at least.

Check this out
http://home.mira.net/~gnb/gps/nmea.html#top

[dbasnett]

Here are some classes that I used

    Public Class GPSSystemFixData
        Property wasError As Boolean
        '$GPGGA - Global Positioning System Fix Data
        Property fixTaken As DateTime '123519 UTC
        Property clockDiff As TimeSpan
        Property clockWithinSecond As Boolean
        Property Latitude As Decimal '4807.038
        Property NS As String 'N
        Property Longitude As Decimal '01131.000
        Property EW As String 'E
        Property fixQuality As Integer '0 = invalid, 1 = GPS fix (SPS), 2 = DGPS fix, 3 = PPS fix, 4 = Real Time Kinematic, 5 = Float RTK
        '                               6 = estimated (dead reckoning) (2.3 feature), 7 = Manual input mode, 8 = Simulation mode
        Property numSatellitesTracked As Integer
        Property HDOP As Decimal 'Horizontal dilution of position
        Property Altitude As Decimal 'Altitude, in Meters, above mean sea level
        Property geoidHeight As Decimal 'Height, in Meters, of geoid (mean sea level) above WGS84 ellipsoid
    End Class

    Public Class GPSActiveSatelliteDOP
        Property wasError As Boolean
        '$GPGSA - GPS DOP and active satellites
        Property AutoManual As String
        Property mode As String
        Property PRN As New List(Of String)
        Property PDOP As Decimal
        Property HDOP As Decimal
        Property VDOP As Decimal
    End Class

    Public Class GPSTransit
        Property wasError As Boolean
        '$GPRMC - Recommended minimum specific GPS/TRANSIT data
        Property fixTaken As DateTime '123519 UTC
        Property positionValid As Boolean
        Property Latitude As Decimal '4807.038
        Property NS As String 'N
        Property Longitude As Decimal '01131.000
        Property EW As String 'E
        Property speed As Decimal 'knots
        Property course As Decimal
        Property variation As Decimal
        Property variationEW As String
        Property modeIndicator As String 'Mode indicator, (A=Autonomous, D=Differential, E=Estimated, N=Data not valid)
    End Class

    Public Class GPSSatellitesInView
        Property wasError As Boolean
        Property numOfSatellites As Decimal
        Property PRN As String
        Property elevation As Double
        Property azimuth As Double
        Property SNR As Double
    End Class

As I recall I processed the sentences and updated instances of these classes in the background thread. In a timer that ran 10 times per second I updated a form with the data.