Threads + Synlock + Buffers

[mickey_pt]

Hello

Need some advice, for this application that i'm building.

I have a machine that sends messages (ant+ messages with 5 different types and each one has 8 bytes), the timing isn't defined, but i get several messages in one second, and doesn't have any specific order. The data that i save for each message will vary with the message type, but will be short or integer.

I have 3 requirements, from order of importance to the system:

1 - Save all the data in buffers, while testing, i'm using 5 sorted lists, the keys are the ticks, and the value it's the value in the message.
Question: Is this the best way to store the data? And about the event handler, the sdk does all the job to receive the data (event onDataReceived), but then defining the procedure to handle this data to save, how to make it run in a different thread?

2 - One of the buffers need to be re sent in another channel but with a predefined timing 250ms/500ms/1s (not defined by now), to do this, i was thinking in creating a timer and in the elapsed event, i'll get all values received, within the the predefined timer, if it's one second, i'll get all itens in the list that the keys are greater than the tick value of the previous second, and get the average of the values and sent.
Question: The buffers will be "shared", so i need to control the access, synlock? Other way? Or just use a double buffer approach? What's the best way to receive all data, and to do this? If the sortedlist it's the ay to go, binarysearch and extension method select?

3 - Interface, that shows the data received, some thread, that get the last value of each buffer at intervals of one second.
Question: The same problem, getting data from the buffers and the shared memory.


The general workflow of the app, start the GUI, user click in start, the system resets and start listen and start sending. Main problem, shared access without losing anything.

So this is programming and approach to the problem question, doesn't know in what section to post so i posted it here.

Thanks

[Grimfort]

I have a similar system, more complex but each part is fairly simple. I get data in from connections which I tag with an ID , just use a class, and add to a Queue. The Queue can then be read by a Timer on a different thread when you can control how often you pickup the messages. The Queue can do all the work for you, just SyncLock it when adding and removing, create an add and a remove Method to do this, not much more is required.

[mickey_pt]

@Grimfort
By the way i don't want to remove the elements of the collection. In the end the user will have the option to save all the buffers to a log file, just for control.

So SyncLock it's a good way to go.
Now looking to the other "problem", get the values between one start time until the end of the collection.
Pseudo Code:

Get the Keys
BinarySearch of the StartTime (ticks)
ExtensionMethod Select From the Index
Return Average of Values

It's a good "option" to handle this situation?

Thanks

[Grimfort]

O, and mark each class in the queue as Serializable, and then you can Serialize the Queue itself in a couple of lines straight to a file to keep context.

[TnTinMN]

Another option is to use one of the collections from the System.Collections.Concurrent Namespace.

[passel]

It doesn't seem like you have too many messages to process so you could probably get by with a couple of threads, one to receive the messages and another to process them (and of course the GUI thread to display things).
As for organization, for the lightest weight access control would be a concurrent queue.
I would have a background thread that receives a message and just pushes it on a concurrent queue and waits for the next message.

A second thread would read from the concurrent queue and process the messages in the order received.
Storing the messages in separate lists should be a valid approach.

It sounds like you want to average the value from one of the messages over a period of time. Will the length of time be a single time, just not decided what yet, or will you have several averages.
In either case, in addition to adding the message to a list for history I would add the message, or data from the message to a queue.
That way when it is time to average the value for the shortest period I would dequeue the messages accumulated and average them.

If you have multiple time periods to average, you could push that average on another queue and extract and average them at an appropriate time.

If the values to display are simple types (not an object type), then there is no access control needed for those. The thread reading the queues and storing values in the lists can update public values that the GUI thread can read and update displays at the rate of your choosing.
If the GUI has to access the lists themselves, then you would need to guard access by a mechanism such as SyncLock.

[mickey_pt]

I dont know what are to many messages...

The averages, by default i'll have 3, and thanks for remember me to using the extra queues, simplifies everything.

The GUI, only display simple numeric values.

[passel]

Its not so much the number of message types, but the net frequency of the messages being received.

The fact is, that while I have used the two thread process for receiving messages in one thread, and processing them in another, most of the time that is unnecessary.
The majority of the time, I have the receiving thread just examine the header of the message to identify the message type then using a case statement or number of If ElseIf blocks, parses the simple messages and stores the data, or for more complex messages calls a parse routine provided by a class that handles that message type.
After the message is parsed and global values updated, the thread just loops back up to read the next message.

The receiving thread processing the message read should be able to handle 500 simple messages per second with no problem, and probably handle more than 3000 per second without too much latency.
If you have less than 500 messages per second to process, you could probably skip the concurrent queues and just process the messages in the receiver as you receive them.
Since the receiver is in a background thread it won't be hung by things going on in the GUI, likewise the GUI won't be hung up by the receiver.