Threading questions

[Techno]

I have done multi threading before but has been a while and need to refresh my memory and the best practice to do the following:

to process a high volume of input and output messages (messages would be placed in some form of queue and depending upon the type of message, it would request a certain "handler" to process that mesage. The "handler" would then place it into another processing queue for that type of message) - how would this be done and what would be the best way of doing this, making sure its scalable? What about making sure that this processing system is independant, more like an SOA/SAAS - should it be hosted as WCF? (I do think so but just curious, since there will be other "outsiders" consuming such a service)

To add complexity, what about dealing with transactional data in terms of the database? Making sure that things are being processed once and not multiple times (thread racing I guess)? So I may wish to CRUD that message - want to make sure that it is not done x amount of times (yes I know, perhaps place a lock around it but I want to make sure there is far less chance of this always being the case due to performance and scalability reasons)

Thanks

[jmcilhinney]

With regards to the threading specifically, sounds like a job for the ThreadPool and perhaps a ConcurrentQueue.

[Evil_Giraffe]

What does hosting it in WCF give you? It sounds like communication in and out of the process is done through messages, not procedure calls. In which case your message queues are the interface, not web methods. I would look into messaging frameworks like NServiceBus and MassTransit. These will handle the 'threading' bit for you a swell, just register the handlers for the message types and go for it (modulo configuring the engine, of course).

[Techno]

Thanks

Re: Threading - yes, totally aware of the ThreadPool. I guess I want to make sure that anything is processed immediately as there would be literally hundreds/thousands of things to process (since the system is a high performance type application). As for the concurrentQueue.... interesting.

Re: "Bus". Thanks. I am not sure if we will be using MSMQ, due to previous little experience where apperently some messages were just disappeared from the queue or something of that nature. Do not want any data to be lost at all but does need to be placed in some queue, waiting to be processed and to have the ability of listening to messages and sending messages.

Again, not sure if this should be a WCF service or something else? Need to make sure that this business logic is independant of the applications surrounding this unit of business logic.