Help with threading

[ianbhenderson73]

Hi

A couple of years ago I built a suite of Windows Services, one of which was concerned with the sending out of emails to customers based on pre-defined business rules. The service works well, but the volume of emails that we need to handle has increased quite significantly in the last six months and the service is now swamped.

To provide a bit of background, I set up a system which collates all of the data for the outgoing emails together, and populates a database table with the basic information required for each outgoing message - sender address, recipient address, email subject and email header. It also allocates a timeslot to each outgoing email so that the messages can be transmitted in waves throughout the day. This "batching" step works on the basis that the email server can handle 600 messages going out in a 15-minute period. We start the email transmission at 7am and try not to send anything out after 7pm.

When it comes to the actual email transmission, this is done sequentially:

1. Select the next pending email
2. Send the email
3. Wait for the email server to confirm transmission
4. Update the record with the transmission date & time

The issue I've got now is that demand for this service has increased in the last month or so, and I'm now struggling to get this increased volume handled in that 7am-to-7pm window.

I would like to continue splitting the outgoing messages into batches and assigning a time-slot to each outgoing batch, but I would like to send the contents of each batch out concurrently rather than sequentially. In my head, the system would monitor the database as it does at the moment (every 10 seconds). When it finds a batch to send out, it would read in the contents of that entire batch and then utilise some sort of multi-threading approach to send out the emails. I would want some sort of method of controlling the number of threads being created so that the process didn't use all available server resource, but would create a finite number of threads (10 for example) and reuse those threads until there were no further emails to be sent.

I've done some work with multi-threading in desktop application environments where I want the screen to continue to be responsive while something else is running in the background, but I'm struggling to get my head around how this would translate to the system I've detailed above.

TIA

[jmcilhinney]

I would suggest that you use the Parallel.ForEach method and let the ThreadPool manage the number of threads automatically.