queuing requests / async processing

I'm putting together a web app that will process time consuming jobs in the background.

so for example 100 users are on the site 20 of them submit jobs to the server. these 20 jobs get added to a list and the background worker pickes them up one by one. I want to be able to have up to 10 background workers doing the tasks.

for scaleability the job list is likely to be a database table, however at the proof of concept stage I will probably use an application variable (stack of some sort) with a GUID to identify each job.

So I *think* what I need is ....

a single background worker process that starts when the web app starts.

This process monitors the stack / job list. It dynamically creates a worker process for the first item, removes the job from the stack and increments the thread count.

If there is a second job it creates another thread (up to 10) and increments the thread counter.

When each thread finishes it closes itself and decrements the thread count.

Does this seem possible / sensible?

If I move to the database table job list would it be better to have ASP.Net add jobs and a windows service do the processing?

The first one seems to be better option.

Hey Dan,

Where is this background worker process going to be running? Within the context of the page that is currently being viewed? If so, you are likely going to run into problems with this approach. A better solution, if you have the option, would be to create something like a scheduled task to hit a Web Service, or a Windows Service, to process all the jobs, when there are some in the queue.

Gary

The background worker(s) will be started at application start up. The page will poll them using a GUID to see it it's job i queued or finished.

The only problem I see with this approach is if the web app restarts due to an exception or something.
Hence why I thought a database would be best for the job list as it be persisted even if something goes wrong.

The reason I thought about the windows service was because it would not stop if the web app restarted. however with a database driven job list the web app restart isn't a huge problem unless it happens very often.

Also a windows service could be farmed out to one or more servers if my job lists need extra muscle to get through it, but for now people can just be patient!

Hey,

At the end of the day, the decision is up to you, but anytime I have something like this to do, I don't do it within the actual web application. In my opinion, this is the wrong place to do it.

Gary

ok, so it looks like a windows service is the way forward.

Hey,

This is the way that I would have done it. This would mean that you are going to need to have access to the server that is running your site, i.e. in order to install the Windows Service. Do you have that access?

Gary

yes, I use dedicated/VPS cloud servers and the moment so should be fine, although I've been thinking about moving to Azure or similar.
I assume this wouldn't be viable with Azure?

Hey,

On that one, I will need to pass, I haven't looked into Azure yet :(

Gary