Mass Queuing type system

[Techno]

I'm trying to figure this out and need some suggestions.

OK so, lets say I have a system where I store a list of jobs to "do".
The jobs would have a specified time they should run at to perform the task.
There will be hundreds of jobs "to do" for a specific time.

for clarity, lets say a job is to send emails to customers or customers of clients and so on.

The user can set a time on when they want the job to run.

But how does one know, very efficiently, when a job has been added to run at a specific time and then go ahead and keep updating the system so that the Windows Service that will "monitor" the jobs which are to run, gets invoked?

Sure, I could keep polling the DB every few minutes but that is expensive and inefficient.

But even if I did poll, since there would be hundreds of jobs to run (Each job has x amount of emails to send), this would take some time to go through and what if there are several clients who want to run a job which overlaps with another job? how can that be run efficiently?

maybe im not making it clear as my mind is a little fuzzy on this.

I just want to know what is the best way on keeping the Service updated on when a job should be scheduled to run (which can be changed) and when it is to be run to make sure that it doesnt have issues "scheduling" a job if a job is overlapping or overrunning a previous job.

No doubt threading will be used carefully but the issue is not threading here.

[jmcilhinney]

If you're using SqlServer then you can use the SqlDependency class to receive notifications from the database when changes occur.

As for the scheduling, you've basically got two choices:

1. Use a single Timer with a relatively small Interval, e.g. 60000 milliseconds so it raises an event once per minute. You can then test the current time and, if a task is due, start a new thread to execute it.

2. Create one Timer per task with its Interval set so that it will raise its Elapsed event exactly when the corresponding task is due.

[Techno]

agreed and this is what I was thinking but what I am concerned about is the performance in terms of notifications from SqlDependancy or polling the DB every 15 minutes to see if there are jobs ready to be run for an "agent".

an "agent" is a Windows Service, per physical server due to the quite possible large demand of "jobs" to do.
you can create a job and assign it to an Agent to perform

[jmcilhinney]

SqlDependency is a good option as long as you aren't creating a large number of them. How many servers are there? I've never actually used SqlDependency in anger but I think it should be OK up to about half a dozen.

That said, if polling every 15 minutes will work for you then it might be a bit easier, because I tried to test SqlDependency once and couldn't get it to work.

[Techno]

Sure.

well initially there will be 1 server and see how it goes, if I find the demand becomes more over time, then I will add another server and keep expanding as necessary.

The plan was:

Have a QueryTimer which every 15 mins sees if there are any jobs to run for Agent X
If there is, it will create x amount of job timers to run at the time specified and then execute that when it needs to, which will involve threading (thats fine)
Once the job is complete, it will update the DB as necessary and then remove itself from the list of jobs to run in memory.

During this time, the QueryTimer will keep polling the DB for any jobs which are needing to be run and as necessary, keeping adding it into a job queue list.

So yes, there would be say multiple jobs running at the same time (a job would overlap another job which is running, realistically)

[Techno]

The other issue I have with polling the DB is that, I hate it because of the fact that its inefficient and has a perf hit.
The fact that 1 agent would do it every 15 minutes is one thing but then say 10 agents do the polling is another....

[Milk]

I can't say I'm any expert with DB's or interprocess communication.

Maybe you could have something that sits between the DB and the Agent(s), an Agency if you like.

The Agency would be the listener that interacts with the DB, and the server to the Agent(s) via Remoting or some such.

[Techno]

you still need access to the DB to find out what jobs are needing to run

[jmcilhinney]

Yes, but you're guaranteed to only have one point of contact. Look, you've got two choices: push or pull. You know the implications of each one so just make a decision. I'd be going with push, i.e. SqlDependency.

[Techno]

Indeed, that point of contact would be an AgentController would then co-ordinate which agent gets the job to do based on what its been assigned in the DB.

ill see what happens with the SqlDependancy
Thanks