[2005] Need some conceptual direction re threading

My application takes a verbal passage and performs an internet search (on Google Books) on each sentence in the passage gathering citations. It then goes ahead to manipulate the citations. It works just fine, but slowly and I am trying to speed up the process. The speed problem stems from the fact that response time and download time for the website being searched varies according to passage.

The first thing I did was to break the passages down and use separate WebBrowser Controls to run the searches simultaneously. That had limited success and is further hampered by the fact that I will not know at design time how many searches any given passage will require.

Next I thought that I would simply through the WebBrowser into a separate thread, but here the problem is that I'm not sure I can create a thread on the fly. (I would appreciate input on that).

Next I decided that I would put the WebBrowser and the new thread into a separate class object, but I have realized that that probably doesn't have the desired effect because the main thread is going to be waiting on the class object before proceeding with execution irrespective of the fact that the class object has instantiated a new thread for the WebBrowser.

Any guidance would be appreciated.

I doubt you will get much improvement by using threading in this case. If you can have several web browsers going after several different things, then putting those web browsers into different threads doesn't, on it's own, gain you anything more.

There could be some advantage if you can have a main thread that is capable of doing things as the web browsers are working, but that doesn't sound like it is initially possible. You do mention that there will be manipulations on the citations. Is this a slow process in itself? Can it be done after some of the web browsers have completed, but before ALL of them have completed? If so, then there is some advantage to multithreading, because each thread could signal the main thread that it has data for it to work on when it completes.

Multithreading doesn't provide an instant boost if you have only a single processor. You might get a boost if you have multiple processors (or multiple cores), because your program can use the multiple processors to have multiple threads running simultaneously. If you have only a single processor, then it will have to run all the threads, so each thread will get a slice of time in turn, and there will be a small cost as the processor switches from one thread to the next.

If the multiple web browsers will actually all work simultaneously, then you are getting the multithreading without the context switching overhead.

What you could do is to use a ThreadPool to queue all your work items and perform them on background threads as they become available. You could use a WebClient or HttpWebRequest/HttpWebResponse to perform the work and then only use a WebBrowser to display the results once they've arrived.