Loading from database into grid in a background thread

Hi,

I have a grid that is supposed to show records from a database that depict scientific papers (title, authors, year, journal, etc). The user can categorize each paper into multiple categories.

Beside the grid I now have a CheckedListbox that lists all categories. The user can check or uncheck categories, and the grid should then display only the papers that belong to the checked categories.

I would now like to load the grid asynchronously in a background thread, because it might take a while to load all papers from the database and then check them against the checked categories.

I tried two different approaches:

1. Using a BackgroundWorker
2. Creating a thread manually

For the BackgroundWorker, I have this code:

csharp Code:

private EntityCollection<Paper> papers = new EntityCollection<Paper>();
        
        // Called when the user checks an item in the categories list
        public override void RefreshData()
        {
            dataLoader.CancelAsync();
            dataLoader.RunWorkerAsync();
        }
        
        private void dataLoader_DoWork(object sender, DoWorkEventArgs e)
        {
            // Get the checked categories:
            var categories = ViewFormObserver.Instance.MainForm.GetSelectedPaperCategories();
 
            // Get all papers:
            var allPapers = PaperManager.Instance.Load();
 
            // Clear the list of papers
            papers.Clear();
 
            // Add only those papers that have any of the checked categories
            bool found = false;
            papers.Clear();
            foreach (var paper in allPapers)
            {
                found = false;
 
                foreach (var category in categories)
                {
                    if (!found)
                    {
                        if (paper.Categories.ContainsId(category.Id))
                        {
                            papers.Add(paper);
                            found = true;
                            continue;
                        }
                    }
                }
            }
        }
 
        private void dataLoader_RunWorkerCompleted(object sender, RunWorkerCompletedEventArgs e)
        {
            // Set the datasource of the grid
            this.Grid.DataSource = papers;
            this.Grid.DataBind();
        }

'dataLoader' is the BackgroundWorker component.

EntityCollection<T> inherits List<T> and provides some useful methods for my database code (such as ContainsId, which checks if the collection contains a database record with the specified Id).

The PaperManager.Instance.Load() method loads all papers from the database (this is the method that probably takes the longest). Then a loop runs through every paper and every selected category and 'filters out' the papers that don't have any selected category. For a large number of papers this could also take a while.


Anyway, this works, but not if I click too quickly in the CheckedListBox. If I check an item while the grid is loading, for example I can doubleclick an item so it's checked and immediately unchecked again, then the BackgroundWorker throws an exception on RunWorkerAsync:

Code:

---

This BackgroundWorker is currently busy and cannot run multiple tasks concurrently.

---

I suppose it is not done cancelling yet thus cannot start another load operation.


So, I switched to using my own thread manually:

csharp Code:

private EntityCollection<Paper> papers = new EntityCollection<Paper>();
        private Thread loadingThread;
        
        public override void RefreshData()
        {
            // If we are already loading, abort
            if (loadingThread != null)
            {
                loadingThread.Abort();
                loadingThread = null;
            }
            
            // Create a new thread and start it
            var finishedCallback = new MethodInvoker(LoadDataFinished);
            loadingThread = new Thread(LoadData);
            loadingThread.IsBackground = true;
            loadingThread.Start(finishedCallback);
        }
 
        private void LoadData(object finishedCallback)
        {
            var callback = (MethodInvoker)finishedCallback;
 
            var categories = ViewFormObserver.Instance.MainForm.GetSelectedPaperCategories();
            var allPapers = PaperManager.Instance.Load();
 
            bool found = false;
            papers.Clear();
            foreach (var paper in allPapers)
            {
                found = false;
 
                foreach (var category in categories)
                {
                    if (!found)
                    {
                        if (paper.Categories.ContainsId(category.Id))
                        {
                            papers.Add(paper);
                            found = true;
                            continue;
                        }
                    }
                }
            }
 
            callback.Invoke();
        }
 
        private void LoadDataFinished()
        {
            this.Grid.DataSource = papers;
            this.Grid.DataBind();
        }

Code is basically the same, except instead of calling BackgroundWorker.RunWorkerAsync I create a new thread and start it. I pass a MethodInvoker to the thread which it can invoke when the loading is finished.

The problem here is that I have to Abort the thread if it's already running. As far as I'm aware you should never abort a thread unless you're application is exiting, so this seems completely wrong...

Also, sometimes it seems to grid 'crashes'. It throws an unhandled exception and draws a red cross through the control. The exception is thrown on a seemingly random method, somewhere inside the grid control (the grid is a third party grid, but that shouldn't matter I think...).

So there's definitely something wrong with my threading....



Any idea what I'm doing wrong and how I can avoid it?

In case it isn't clear, here's what I'm trying to do:

- User checks or unchecks a category in a listbox.
- Grid starts to refresh asynchronously, in the background
- If the user checks or unchecks another category while the grid is loading, the current loading operation is aborted, and it starts to load again.

There should obviously not be a 'queue' of loading operations, if the user checks 4 items in a row I want it to abort the first three loading operations and only finish the fourth, I don't want it to load the first, then load the second, then the third and finally the fourth...

Thanks!

I post to many threads advising people to read the appropriate documentation and I'm going to do it again. Read the documentation. You obviously haven't in either case because you have a call to the CancelAsync method of a BackgroundWorker without anywhere checking the CancellationPending property and you also have a call to the Abort method of a Thread without anywhere catching a ThreadAbortException. You've assumed that certain members work a certain way and, when they didn't, you should have immediately read the documentation to see whether your assumptions were correct. Doing so may not have provided a full solution but it would certainly have highlighted the main issue with your code in each case.

I didn't read the documentation for the BackgroundWorker, mainly because I already decided I wanted to do it manually without a BackgroundWorker; I was just trying a BackgroundWorker to see if it solved the problem more easily. When it didn't I abandoned it and tried my own manual approach without bothering to solve the problem. I merely posted it to make the post more complete.

I did read the Thread.Abort documentation, and I'm aware that it raises a ThreadAbortException. I just don't know what I should do with it. I could catch a ThreadAbortException in the LoadData method, but what should I do in that case? I don't have anything to dispose for example. The catch block would be empty, so it seemed redundant. Am I not understanding the ThreadAbortException then?

I think my main problem is that there is no 'neat' way of stopping the code. The PaperManager.Instance.Load method is what takes the most time, but once I call it I cannot 'cancel' it. The entire PaperManager class (as well as the Paper entity classes it loads) is automatically generated with T4 templates, and building cancellation support into that would be a lot of work. If it's required then I'll do it, but unless I don't understand how aborting a thread works it shouldn't be required because the code just stops when the thread is aborted, right?

What if you try calling BeginInvoke instead of Invoke?

Have you considered simply loading all the papers asynchronously, and wiring up the checkboxes to filter the items that are shown in the grid on the client side?

(p.s. if you have rejected that due to performance or memory concerns... I really hope you've measured first)