Results 1 to 19 of 19

Thread: Best practices for debugging error that happens outside IDE in Multi-Threaded app

  1. #1

    Thread Starter
    Hyperactive Member
    Join Date
    May 2007
    Posts
    300

    Best practices for debugging error that happens outside IDE in Multi-Threaded app

    Multi-Threaded apps are hard enough to debug within the IDE, so how about one that gives errors only when outside the IDE?

    A customer was reporting a crash of the program, and I couldn't reproduce the problem until I ran the .exe on it's own. The program has lots of try..catch blocks to catch exceptions, but in this case, the try..catch works when in the IDE, but when running the .exe, it seems to jump outside the block.
    Pinning this down is really troublesome and I could use some help figuring out a better way of debugging this one.

    First of all, what is the usual cause for a program working properly in the IDE but not when run stand-alone?

    Second, is there a trace mode that will list the line numbers that are being executed while the program is running? I was thinking I could start that logging just before the point where the crash happens and see what's going on in the program flow.

  2. #2
    PowerPoster
    Join Date
    Oct 2010
    Posts
    2,141

    Re: Best practices for debugging error that happens outside IDE in Multi-Threaded app

    VS2015 has more than its share of bugs. You are not alone in seeing differences between optimized code (normal run mode) and while running in the debugger. See: VB.NET Iterator Function Loses Local Variables for one example.
    Last edited by TnTinMN; Mar 16th, 2016 at 05:36 PM.

  3. #3

    Thread Starter
    Hyperactive Member
    Join Date
    May 2007
    Posts
    300

    Re: Best practices for debugging error that happens outside IDE in Multi-Threaded app

    That's not too reassuring. I spent a LOT of time getting my program to work under VS2015/.net 4.6, (even though it's currently built for 4.0 so it'll run on XP) so I'd like to not have to go back.

    What is the next step in figuring this out?

  4. #4
    PowerPoster
    Join Date
    Oct 2010
    Posts
    2,141

    Re: Best practices for debugging error that happens outside IDE in Multi-Threaded app

    Try compiling the project with optimizations disabled and see if that is the culprit. If it is, can you live with the un-optimized performance? If not, you are on a forensic bug hunt comparing dis-assemblies.

    Are you using any language features specific to VS2015? If not, you could just recompile it in V2012/2013.

  5. #5

    Thread Starter
    Hyperactive Member
    Join Date
    May 2007
    Posts
    300

    Re: Best practices for debugging error that happens outside IDE in Multi-Threaded app

    Thank you for those suggestions.
    Optimizations disabled makes no difference, unfortunately.

    I re-wrote the program using a lot of VS2015-specific shortcuts/features, unfortunately. Going back would be very difficult.

    How about my idea of logging line numbers to follow the flow of the program? Any thoughts on a way to do that other than adding hundreds of debug/trace lines?

  6. #6
    PowerPoster
    Join Date
    Oct 2010
    Posts
    2,141

    Re: Best practices for debugging error that happens outside IDE in Multi-Threaded app

    Quote Originally Posted by Meestor_X View Post
    ...
    Optimizations disabled makes no difference, unfortunately.
    Quote Originally Posted by Meestor_X View Post
    How about my idea of logging line numbers to follow the flow of the program? Any thoughts on a way to do that other than adding hundreds of debug/trace lines?
    No idea on that one. Do you have an un-handled exception handler? If so, is it capturing the error?

  7. #7
    You don't want to know.
    Join Date
    Aug 2010
    Posts
    4,578

    Re: Best practices for debugging error that happens outside IDE in Multi-Threaded app

    Insert logging statements like crazy. You can use File.AppendText() or whatever for now. Problem: it changes the timing of code because it takes time to write to the file, so with these kinds of errors you can sometimes "fix" the problem by moving code around.

    For threading issues, the best solution is a thorough code review by experts. And rabid overuse of SyncLock.
    This answer is wrong. You should be using TableAdapter and Dictionaries instead.

  8. #8
    Smooth Moperator techgnome's Avatar
    Join Date
    May 2002
    Posts
    34,531

    Re: Best practices for debugging error that happens outside IDE in Multi-Threaded app

    Start the program... open VS, open your project/solution....
    Press Ctrl+Alt+P ... that brings up the Attach to process dialog... find your running process... select it and then Attach... now perform the steps needed to reproduce the problem.

    the only catch is that the version of the code needs to match the version of the code that's running (IE, if you've made changes to the code and don't re-compile, things will get out of synch real fast. At any rate, it can be handy to find out what's going on in the "running" world as opposed to the IDE world.

    -tg
    * I don't respond to private (PM) requests for help. It's not conducive to the general learning of others.*
    * I also don't respond to friend requests. Save a few bits and don't bother. I'll just end up rejecting anyways.*
    * How to get EFFECTIVE help: The Hitchhiker's Guide to Getting Help at VBF - Removing eels from your hovercraft *
    * How to Use Parameters * Create Disconnected ADO Recordset Clones * Set your VB6 ActiveX Compatibility * Get rid of those pesky VB Line Numbers * I swear I saved my data, where'd it run off to??? *

  9. #9

    Thread Starter
    Hyperactive Member
    Join Date
    May 2007
    Posts
    300

    Re: Best practices for debugging error that happens outside IDE in Multi-Threaded app

    Quote Originally Posted by TnTinMN View Post
    No idea on that one. Do you have an un-handled exception handler? If so, is it capturing the error?
    Yep.

    The error it seems has nothing to do with the actual problem, as I've seen time and time again with multi-threaded apps.
    Last edited by Meestor_X; Mar 17th, 2016 at 01:51 AM.

  10. #10

    Thread Starter
    Hyperactive Member
    Join Date
    May 2007
    Posts
    300

    Re: Best practices for debugging error that happens outside IDE in Multi-Threaded app

    Quote Originally Posted by techgnome View Post
    Start the program... open VS, open your project/solution....
    Press Ctrl+Alt+P ... that brings up the Attach to process dialog... find your running process... select it and then Attach... now perform the steps needed to reproduce the problem.

    the only catch is that the version of the code needs to match the version of the code that's running (IE, if you've made changes to the code and don't re-compile, things will get out of synch real fast. At any rate, it can be handy to find out what's going on in the "running" world as opposed to the IDE world.

    -tg
    Thank you for that suggestion. Unfortunately, "attaching" works the same as debugging. The program works properly in that scenario

  11. #11

    Thread Starter
    Hyperactive Member
    Join Date
    May 2007
    Posts
    300

    Re: Best practices for debugging error that happens outside IDE in Multi-Threaded app

    Quote Originally Posted by Sitten Spynne View Post
    For threading issues, the best solution is a thorough code review by experts. And rabid overuse of SyncLock.
    I would like to be that expert. It seems hardly anyone does multi-threading apps, and mine are fraught with threads.
    Anyway, let me know what you suggest.

  12. #12
    Frenzied Member
    Join Date
    May 2014
    Location
    Central Europe
    Posts
    1,372

    Re: Best practices for debugging error that happens outside IDE in Multi-Threaded app

    have you taken real care of syncing Access to shared ressources? i do multithreaded apps from time to time and as Long as you understand whats going on and Needs to be synced it is all well. however i realized that many People do stuff with multiple threads without any real Need just because it is so sexy and without knowing or caring about the pitfalls. you can often see a CheckFor illegalCrossThreadCalls=false in code posted here and that says all about it. so do you really know what you are doing or are you more the "threads are sexy" Kind of guy? no offense but "fraught with threads" makes me a bit curious. i guess if you say you are an expert on multithreading then the already made suggestions of extensive logging are your next steps, if you say that there might be some uncertaincies then i'd advice you check all your cross thread Access to vars, files etc. that they are synched properly.

  13. #13
    Powered By Medtronic dbasnett's Avatar
    Join Date
    Dec 2007
    Location
    Jefferson City, MO
    Posts
    9,754

    Re: Best practices for debugging error that happens outside IDE in Multi-Threaded app

    Quote Originally Posted by Meestor_X View Post
    I would like to be that expert. It seems hardly anyone does multi-threading apps, and mine are fraught with threads.
    Anyway, let me know what you suggest.
    Sitten was suggesting that you gives us code and if it is too big the pseudo code overview.

    Is the problem related to load? Are you logging how many threads are running periodically or as a part of the normal logging? Are the threads communicating with one another? Are they dependent on one another?

    Many of the people responding here probably use threads often when needed, but it is hard to help without detail.
    My First Computer -- Documentation Link (RT?M) -- Using the Debugger -- Prime Number Sieve
    Counting Bits -- Subnet Calculator -- UI Guidelines -- >> SerialPort Answer <<

    "Those who use Application.DoEvents have no idea what it does and those who know what it does never use it." John Wein

  14. #14
    PowerPoster SJWhiteley's Avatar
    Join Date
    Feb 2009
    Location
    South of the Mason-Dixon Line
    Posts
    2,256

    Re: Best practices for debugging error that happens outside IDE in Multi-Threaded app

    You most likely have a race condition of some kind, but as DB said, it's in the details. In the IDE it's not apparent, but when the code is allowed to run better optimized without the overhead of debugging, that's when it occurs.

    What do you have in the try/catch blocks? Just because you have some 'logging' in the try catch, it may be that it starts the logging process, jumps out of the routine, then the failure of the routine to do it's job results in another exception which causes the app to crash.

    Personally, apart from a few, well documented issues, I've not seen such subtle VS bugs (true, I'm stuck on VS2010, though); it's always a code issue, particularly with threads. Specifically, the code isn't doing what you think it's doing.

    How have you added the unhanded exception handler?
    "Ok, my response to that is pending a Google search" - Bucky Katt.
    "There are two types of people in the world: Those who can extrapolate from incomplete data sets." - Unk.
    "Before you can 'think outside the box' you need to understand where the box is."

  15. #15
    You don't want to know.
    Join Date
    Aug 2010
    Posts
    4,578

    Re: Best practices for debugging error that happens outside IDE in Multi-Threaded app

    It can also often help to write the code synchronously, without threads, and verify it works properly before adding threads. Unit tests are common in the industry, what do yours say about your code? Are any failing?

    That said, you've got a lot of suggestions, there won't be many more:
    • Show your code to other people and ask them what they think. "I have a lot of threads" isn't as rare as you think. The reason you don't see a lot of tutorials is there's no "process" for tracking down errors, every program is a unique snowflake.
    • Ask yourself why you have so many threads. If we do a thing that is hard for us to comprehend, it follows we'll have a hard time debugging it. Perhaps you could use fewer separate threads. What exactly does your program do? What does each of the threads do?
    • Show us your code. For some of us, our job's been fixing problems like this for years.
    • What are the things your threads share? Do they have to be shared by multiple threads? Are they adequately protected from shared thread access? If you think the answer to both of those questions are "yes", show us what you're doing and we'll find the one where you're wrong.
    • Seriously, for every thing a thread can access, ask yourself if you really know how many threads can change it, if only one can change it at a time, and if it's a bad thing that one might have to wait on another. It is very likely contention over a shared resource is the root of our problem. Show us your code, we're good bloodhounds. (Well, not me. I'm a pony. But you get the metaphor.)
    • Instrument the heck out of your code. I don't mean just in Try..Catch blocks. Put a line that writes what it is doing (and the value of Thread.CurrentThread.ManagedThreadId) everywhere that makes sense. That way, you'll have a good idea of what each thread was doing when it crashes. The problem being this can change timings enough to make the problem go away.
    • Are all of your threads important? If not, disable some of them. See if the problem goes away, or is less frequent. Then, disable some other subsset. Try and see if there's one particular thread that seems to cause the issue.
    • Have you showed your code to someone else? A lot of times we overlook errors in our own code that are obvious to other people. Nothing makes a programmer happier than telling another programmer he is wrong.
    • Can't disable a thread? Scatter some short Thread.Sleep() calls in a few methods, or preferably one at a time. 100ms or so, not something drastic like 5s.
    • Blame Microsoft and VS 2015 and spend a few hours installing a different VS and removing any .NET 4.6 dependencies.

    Some of these will be more successful than others, generally those are the ones that start with "show someone else your code."

    Debugging an error without the IDE is the same as with the IDE, you just have worse tools. It's a process of eliminating code that you're pretty sure isn't causing the problem so you can focus on smaller bits of code that might have the problem. In a synchronous program, this is much easier, but having threads doesn't change the game. It changes how you play.

    You need to start eliminating code that isn't causing the problem. Or isolating bits so that you're only dealing with a handful of threads. The first step is identifying every place where multiple threads access a thing. Is it hard to figure that out? There's part of your problem. If it's hard to keep track of it, show us that code and we will show you how to make it easier. Or we might tell you ways the threads don't have to share information. But there's dozens of tricks. I'd rather tell you the ones you need than spend a week trying to write an exhaustive tutorial. In general:

    • It is never safe to access a Form or any of its Controls from a thread. The same goes for anything COM, like Office Interop. It's best to write your code so the threads can't even see Forms and Controls and Office. They should produce data, and raise events or set flags, and code within the form should transfer to the right thread, then act on the data.
    • If n threads share a variable, I like to put both the threads and that variable inside a single class so I understand they are related.
    • If more than one thread is going to access a variable, it should be protected by SyncLock. No exceptions. It is never right to not have a SyncLock around a variable many threads might access.
    • If a method can be called by two threads at once, it also needs strategic SyncLock statements. If it's going to work with a list, another class, etc. and change multiple properties, it is not enough for the class/list to have SyncLock.
    • Signaling thread state with Booleans is risky, especially if some other thread waits for the Boolean to be true. Use classes like ManualResetEvent for this.
    • When you have threads that wait on each other, or shared methods with SyncLocks, you have to walk through EVERY call chain to see if you can cause deadlock. It is easier if you can minimize SyncLocks by minimizing shared resources.
    • It's all around easier to work with immutable objects that are copied than mutable objects shared by reference.
    • It's easier if data consumed by threads is held in queues they poll rather than sent directly to them.
    • It's easier if you have a way to execute the process synchronously, with no threads, and automated tests that verify it works if everything happens in the "right" order.
    • It's easier if you can turn some threads off, though not every program has optional threads.

    I can expound on a lot of those, but most take a page or more to really explain themselves, and there's generally dozens of good tutorials for any of them. But that list doesn't help you now: it tells you how to avoid this situation in the future.

    I want to expound on one of them, but the coffee's done and I'm 15 minutes late for work already. I'll be back.
    This answer is wrong. You should be using TableAdapter and Dictionaries instead.

  16. #16
    You don't want to know.
    Join Date
    Aug 2010
    Posts
    4,578

    Re: Best practices for debugging error that happens outside IDE in Multi-Threaded app

    OK, what I want to talk about is one example of how you can think about dealing with shared resource access between threads in a way that isn't as risky.

    The main idea is you want to draw a line in the sand and think differently on boht sides of it. On one side, threading is important, and you're thinking about when to use SyncLock statements, thread marshalling, and other thread-related ideas. On the other side, you're only allowed to use things that are guaranteed thread-safe. You hide the ugly threading code behind friendly methods that do the work for you, then write your real code in terms of that thread-safe code.

    Let's pick a common task that requires coordination between two threads. Thread A is generating some data, it is the "producer". Thread B does something with that data. It's called the "consumer". We can connect them in many ways, some better than others, all varying in utility depending on how the rate of production compares to the rate of consumption. I like to use a queue in this case. That's because I know they're one of the best synchronization data structures. Let's talk about lists, so we can learn why queues are better.

    The idea is I have a List(Of T) that is shared between the two threads. When the producer creates an item, it adds it to the list. The consumer, periodically, checks the list. If it's not empty, the consumer takes some data and does something with it. "Periodically" might mean the producer notifies the consumer there is data. Or the list might notify. Or it might be "every 5 seconds". Those choices are irrelevant to what I'm getting at.

    How might we write the consumer thread? Here's the worst way:
    Code:
    While NotCancelled
        If _dataList.Count > 0 Then
            Dim myData = _dataList(0)
            _dataList.RemoveAt(0)
            Work(myData)
        End If
    
        Thread.Sleep(1000)
    End While
    This pays no concern to threading issues, and will fail miserably if there are multiple consumers, and can fail with only one. Why? To figure out why we have to do something you get used to when writing threads. The game is, "What if two threads execute this code at the same time?"

    Let's say two consumers wake up at the same time, and there is one item in the queue. Both see that there is 1 item. Both take that one item. Consumer1 removes the item slightly before Consumer2. When Consumer2 tries to remove it, an exception is thrown. This is only one of many failures, including "1 removes the item after 2 checks count but before it gets it". Also, bad things can happen if List.RemoveAt() is still executing while the producer adds an item.

    (This is a hard game, and you have to do it for every combination of every line. "What if 1 executes line 1 before 2 does line 2? What if it's before 2 does line 3? ...". This is why you follow the practices I'll suggest, to minimize the need to play this tedious game.)

    The solution is a SyncLock statement. This statement puts up a barrier through which only one thread may enter at a time. Think of it like a bouncer at a club, whose job is to make sure every person on the dance floor has the same color shirt. Each thread has a different shirt color, so only one thread gets in at a time. We could do it a lot of wrong ways, here's one that's almost correct:
    Code:
    While NotCancelled
        SyncLock _dataList
            If _dataList.Count > 0 Then
                Dim myData = _dataList(0)
                _dataList.RemoveAt(0)
                Work(myData)
            End If
        End SyncLock
    
        Thread.Sleep(1000)
    End While
    Now we play our game again. Consumer1 and Consumer2 wake up at the same time, and there's 1 item. Consumer1 reaches the SyncLock first, and enters. Consumer2 has to wait. Consumer1 checks the count, takes the item, removes the item, then does its work. When it's finished, it exits the lock. Consumer 2 immediately gets in, sees there's no data, then sleeps. What if the producer made an item in between? Well, if it's behaving and using the list as the SyncLock key, it has to wait. Either Consumer2 gets in first, or Producer gets in first. Either way, we're safe.

    Why is this almost correct? What if Work(myData) takes a long time? Part of why we made a thread is it probably does. So if we include it inside the lock, we've made it so all our Consumers have to stand in line and execute serially, not in parallel. It turns out it's safe to do the work outside of the lock, becuase we already have removed the data so no other thread can get it. We want to protect the list, we don't have to protect the rest. So this is correct:
    Code:
    While NotCancelled
        Dim myData As Data = Nothing
        SyncLock _dataList
            If _dataList.Count > 0 Then
                myData = _dataList(0)
                _dataList.RemoveAt(0)
            End If
        End SyncLock
    
        If myData IsNot Nothing Then
            Work(myData)
        End If
    
        Thread.Sleep(1000)
    End While
    The purpose of this boring exercise is to demonstrate how hard we had to think about a very simple task. All we want to do is take the first item and remove it if there is one. But we have to think about locking, and other threads, and the producer, etc. This isn't saying "preheat the oven", this is discussing the details of particular knobs.

    The first thing to observe is this list should be a queue. Queues are a first-in-first-out data structure, FIFO. The Enqueue() method puts something at the "end". The Dequeue() method takes something from the "front". It's more efficient at both of these things than a list. But it's still not thread-safe, so we still have to write this:
    Code:
    While NotCancelled
        Dim myData As Data = Nothing
        SyncLock _dataQueue
            If _dataQueue.Count > 0 Then
                myData = _dataQueue.Dequeue()
            End If    
        End SyncLock
    
        Work(myData)
    
        Thread.Sleep(1000)
    End While
    This still makes me frown. We're more efficient, but we're still thinking about threads. .NET Framework 4 brought us System.Collections.Concurrent, classes designed specifically to control access by multiple threads at once. ConcurrentQueue(Of T) is far more convenient for this. Using it, our correct code can look like:
    Code:
    While NotCancelled
        Dim myData As Data = Nothing
        If _dataQueue.TryDequeue(myData) Then
            Work(myData)
        End If
    
        Work(myData)
    
        Thread.Sleep(1000)
    End While
    Now we're not thinking about the other threads, but we're just as safe as we were before. And TryDequeue() isn't doing anything fancy we couldn't have written ourselves before .NET 4. Let's pretend we are, and for some reason we need a List. I could write this function:
    Code:
    Function DequeueItem(ByRef item As ???) As Boolean
        SyncLock(_queue)
            If _queue.Count = 0
                item = Nothing
                Return False
            End If
    
            item = _queue(0)
            _queue.RemoveAt(0)
            Return True
        End SyncLock
    End Function
    This makes the correct code:
    Code:
    While NotCancelled
        Dim myData As Data = Nothing
        If DequeueItem(myData) Then
            Work(myData)
        End If
    
        Work(myData)
    
        Thread.Sleep(1000)
    End While
    There is now no unsafe way to check for new data or to get it. This is how I try to write code. The more I push the thread code in to very small corners that are easy to analyze, the fewer mistakes I can make. The less I have to think about threads when writing my code, the more correct I will be.

    When code looks more like the examples at the top, it is harder to find and diagnose threading issues. When code is designed more like the examples at the bottom, it's less likely that the issues will exist. It takes a LOT of thought and hard knocks to get there.

    The more you think, the more mistakes you will make. It's our job to think very hard in some places, so we can be lazy in others. The lazy code's never where we make mistakes.
    Last edited by Sitten Spynne; Mar 17th, 2016 at 10:40 AM.
    This answer is wrong. You should be using TableAdapter and Dictionaries instead.

  17. #17

    Thread Starter
    Hyperactive Member
    Join Date
    May 2007
    Posts
    300

    Re: Best practices for debugging error that happens outside IDE in Multi-Threaded app

    Wow! So much help! Thank you everyone.

    First, I want to say that "fraught with threads" I think was a bad expression. There's really only 2-3 threads. I am doing asynchronous I/O (MIDI and sometimes UDP) so no choice there.

    Meanwhile, whilst putting together a reply and some code to show the problem, it hit me in the face. I know better, but for some reason, I had a form on another thread being opened inside the catch block. Stupid beginner error, as I learned all about the forms vs. threads thing years ago and solved all of those cross-thread communication issues.
    Just a dumb mistake after all that.

    Now, my issue is to figure out how to show the user that there is a problem that the try...catch block caught, in my case, closing the main window and going back to the settings page.

    I'm sorry to make everyone write so much when in the end it was just a stupid mistake on my part. Interesting though that the error doesn't surface in the IDE, only in the .exe when run standalone.

    Thanks again for everyone's suggestions and input!

  18. #18
    You don't want to know.
    Join Date
    Aug 2010
    Posts
    4,578

    Re: Best practices for debugging error that happens outside IDE in Multi-Threaded app

    Those are difficult to deal with. When you use the wrong thread, behavior is unpredictable. I've seen all of the following symptoms:
    • Nothing happens.
    • It works.
    • The application crashes with no explanation.

    This isn't something you can catch with an exception. It's something that gets fixed by discipline that gets drilled into your head by every hour you spend debugging it. I've lost hundreds of hours to these mistakes, which is why I get grandiose and paranoid with my threading architectures. My rule of thumb is "if it happens on another thread, it belongs in another class." I don't always follow it, but when I break it I work more slowly.

    But it's also why I /love/ the Task API, it's a lot easier to keep things straight with Async, Await, and tasks.

    If the problem was that you were dealing with a form from the wrong thread, there's not a Try..Catch you can create that will diagnose the problem. The solution, in this case, is to make sure it can't happen. (Yes, in debugging, a CrossThreadException can be thrown. That's a special facility of the Visual Studio debugger, and I'm not sure if it works outside VS. But it's really not something I'd catch, it's something I'd rather make impossible.)
    This answer is wrong. You should be using TableAdapter and Dictionaries instead.

  19. #19

    Thread Starter
    Hyperactive Member
    Join Date
    May 2007
    Posts
    300

    Re: Best practices for debugging error that happens outside IDE in Multi-Threaded app

    Great advice. Thank you!

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  



Click Here to Expand Forum to Full Width