[RESOLVED] A Hidden Problem

[Shaggy Hiker]

I have a program that is made up largely of plugins. An event gets raised from one, and others can handle the event. That generally works well, but sometimes it can be difficult to debug. If one of the plugins is out of date, it may be that VS can't show the code.

That seems to be what is happening in this case: An event gets raised, then everything freezes. I have tracked it down to the code waiting on a SQL call. The SQL call is very simple, and would normally return results almost instantly, but there's a deadlock that keeps the query from being processed. There are two possible causes for this, and both would be really easy to solve...so long as I could tell who was making that SQL call.

To be clear, I do know who is ultimately making the SQL call, but somebody is triggering a chain of events that leads to the SQL call, and it happens when a particular event is raised. This should then be simple. All I need to do is look at the stack trace. The stack trace shows a series of methods leading up to the SQL call, and it shows the event being raised, but between those two is [External Code].

That means that there is some part of my program that VS feels it can't identify, which is almost always because the latest version of the source code doesn't match the code in the dll that VS has loaded for it. This is annoying in a plugin system because there are some 30 plugin dlls, along with four or five that are directly referenced by the main program. One of those is the culprit, but which one?

The first thing I did was remove ALL the plugins that I could, which reduced the number down to five. Of those, four of them do not handle the event that triggers this cascade of events, so they seem unlikely to be candidates. The fifth is 'special'. For one thing, I know that I can see the code in that one, because it's the one that causes the event to be raised. It doesn't raise the event, but it causes the event to be raised, and it handles the event. I've stepped through to the point the event is raised, and I've had a breakpoint in the event handler. That breakpoint isn't reached.

Interestingly, there are several other plugins that would cause that event to be raised. I haven't tried ALL of them, but I've tried a couple, including one that is very similar to the one that causes the problem, and in every case, all is well. In fact, the plugin that causes the problem then handles the event just fine. I can step through the event handler for that plugin when some other plugin triggers the event, so not only does that plugin seem fine, but I know that VS can see it and will show me code from it.

So, what I'm looking for is this: Can anybody suggest any means to figure out what is behind that [External Code]? I've removed every plugin except the few I'm testing. I've rebuilt every dll referenced by the main program. I've gone hunting for anything in the remaining code that handles that event, and I've found nothing. If I could figure out who is handling that event, I believe I could solve it, but outside of the plugins, I have yet to find anybody who handles the event. So, I'm really looking for a means to figure out who is handling the event when the stack trace only shows me that somebody is, but not who that somebody is.

[Shaggy Hiker]

Ever have one of those problems....

So, one might say that I solved this, but one would be wrong, because one is lazy.

I began to focus increasingly on the fact that I had two very similar plugins, one worked and one did not. That strongly suggested that the one that did not had a bug in it, though I couldn't explain how this manifested in the form of a deadlock coming from unrecognized code. Still, I decided to carefully study what was different between the code in the two plugins.

On the face of it, there was no difference. The same methods were called, and the arguments were essentially the same. The only obvious difference was something that shouldn't matter (and didn't). But then I stepped through the code a bit more and noticed that prior to the problematic event being raised, both the working and non-working plugins would raise some number of other events. The working plugin raised roughly 8 of one type, and one each of two other types. The non-working plugin raised the roughly 8 of one type, but not the other two. I got to studying that, and realized that there was a very simple bug that had caused some incorrect behavior in the non-working plugin. A simple fix got the non-working plugin to also raise the two additional events, and that fixed the underlying problem.

In other words, adding two more events, which seemed unrelated to the problem event, and happen well ahead of the problem event, caused the problem event to no longer be a problem. That's a truly absurd solution. I know the problem was a deadlock on a DB query, I know roughly what was causing the deadlock, I just couldn't figure out what code was initiating the sequence of calls that resulted in the deadlock. Whoever was initiating that, they shouldn't have been. And yet, adding two more events that don't seem related, has caused whoever was initiating that sequence to not initiate that sequence.

There's a bug. The bug is still there. The bug would be easy to fix if I could figure out which dll the bug was in. The bug is related to handling one specific event raised by one specific class. I can narrow down the potential candidates to about half a dozen, but I can't seem to find the one that is the culprit. That bug has scuttled out of sight, to my very great annoyance.

[OptionBase1]

Could it simply be a timing thing, and raising those two additional events takes enough time for some earlier call to finish?

[Shaggy Hiker]

I thought of that, as well. There isn't any threading going on, at the time. It's all synchronous, so it's a bit hard to see how that could be happening....but, I'd tried pretty much everything else, so I decided to see what happened if that problematic event wasn't raised, or if I got to the line that showed the deadlock if I put a breakpoint on the innocuous line prior to raising the event.

In both cases, the answer was negative. If the problematic event was commented out, things proceeded apace...though, since some important stuff did get skipped when the event didn't get raised. No deadlock, though. Also, if a breakpoint was placed immediately before the event was raised, the line that showed the deadlock was never reached. The sequence of methods that led to that simply didn't get called.

However, now that you mention it, I did find something out while finding the bug that I DID fix. That method got called once when the problematic plugin was bugged, but twice in the non-bugged plugin. These method calls happened well ahead of raising the problematic event, but I didn't study that enough. I didn't think it was important, but it probably is. The method is getting called as a result of some other event being raised. It appears that the earlier call may be solving the problem of the deadlock for the later call. I still don't know who is making the call, though, but it's a point to focus on.

[Shaggy Hiker]

This has become a totally different question, so I'll start a new thread for it.

[Shaggy Hiker]

I found the culprit. The issue was that I wasn't able to step through handlers of that particular event. I don't know why that is, so I started a thread on it here:

https://www.vbforums.com/showthread....51#post5585051

However, I also found this blog post:

https://blog.slaks.net/2011/07/track...strations.html

The key point in there comes in the final paragraph. I had never set a function breakpoint on an event handler accessor, and when I did...well, it was interesting. For one thing, code pauses where there isn't a visible breakpoint, so you have to recognize that YOU did that, without setting a visible breakpoint. I suspect that if you had several function breakpoints set, it would confuse the holy heck out of you, but if you only have one, then it HAS to be that. For that reason, you do have to be somewhat careful when setting function breakpoints like that.

In this case, execution broke in several places that really surprised me. It probably shouldn't have surprised me. If you add an event handler for the event you have the breakpoint on with AddHandler, then execution will break at that line. But what behavior should you expect if you declared the Object WithEvents and used a Handles clause on the event handler method? The break is going to happen when the event is wired up, but when does that happen?

It isn't always clear when that happens. The first breakpoint I reached was on this line:

GrowthTool = New GrowthCore.GrowthSegmentManagement(globalERaiser, mSchemaKey, Nothing)

I had totally forgotten that the GrowthSegmentManagement class had to handle the Move event, but it totally makes sense that the event handler is hooked up when the GSM class is created. It has the object that raises the Move event declared WithEvents and has a Handles clause, so the event handler gets hooked up when the GSM class is created.

The next breakpoint was far more surprising. The line it broke on was this:

 o.Initialize(globalERaiser)

o is 'some object that implements a certain interface', and Initialize is not a constructor, but just a method in that interface, so why is the event handler only wired up then? Inside the Initialize method for the class is this line:

mIFBO = mERaiser.IFBO

The mIFBO variable (it is an acronym that has meaning that is well documented for the program, so it's nowhere near as opaque a name as it might first appear) is declared WithEvents, but doesn't get a value until that line. Therefore, when an actual object is assigned to it, that is when the event handler gets hooked up.

So, by putting a function breakpoint on the accessor of the event in question, execution was paused at every line where an event handler was added, even though I wasn't always able to see who was adding handlers, and wasn't always able to step through the handlers when the event was raised.

I found that both obscure, and quite useful for this particular, narrow, situation. Hope somebody else finds it to be of value.