VB.NET 2010 calling c dll - PInvoke error

[jbmckim]

I've recently gone from VS2008 to VS2010. I've learned that there are some differences in the defaults with PInvoke and I seem to be stumbling over them.

Here's the VB code:

Public Declare Function open_device Lib "C:\windows\system32\MyDLL.dll" (ByVal index As Int32, ByRef errorCode As Integer) As Integer

...


  Dim DoneOrNot As Integer = 0
  Dim iWorking As Int32 = 0
  Dim errorCode As Integer

...

  DoneOrNot = open_device(iWorking, errorCode)

The c code:

int STD_CALL
open_device(int index, int *error_code) {
    deviceWrapper *wrapper = deviceWrapper::getInstance();

    return wrapper->opendevice(index, error_code);
}

All this in the debugger throws the ever popular, helpful and specific:

"Managed Debugging Assistant 'PInvokeStackImbalance' has detected a problem in 'C:\blahblahblah\Visual Studio 2010"

Other ragging:

1) As mentioned above this was no problem in VB.NET VS2008 due to a difference in defaults.

2) I know I can change the defaults for the debugger but since this is a commercial app, I'd like to get the type "imbalance" straightened...Even though I've run this for days without any discernible side affects.

In addition to any specific guidance with this issue, I'd also gladly give my first born for a table mapping VB.Net data types to c & c++ types - that avoid this error! I've found a couple for c++ types (MS implementation) but nothing generic. This is a rare case where Google has failed miserably.

Last, I've experimented already with various uses of Int32 and IntPtr. So far no luck.

Thanks.

[Sitten Spynne]

Oh no, this problem. I maintain a couple of libraries that make extensive use of P\Invoke and this one hurts.

The defaults did not change between VS 2008 and VS 2010. The default calling convention is and always has been CallingConvention.Winapi because this makes calling Windows API functions the most convenient. So why do you only get this error in VS 2010?

It's not really an exception. It's hard to tell the difference, but what pops up is called a Managed Debugging Assistant (MDA). When you debug your application, Visual Studio watches for a bunch of problems and throws one of these up if they happen. These are problems that are indicative that something's going wrong in your code but may not necessarily crash the program because Windows is pretty hardy.

What you need to do is change the calling convention. You can't do this using the Declare keyword; you have to use the DLLImport attribute to change the calling convention. (I could be wrong on that, but this is what I know.) It's not that different, but it's helpful to import System.Runtime.InteropServices to keep it short. With that import in place, this ought to be an appropriate way to declare the function you mentioned:

<DllImport("MyDLL.dll", CallingConvention:=CallingConvention.StdCall)>
Public Function open_device(ByVal index As Int32, ByRef errorCode As Integer) As Integer
End Function

Note you may have to put the path to the DLL as before. You might also want to clean up your types; if you use Int32 in one place you ought to use it in all of them for consistency (I'm not really afraid MS will change it out of the blue but consistency is nice.)

I'm not 100% sure that will solve your problem because the information I have indicates that the WINAPI calling convention is actually defined to be the same as stdcall; it's worth a shot.

Also, to answer the second question, this language equivalents table might be useful. It's missing some of the more annoying ones like DWORD but search engines usually suffice for that one.

[jbmckim]

Thanks,

EDIT: OK, DLLImports have to be 'Shared' to help Bill Gates get past the idea that we're not calling managed code

I'd tried this earlier with this as well as CallingConvention:=CallingConvention.Cdecl

Per:

    <DllImport("MyDLL.dll", CallingConvention:=CallingConvention.StdCall)> _
    Public Shared Function open_device(ByVal index As Int32, ByRef errorCode As IntPtr) As Integer

    End Function

Now I'm getting "'System.Runtime.InteropServices.DllImportAttribute' cannot be applied to instance method." as a syntax error - which I'm just not seeing (eyes are crossing now).

- to my earlier post, I had read someplace that the default for the debug time had changed from 2008 to 2010 in so far as handling pinvoke imbalances was concerned. sounds like you're saying that's not true.

- also, I do take your point about data consistency. again, as I've been thrashing this for a bit now, I went through a spread sheet of int and intptr values and didn't apparently get them all back in the example.

this really is unnecessary MS suckage I think.

[jbmckim]

So far, no good with DllImport

[Sitten Spynne]

Well, AFAIK "STD_CALL" isn't really the standard way to denote that calling convention; the references I was looking at called it "__stdcall". You could look around in the headers/source to see if maybe "STD_CALL" is a macro definition. (Though C/C++ are not my strong point and it's possible that's a standard alias.)

You could also try all of the calling conventions until you find one that works; I don't recommend this but hey one of them has to be right.

RE: the Shared keyword it makes sense if you think about it. If it didn't have Shared in front, it'd be an instance method, which could be weird from an internals standpoint. I believe C# lets you create instance methods with DLLImport applied, but it's generally held as a bad practice. I didn't have Shared in my example because I was fiddling around in a console application, which means I was in a Module, which means everything's Shared by default.

Also I don't think Bill Gates has written code or participated in any of the architecture discussions for the Visual Basic language in many years. Blame whoever works on that team these days.

[jbmckim]

Really MS?

http://msdn.microsoft.com/en-us/library/ff361650%28v=VS.100%29.aspx

It works now. I'm just sayin' that if you're going to change something that can effectively kick the cord out of the wall on purpose, it seems like you'd also provide a utility to AT LEAST read the .h header of the dll and provide proper typing for a .NET call. If it's not deterministic, just PICK ONE or list all possible...like intellisense was intended to do.

If anyone can help with compatible parameter typing (including ByRef's to c ptr's) in VB.Net, from what I've seen over the past couple days, this would be much appreciated by a great many people.