C++ DLL from VB6 (Run-time error ‘49’: Bad DLL calling convention)

[jkeller]

Could you throw a messagebox in the .NET code and see what values the two IntPtr hold?

Second, try:

Public Sub LoadFromFile(ByVal BmpFile As String)
    Dim Template As Integer
    Dim Mask As Integer
    Call SomeFunction (BmpFile, VarPtr(Template), VarPtr(Mask), Width, Height)
End Sub

Try that with both Template and Mask as Integers and as Longs.

I did display the IntPtr values in the .NET version when I was coding that; they were indeed big negative numbers (memory addresses) which eventually dereferenced fine using the Marhal calls.

The VarPtr looks interesting. I tried it using your syntax, and this slightly modified version:

Call SomeFunction (BmpFile, VarPtr(TemplatePtr), VarPtr(MaskPtr), VarPtr(WidthPtr), VarPtr(HeightPtr))

I still get the '49' error for both calls.

I'll research VarPtr and see if I can learn more about it though; it does sound promising, thanks.

[jkeller]

I added this declaration:

Declare Function VarPtrArray Lib "msvbvm60.dll" Alias "VarPtr" (Var() As Any) As Long

And tried various versions such as this:

    Dim TemplatePtr As Long
    Dim MaskPtr As Long
    Dim WidthPtr As Long
    Dim HeightPtr As Long
    
    TemplatePtr = VarPtrArray(PackedTemplate())
    MaskPtr = VarPtrArray(PackedMask())
    WidthPtr = VarPtr(Width)
    HeightPtr = VarPtr(Height)
    Call SomeFunction(BmpFile, TemplatePtr, MaskPtr, WidthPtr, HeightPtr)

Same error in all cases so far.

[Comintern]

Have you tried passing a string pointer instead of the actual string (probably a long shot)?

Private Declare Function SomeFunction Lib "XYZ.dll" ( _
    ByRef FileName As Long, _
    ByRef PackedTemplatePtr As Long, _
    ByRef PackedMaskPtr As Long, _
    ByRef Width As Long, ByRef Height As Long_
    ) As Long

Call SomeFunction(StrPtr(BmpFile), TemplatePtr, MaskPtr, Width, Height)

Also, try discarding the return value by declaring it as a Sub instead of a Function.

[EDIT] This link may shed some additional light...

[jkeller]

Very helpful, thanks!

The DLL developers confirmed they are using _cdecl, so they are going to have to change how they are doing it.

From your link:

It is not possible to directly call a C function in a DLL if that function uses the _cdecl calling convention. This is because Visual Basic uses the _stdcall calling convention for calling functions. This is a problem because if _cdecl is used, the calling function is responsible for cleaning up the stack. However, if _stdcall is used, the called function is responsible for cleaning up the stack.

NOTE: An .EXE file created in Visual Basic will allow you to call a DLL function that has been declared with the _cdecl calling convention without an error. It is only when you try to call such a function when running a program from the Visual Basic IDE, that Visual Basic generates the following error:

Run-time Error '49': Bad DLL Calling Convention

The fact that the EXE version allows you to call such functions has been confirmed to be a bug by Microsoft. You should not rely on this behavior as this might change in future versions of Visual Basic.