C++ DLL from VB6 (Run-time error ‘49’: Bad DLL calling convention)

[jkeller]

Hello. I have a C++ DLL ("unmanaged" I think, though I'm not sure exactly what that means).

I need to call it from both VB.NET 2003, and VB6. I can do so from VB.NET (see below). However, I can't figure out how to call it from VB6.

Here's the definition of the function I need to call:

int SomeFunction(const char *fileName,
                        unsigned int **packedTemplate,
                        unsigned int **packedMask, 
                        int *width, int *height);

I've tried various declarations such as:

Private Declare Function SomeFunction Lib "XYZ.dll" ( _
    ByVal FileName As String, _
    ByRef PackedTemplatePtr As Variant, _
    ByRef PackedMaskPtr As Variant, _
    ByRef Width As Long, ByRef Height As Long _
    ) As Integer

But I always get: Run-time error ‘49’: Bad DLL calling convention

I can call this from VB.NET using this declaration:

   
    <DllImport("XYZ.dll")> _
    Private Function SomeFunction _
       (ByVal FileName As String, _
        ByRef PackedTemplatePtr As IntPtr, _
        ByRef PackedMaskPtr As IntPtr, _
        ByRef Width As Integer, ByRef Height As Integer) _
       As Integer
    End Function

This requires some memory management, which I do using Marshal:

        Dim TemplatePtr As IntPtr '= Marshal.AllocHGlobal(TemplateSize)
        Dim MaskPtr As IntPtr '= Marshal.AllocHGlobal(TemplateSize)

        SomeFunction (BmpFile, TemplatePtr, MaskPtr, Width, Height)

        Marshal.Copy(TemplatePtr, PackedTemplate, 0, TemplateSize)
        Marshal.Copy(MaskPtr, PackedMask, 0, TemplateSize)

Any suggestions greatly appreciated, thanks!

[Paul M]

unmanaged code is, code that does is executed by the CPU. Where as managed code (C#, VB.NET, Java) have their code executed via a virtual machine. You may have heard of the CLR? Well that is Microsoft's Virtual Machine that manages .NET code and executes it.

As for your problem with the calling of a function from a C++ DLL, do you think you could upload or link us to a download so we can test for ourselves

[Disiance]

I don't believe you can pass Variants to DLLs. Change

    ByRef PackedTemplatePtr As Variant, _
    ByRef PackedMaskPtr As Variant, _

to

    ByRef PackedTemplatePtr As Long, _
    ByRef PackedMaskPtr As Long, _

[jkeller]

Unfortunately, it's a 3rd party licensed DLL that requires a USB dongle to run, so uploading it is not an option. I think all the relevant parts of my code are in the OP above.

As for the type Long suggestion -- I've tried Byte, Integer, Long, Byte(), Integer(), Long(), Variant, and Any; but I get the same error 49 in each case.

I'm new to both VB.NET and VB6, so I may be missing something obvious though. Does VB6 have something like VB.NET's IntPtr (which works fine for me)?

[Disiance]

No, VB6 doesn't have the IntPtr. Could you post the VB6 code you're using to:
1) Define the PackedTemplatePtr & PackedMastPtr variables
2) Assign the value to those variables
3) Line where you are calling the function and getting the error

I see the code here for VB.NET, but not the VB6 code. Maybe I'm just blind right now.

[jkeller]

Certainly. I've tried lots of variations with different types, but here's my "best guess" approach. It crashes on the "Call SomeFunction" line with the error 49 noted above.

Once the call succeeds, I'll still need to move the data into my PackedTemplate and PackedMask arrays (as I did using Marshal with VB.NET - the code that works), but I have not got that far yet.

Option Explicit

Private Const TemplateSize As Integer = 1200
Private Const TemplateLast As Integer = TemplateSize - 1

Private PackedTemplate() As Byte
Private PackedMask() As Byte
Private Width As Long
Private Height As Long

Private Declare Function SomeFunction Lib "XYZ.dll" ( _
    ByVal FileName As String, _
    ByRef PackedTemplatePtr As Long, _
    ByRef PackedMaskPtr As Long, _
    ByRef Width As Long, ByRef Height As Long _
    ) As Integer

Private Sub Class_Initialize()
    ReDim PackedTemplate(TemplateLast)
    ReDim PackedMask(TemplateLast)
    Width = 480
    Height = 20
End Sub

Public Sub LoadFromFile(ByVal BmpFile As String)
    Dim Template As Long
    Dim Mask As Long
    Call SomeFunction (BmpFile, Template, Mask, Width, Height)
End Sub

[jkeller]

And just in case this helps, here's my VB.NET version, which works fine:

Imports System.Runtime.InteropServices

Public Class MyTemplate
    Public Const TemplateSize As Integer = 1200
    Public Const TemplateLast As Integer = TemplateSize - 1

    Public PackedTemplate As Byte()
    Public PackedMask As Byte()
    Public Width As Integer
    Public Height As Integer

    Public Sub New(ByVal BmpFile As String)
        ReDim PackedTemplate(TemplateLast)
        ReDim PackedMask(TemplateLast)
        LoadFile(BmpFile)
    End Sub

    Private Sub LoadFile(ByVal BmpFile As String)
        Dim TemplatePtr As IntPtr
        Dim MaskPtr As IntPtr

        SomeFunction(BmpFile, TemplatePtr, MaskPtr, Width, Height)

        Marshal.Copy(TemplatePtr, PackedTemplate, 0, TemplateSize)
        Marshal.Copy(MaskPtr, PackedMask, 0, TemplateSize)
    End Sub

    <DllImport("XYZ.dll")> _
    Private Function SomeFunction _
       (ByVal FileName As String, _
        ByRef PackedTemplatePtr As IntPtr, _
        ByRef PackedMaskPtr As IntPtr, _
        ByRef Width As Integer, ByRef Height As Integer) _
       As Integer
    End Function

End Class

[Disiance]

Okay, first, I was wrong about the datatype, your Declare statement should have the PackedTemplatePtr and PackedMaskPtr as Integer, not Long (this goes for the Template and Mask variables as well). Try that and see what happens.

Am I right in assuming the called function will return the pointer into those passed variables?

[jkeller]

I made the change from Long to Integer for those four. I am getting different behavior (which is probably good). Where I used to get the run-time error 49, the VB IDE now crashes (which is bad, but it may indicate progress). So it's possible I may be getting past where I got before.

To complicate things, I just received (and am using) a new version of the DLL (supposedly with lots of bugs fixed); I'll retest with the old version to eliminate that variable.

The called function returns data via the four output parameters (template, mask, width, and height). The memory for the first two must be allocated inside the DLL, since my VB.NET code just sends IntPtr, and works fine. The actual return value of the function is not documented; I assume it's probably an error code.

Thanks for the help Disiance. If you ever need a hand with some Java code, maybe I can return the favor.

[jkeller]

Ok, using the old DLL, and the Integer types, the IDE crashes also. So that's good (well, you know...)

[Disiance]

The DLL wouldn't internally handle the memory allocation, but probaby just passes the information to Windows and has it manage it... Technically, you could create bitmap object through Windows with Program1, pass the pointer to Program2, and let Program2 do things with it. Not the smartest or cleanest, but possible.

Crashing IDEs are so much fun with DLLs & APIs. *rolls eyes* I'm actually struggling with this on my own DLL excursion.

Let's see... VB now thinks you're passing things right, but it's not sending something correctly. Is there a chance you can add the DLL as a reference and do something like:

Dim MyObj As New DLLClassName
MyObj.SomeFunction(var1,var2,var3,var4)

Thanks for the JAVA offer, that is definitely my enemy language

[jkeller]

Hmm... I *think* it's a static function, so I don't think that approach will work.

I'm not convinced that I was using the correct types yet; I think the crashing was caused by something else (not sure what).

I found this web-page, which gives a pretty good description of what I'm trying to do: http://support.microsoft.com/kb/106553

Based on that, I'm using the declaration:

Private Declare Function SomeFunction Lib "XYZ.dll" ( _
    ByVal FileName As String, _
    ByRef PackedTemplatePtr As Integer, _
    ByRef PackedMaskPtr As Integer, _
    ByRef Width As Integer, ByRef Height As Integer _
    ) As Integer

One question I still have: How should the VB6 declaration indicate this is a static function? In VB.NET, I use the "shared" keyword (though admittedly, the VB.NET code seems to work fine whether I use the "shared" keyword or not).

Based on the linked docs above, it seems like one of these approaches should work:

Public Sub LoadFromFile(ByVal BmpFile As String)
    Dim Template As Integer
    Dim Mask As Integer
    Call SomeFunction (BmpFile, Template, Mask, Width, Height)
End Sub

Public Sub LoadFromFile(ByVal BmpFile As String)
    Dim Template(TemplateLast) As Integer
    Dim Mask(TemplateLast) As Integer
    Call SomeFunction (BmpFile, Template(0), Mask(0), Width, Height)
End Sub

Both versions (and lots of other variations I've tried) still give me the error 49 though.

[Disiance]

Curious... I'm not at all used to any .NET langauge (I go up to VB6 and stop). I've translated from .NET to VB6 once, actually, it was C# to VB6. Though I know the basics of C++ and a couple other langauges, so I'm not in the dark. What does VB.NET have in the way of pointers? Do you know how they relate to variables and such? Actually, what is an IntPtr... Okay.

It's your second idea that prompted all this, so I blame you (and bear with me)

Could you throw a messagebox in the .NET code and see what values the two IntPtr hold? Probably won't avail to anything, but it might. Using an Integer data type just doesn't seem right for anything pointer related.

Second, try:

Public Sub LoadFromFile(ByVal BmpFile As String)
    Dim Template As Integer
    Dim Mask As Integer
    Call SomeFunction (BmpFile, VarPtr(Template), VarPtr(Mask), Width, Height)
End Sub

Try that with both Template and Mask as Integers and as Longs.

[Comintern]

This should be the correct declaration:

Private Declare Function SomeFunction Lib "XYZ.dll" ( _
    ByVal FileName As String, _
    ByRef PackedTemplatePtr As Long, _
    ByRef PackedMaskPtr As Long, _
    ByRef Width As Integer, ByRef Height As Integer _
    ) As Integer

Note that in the C++ declaration, PackedTemplatePtr and PackedMaskPtr are pointers to pointers, not a variable. So, you'll have to make the call and dereference PackedTemplatePtr and PackedMaskPtr twice to get to the data area. Just copy memory a couple times and you should be fine.

[Merrion]

What is called Integer in .NET is a Long in VB6...

So your function should return a LONG in VB6
e.g.

Private Declare Function SomeFunction Lib "XYZ.dll" ( _
    ByVal FileName As String, _
    ByRef PackedTemplatePtr As Long, _
    ByRef PackedMaskPtr As Long, _
    ByRef Width As Long, ByRef Height As Long _
    ) As Long

[Comintern]

What is called Integer in .NET is a Long in VB6...

So your function should return a LONG in VB6

Forgot about that. Regardless, once the .dll call is made you need to manipulate the addresses passed manually into a variable. It works in .NET because you are getting a pointer variable. In VB6 you get the raw pointer in a long. Something along these lines should work:

Private Sub LoadFile(ByVal BmpFile As String)
    Dim TemplatePtr As Long
    Dim MaskPtr As Long
    Dim lTemp As Long

    Call SomeFunction(BmpFile, TemplatePtr, MaskPtr, Width, Height)

    Call CopyMemory(lTemp, TemplatePtr, LenB(lTemp))        'First dereference.
    Call CopyMemory(PackedTemplate, lTemp, TemplateSize)    'Second dereference.
    
    ...

End Sub

[jkeller]

Thanks for all the feedback. I had tried using Long types for the pointers (and the return value), since I agree, that's the version which should work according to the documentation. I tried again just in case I had typo'd something:

Private Declare Function SomeFunction Lib "XYZ.dll" ( _
    ByVal FileName As String, _
    ByRef PackedTemplatePtr As Long, _
    ByRef PackedMaskPtr As Long, _
    ByRef Width As Long, ByRef Height As Long _
    ) As Long

Public Sub LoadFromFile(ByVal BmpFile As String)
    Dim TemplatePtr As Long
    Dim MaskPtr As Long
    Dim WidthPtr As Long
    Dim HeightPtr As Long
    Call SomeFunction(BmpFile, TemplatePtr, MaskPtr, WidthPtr, HeightPtr)
End Sub

I also agree CopyMemory is what I want (similar to the Marshal calls in my working VB.NET version in the OP).

But I don't get that far; the program dies on the SomeFunction call with this error:

Run-time error '49': Bad DLL calling convention

So, I must have something wrong still.

[jkeller]

Could you throw a messagebox in the .NET code and see what values the two IntPtr hold?

Second, try:

Public Sub LoadFromFile(ByVal BmpFile As String)
    Dim Template As Integer
    Dim Mask As Integer
    Call SomeFunction (BmpFile, VarPtr(Template), VarPtr(Mask), Width, Height)
End Sub

Try that with both Template and Mask as Integers and as Longs.

I did display the IntPtr values in the .NET version when I was coding that; they were indeed big negative numbers (memory addresses) which eventually dereferenced fine using the Marhal calls.

The VarPtr looks interesting. I tried it using your syntax, and this slightly modified version:

Call SomeFunction (BmpFile, VarPtr(TemplatePtr), VarPtr(MaskPtr), VarPtr(WidthPtr), VarPtr(HeightPtr))

I still get the '49' error for both calls.

I'll research VarPtr and see if I can learn more about it though; it does sound promising, thanks.

[jkeller]

I added this declaration:

Declare Function VarPtrArray Lib "msvbvm60.dll" Alias "VarPtr" (Var() As Any) As Long

And tried various versions such as this:

    Dim TemplatePtr As Long
    Dim MaskPtr As Long
    Dim WidthPtr As Long
    Dim HeightPtr As Long
    
    TemplatePtr = VarPtrArray(PackedTemplate())
    MaskPtr = VarPtrArray(PackedMask())
    WidthPtr = VarPtr(Width)
    HeightPtr = VarPtr(Height)
    Call SomeFunction(BmpFile, TemplatePtr, MaskPtr, WidthPtr, HeightPtr)

Same error in all cases so far.

[Comintern]

Have you tried passing a string pointer instead of the actual string (probably a long shot)?

Private Declare Function SomeFunction Lib "XYZ.dll" ( _
    ByRef FileName As Long, _
    ByRef PackedTemplatePtr As Long, _
    ByRef PackedMaskPtr As Long, _
    ByRef Width As Long, ByRef Height As Long_
    ) As Long

Call SomeFunction(StrPtr(BmpFile), TemplatePtr, MaskPtr, Width, Height)

Also, try discarding the return value by declaring it as a Sub instead of a Function.

[EDIT] This link may shed some additional light...

[jkeller]

Very helpful, thanks!

The DLL developers confirmed they are using _cdecl, so they are going to have to change how they are doing it.

From your link:

It is not possible to directly call a C function in a DLL if that function uses the _cdecl calling convention. This is because Visual Basic uses the _stdcall calling convention for calling functions. This is a problem because if _cdecl is used, the calling function is responsible for cleaning up the stack. However, if _stdcall is used, the called function is responsible for cleaning up the stack.

NOTE: An .EXE file created in Visual Basic will allow you to call a DLL function that has been declared with the _cdecl calling convention without an error. It is only when you try to call such a function when running a program from the Visual Basic IDE, that Visual Basic generates the following error:

Run-time Error '49': Bad DLL Calling Convention

The fact that the EXE version allows you to call such functions has been confirmed to be a bug by Microsoft. You should not rely on this behavior as this might change in future versions of Visual Basic.