calling __fastcall and __thiscall with ASM, from VB6

**BBN1** · Dec 12th, 2008, 06:51 AM

Hi there!
uh, this is kind of al earning cerv for me, so i'm not sure where best to start.
I have executable code in memory and an address to a __Fastcall function.
What I have done, is compiled a C dll to take an address and param, and call the address passing the param.
The param is a table of vb6 __Stdcall function address's, that the function im calling, will calls back on.

This is the function in my C dll:

Code:

__int32 __stdcall Test(__int32 dwAddr, __int32 dwTable)
{
    __asm {
        mov        ecx, [esp+8]   //dwTable
        call       [esp+4]        //dwAddr
    }
}

This is that function in IDA:

Code:

.text:10001040 Test    proc near
.text:10001040
.text:10001040 arg_0           = dword ptr  4
.text:10001040 arg_4           = dword ptr  8
.text:10001040
.text:10001040                 mov     ecx, [esp+arg_4] 
.text:10001044                 call    [esp+arg_0]
.text:10001048                 retn    8
.text:10001048 Test    endp

This is how I declare it in VB6:

Code:

Public Declare Function Test Lib "Test.dll" (ByVal Address As Long, ByVal dwParam As Long) As Long

This works fine. When I call the Test, with the address of the __fastcall function I want to call, and a pointer to the callback list, the function returns the expected value, plus calls my VB6 functions (in the callback table)

How ever, I don't want to do this from a DLL, I'm trying to do it from a VB6 class.
So far, I have been able to call the function address and get the call backs (so I know the address and param is on the stack in the right palaces)
But, I don't get the return value

plus some nast crash's next time I call it.

Here is my VB6 code:

Code:

Private m_ptrMe         As Long 'pointer of this class
Private m_ptrFunc       As Long 'function pointer of MyFunc
Private m_Mem           As Long 'pointer to the code
Private m_Code          As String

Private Sub Class_Initialize()
    Dim fctAddress      As Long
    Call CopyMemory(m_ptrMe, ByVal ObjPtr(Me), 4)
    Call CopyMemory(m_ptrFunc, ByVal m_ptrMe + 28, 4)
    Call MyFunc(0, 0) 'inits the function
End Sub
Private Sub Compile(ByVal S As String)
    m_Code = m_Code & HexToStr(S)
End Sub
Public Function MyFunc(ByVal lngAddr As Long, ByVal lngParam As Long) As Long
    Call Compile("8B 4C 24 0C") 'mov     ecx, [esp+lngParam]
    Call Compile("FF 54 24 08") 'call    [esp+lngAddr]
    Call Compile("C2 0C 00")    'retn    8
    
    '//Copy code to a block of memory
    m_Mem = malloc(Len(m_Code))
    Call CopyMemory(ByVal m_Mem, ByVal m_Code, Len(m_Code))
    '//Copy code address over MyFunc
    Call CopyMemory(ByVal m_ptrMe + 28, m_Mem, 4)
End Function

Public Function CallFunc(ByVal lngAddr As Long, ByVal Param As Long) As Long
    CallFunc = MyFunc(lngAddr, Param)
End Function

As you can see, the ASM in the DLL was like this:
8B 4C 24 08 FF 54 24 04 C2 08 00
But the only way I could get it even close to working in VB6, is:
8B 4C 24 0C FF 54 24 08 C2 0C 00

Does anyone know why the stack is out of line by 4 bytes?
The above code is an __stdcall that call's a __fastcall, so intheory it should be like trying to call a normal API with 2 params.
This is all kind of new-ground to me, so I maybe missing somthing -- but could anyone please explain to me whats up, what im missing, and why I don't get a return?

big thanks in advance!

**minor28** · Dec 13th, 2008, 08:07 AM

As far as I can follow you are not copying hex data as you think

Compile("8B 4C 24 0C") will give the string m_Code = "8B 4C 24 0C" and
Compile("FF 54 24 08") => m_Code = "8B 4C 24 0CFF 54 24 08"

i.e. the three compile will result in a string m_Code = "8B 4C 24 0CFF 54 24 08C2 0C 00"
Len(m_Code) = 30 characters and not 11 hexvalues

What HexToStr(S) does I don't know but it should compute a string.

The string in hex is actually = 3842203443203234203043464620353420323420384332203043203030 and this is what you copy to memory.

I don't quite understand what you want to achieve. Do you want to write a function in memory and then execute it or are you trying to make MyFunc to a fastcall convention?

**BBN1** · Dec 13th, 2008, 01:39 PM

hi there!
Well, the HexToStr() function shouldnt be a problem -- for every 3 byte's passed to it, it converts to 1 character:

Code:

Private Function HexToStr(ByVal strData As String) As String
    HexToStr = String(Len(strData) / 3, 0)
    Dim iPos As Long
    For i = 1 To Len(strData) Step 3
        iPos = iPos + 1
        Mid$(HexToStr, iPos, 1) = Chr("&H" & Mid$(strData, i, 2))
    Next i
End Function

so HexToStr("30 31 32 33") will = "0123"

uh, I'm finding it hard to explain what i'm trying to do

__fastcall function aside, what i'm trying to do, is make an __stdcall function that calls a __fastcall function with it.
Kind of like any old api/__stdcall function, to be writen in the place of a class function.
So when ever I do somthing like A = Class.MyFunc(B, C), it will execute my ASM code, and not the VB6 code with in MyFunc()
So far I have got it working well, but my main number one problem, is the return value.
This is best I can show what I mean:

Code:

Public Function MyFunc(ByVal lngAddr As Long, ByVal Param As Long) As Long
    Call Compile("8B 4C 24 0C") 'mov     ecx, [esp+12] //param
    Call Compile("8B 44 24 08") 'mov     eax, [esp+8]  //lngAddr
    Call Compile("03 C1")       'add     eax, ecx
    Call Compile("40")          'inc     eax
    Call Compile("C2 0C 00")    'retn    12

If I do somthing like:
Dim A As New Class1
A = Class.MyFunc(1, 2)
It should return (1+2)+1 into the variable A -- since that is what eax holds.
How ever, it always returns zero, no matter what is in eax.
Ive also noticed, even tho VB6 thinks it's calling MyFunc like it would any other class function, it pushs 3 things onto the stack, where as for an API, it would be 2 things:
a pointer to the class [esp+4]
the value of address [esp+8]
the value of param [esp+12]
the registers starting in the function will be somthing like:
eax = pointer to the code pointer (*class+28)
ecx = pointer to a pointer to the class address (ecx->ptr->class)
edx = 0x00000000

This is a stack dump I did, with the ASM (I just made it write each register/stack variable, to some public variables)

Code:

VAL ADDRESS = 0x18CB904  value of address to be called
VAL PARAM   = 0x1D87CC   value of param to be passed
PTR RETURN  = 0x13F868   pointer to A, where A = Class.func(Addr, Param)
PTR ME      = 0x1F3B54   address of the class
PTR CODE    = 0x1D9244   address of the code (writen to *Class + 28)
PTR FUNC    = 0x1D815F   variable address of MyFunc (in Class)


eax       = 0x1F3B70  pointer to this function (*class + 28) eax->ptr->code
ecx       = 0x21F1F8  pointer to class@0x1F3B54
edx       = 0x0
STACK(-8) = 0xFC018C4 [esp-8]
STACK(-4) = 0x2117F4  [esp-4]
STACK(+0) = 0xFC02030 [esp+0]
STACK(+4) = 0x21F1F8  [esp+4]  pointer to class@0x1F3B54 (ecx)
STACK(+8) = 0x18CB904 [esp+8]  value of address being passed
STACK(+C) = 0x1D87B0  [esp+C]  value of param being passed

Here is some test code ive knocked up, to try find out why eax isnt being returned like it would normaly be:

Code:

Private Declare Sub CopyMemory Lib "kernel32" Alias "RtlMoveMemory" (ByRef destination As Any, ByRef Source As Any, ByVal numbytes As Long)
Private Declare Function GlobalAlloc Lib "kernel32" (ByVal wFlags As Long, ByVal dwBytes As Long) As Long
Private Declare Function GlobalFree Lib "kernel32" (ByVal hMem As Long) As Long
Private Declare Function GlobalLock Lib "kernel32" (ByVal hMem As Long) As Long
Private Declare Function GlobalUnlock Lib "kernel32" (ByVal hMem As Long) As Long

Private m_ptrMe         As Long 'pointer of this class
Private m_ptrFunc       As Long 'function pointer of MyFunc
Private m_Mem           As Long 'pointer to the code
Private m_Code          As String

Private Sub Class_Initialize()
    Call CopyMemory(m_ptrMe, ByVal ObjPtr(Me), 4)
    Call CopyMemory(m_ptrFunc, ByVal m_ptrMe + 28, 4)
    Call MyFunc(0, 0)
End Sub
Private Sub Class_Terminate()
    Call CopyMemory(m_ptrMe, ByVal ObjPtr(Me), 4)
    Call CopyMemory(ByVal m_ptrMe + 28, m_ptrFunc, 4)
    Call free(m_Mem)
End Sub

Private Function HexToStr(ByVal strData As String) As String
    HexToStr = String(Len(strData) / 3, 0)
    Dim iPos As Long
    For i = 1 To Len(strData) Step 3
        iPos = iPos + 1
        Mid$(HexToStr, iPos, 1) = Chr("&H" & Mid$(strData, i, 2))
    Next i
End Function
Private Function malloc(ByVal dwSize As Long) As Long
    Dim lngHandle   As Long
    lngHandle = GlobalAlloc(0, dwSize + 4)
    malloc = GlobalLock(lngHandle) + 4
    Call CopyMemory(ByVal malloc - 4, lngHandle, 4)
End Function
Private Sub free(ByVal dwPtr As Long)
    Dim lngHandle   As Long
    Call CopyMemory(lngHandle, ByVal dwPtr - 4, 4)
    Call GlobalUnlock(lngHandle)
    Call GlobalFree(lngHandle)
End Sub

Private Sub Compile(ByVal S As String)
    m_Code = m_Code & HexToStr(S)
End Sub

Public Function MyFunc(ByVal lngAddr As Long, ByVal Param As Long) As Long
    Call Compile("8B 4C 24 0C") 'mov     ecx, [esp+12] //param
    Call Compile("8B 44 24 08") 'mov     eax, [esp+8]  //lngAddr
    Call Compile("03 C1")       'add     eax, ecx
    Call Compile("40")          'inc     eax
    Call Compile("C2 0C 00")    'retn    12
    
    m_Mem = malloc(Len(m_Code))
    Call CopyMemory(ByVal m_Mem, ByVal m_Code, Len(m_Code))
    Call CopyMemory(ByVal m_ptrMe + 28, m_Mem, 4)
End Function

You use it like
Dim C as new class1
Dim A as long
A = C.MyFunc(20, 100)
It should* in theory, return 121, since it moves 20, into ecx, then moves 100 into eax, then adds ecx to eax, then increments eax by 1 and pops the class pointer, address and param off the stack.

I'm really baffled why eax isnt returned.
Do you have any ideas?
thank you for reply, and any help you can give me!

**minor28** · Dec 13th, 2008, 03:17 PM

Ok, with HexToStr convert string "30" to string "0". It is a bstring with hex = 4 byt field with length and 30,00. Each caracter is followed by 00 and string is ended with 00,00. The bstring "0" should be 00,00,00,04,30,00,00,00

You must not use string. Convert each hex to a byte value and use undocumented function PutMem1 to copy to memory.

Declare Sub PutMem1 Lib "msvbvm60" (Ptr As Any, ByVal NewVal As Byte)

I don´t think this will solve your problem, if it is possible at all.

**BBN1** · Dec 13th, 2008, 03:33 PM

Originally Posted by minor28

Ok, with HexToStr convert string "30" to string "0". It is a bstring with hex = 4 byt field with length and 30,00. Each caracter is followed by 00 and string is ended with 00,00. The bstring "0" should be 00,00,00,04,30,00,00,00

You must not use string. Convert each hex to a byte value and use undocumented function PutMem1 to copy to memory.

Declare Sub PutMem1 Lib "msvbvm60" (Ptr As Any, ByVal NewVal As Byte)

I don´t think this will solve your problem, if it is possible at all.

When you pass a string, byval, as any, VB converts it, iirc.
dim B(9) as byte
dim S as string
S = space(10)
Copymemory Byval S, B(0), 10
Would do the same as
Dim B() as Byte
dim S as string
S = space(10)
B() = StrConv(S, vbFromUnicode)

But thats not the problem i'm having anyway, since the ASM executes just fine, but I never get a return value.

**minor28** · Dec 13th, 2008, 04:47 PM

I did this a couple of years ago. It masm code. Perhaps it can give you an idea.

How to run a function from allocated memory. This is the function

Code:

calculate proc x:dword, y:dword
	push ebp		55
	mov ebp,esp		8B EC
	mov eax,dword ptr [x]	8B 45 08
	mov edx,dword ptr [y]	8B 55 0C
	mul eax,edx		F7 E2
	leave			C9
	ret 8			C2 08 00
calculate endp

This is code for an application

Code:

.data
szCaption	db "My message",0
szFormat	db "5x7=%d",0
calculate	db 55h,8Bh,0ECh,8Bh,45h,08h,8Bh,55h,0Ch,0F7h,0E2h,0C9h,0C2h,08h,00h

.data?
pMem 		dd ?
buffer		dd 12 dup (?)

.code
start:

	invoke GlobalAlloc,GMEM_FIXED,32
	mov pMem, eax
	mov ecx, LENGTHOF calculate
	mov esi, OFFSET calculate
	mov edi, pMem
	rep movsb
	
	;calling the function in memory
	push 5
	push 7
	call pMem
	
	;Result in messagebox
	invoke wsprintf,offset buffer,offset szFormat,eax
	invoke MessageBox,0,offset buffer,offset szCaption,MB_OK
	
	invoke GlobalFree,pMem
	invoke ExitProcess,0

end start

**BBN1** · Dec 14th, 2008, 01:36 AM

Tahnks for the help minor28, but ive given up trying to get the VB6 class to return the value.
Now I just compile ASM on the fly, and pass everything by pointer -- it works very well:

Code:

Private Function CallInit(ByVal Address As Long, ByVal Param As Long) As Long
    Dim ASM     As New clsASM
    With ASM                             '//Compile ASM and execute
        .mov__ecx_ptr (Param)            '//Param contains pointer to the true param
        .call_ptr (Address)              '//Address contains pointer to true address
        .mov__ptr_eax (VarPtr(CallInit)) '//pointer to return value
        .retn                            '//exit the function
        .Execute                         '//Execute the code
    End With
End Function

This seems to be alot more flexible as well

Thread: calling fastcall and thiscall with ASM, from VB6

Thread Tools

Display