Subclassing App2

[jmsrickland]

I have two applications, App1 and App2

App1 can send a message to App2 and change App2's caption so I know I have the correct handle to App2 so why can I not subclass App2 from App1 using App2's handle

I have a working WndProc in App1 and it can subclass App1 using App1's handle but if I change the handle to App2's handle it wont catch messages from App2

Can it be done

[The trick]

Because the code of App2 works in other address space. If you want to subclass window in the other application you should do it in the context of this application. Imagine if you assign the address of procedure that is in the caller application to the window that is in the other application. You pass the address of function that is not valid in other application because each process have its own address space. I know the two approaches either dll injection or code injection.
When you inject a dll to the needed application you can subclass a window from the dll. You can pass each message to your application using different IPC methods. This method is described here.
Code injection is like to the previous method. You just inject a code to application and run it by any approach (for example CreateRemoteThread). This method is described here.
You can use hooks and filtering the inessential windows pass messages to your application.
Hijacking of window message in other process is more complex than in the process itself.

[dilettante]

It is usually far easier, far more reliable, and far more stable over time just to pay for a version of a product that exposes an API instead of relying on hijacking. This normally violates your license agreement anyway and is a form of piracy.

Even if the vendor doesn't care, if he doesn't offer a version exposing an API you're usually far better off just replicating the functionality you require yourself.

[jmsrickland]

Hijacking. Piracy. Anything else you see wrong?

For one thing both apps are mine. I wrote both applications so I don't need to pay for a version of a product that exposes an API instead of relying on hijacking, as you put it.

I have now found out that I can subclass App2 if I Shell it (which I really don't want to do this) from App1 instead of it running it as a standalone. In view of this then can it be said that App2 no longer runs in it's own address space but now it runs in the same address space as App1 and if not then how is it I can subclass App2.

App1 is the control panel application and App2 is one of several "robot" apps for a game I am developing. App1 is used by a person to manipulate the movements and other actions to be performed by the "robots"

If I cannot subclass App2 as a standalone then maybe I will have to change the way App1 controls these robot apps - probably going to need to use SendMessage instead.

[Magic Ink]

Now if App2 were an ActiveX.exe ...

[fafalone]

dilettante sees hackers and malware authors hiding around every corner :P

jms if you wrote both apps then you shouldn't even need to subclass one from the other, just implement a strong interprocess communications routine. Maybe look into WM_COPYDATA

[jmsrickland]

Yes, I am aware of other ways to cross communicate like CopyData as you suggest, winsock, DDE, etc. I just thought I would give subclassing a try to see if it would work out better and like I mentioned I can subclass as long as I Shell the other app. Also, as Magic suggested I might even try ActiveX

[dilettante]

Since you have control of both programs there is another option, perhaps the most obvious one.

Extract the code you want to share between these two programs as a DLL project. Then have both programs make use of that.

This is a far easier thing to do than adding an Automation API to an existing program. It is easier to test, easier to deploy, easier to avoid or cope with binary compatibility breaks, and more efficient than cross-process communication anyway.

[jmsrickland]

Well, that appears as an excellent option except I don't have the slightest idea how to go about making a DLL and then share code so both programs communicate with each other

[dilettante]

Well if you were looking for IPC then you should have said so. That's a horse of a different color.

[jmsrickland]

I wasn't looking for IPC. I asked about subclassing. Where did you get the idea I was looking for IPC. If I mentioned it, it was only in response about being aware of IPC

[The trick]

jms, have you read my posts? Do you see the videos? That is subclassing you can use it.

[jmsrickland]

Sorry, trick, but I can't use you project. I doesn't work and I can't figure it out because everything is in Russian and I can't read the message boxes so I don't know whats going on

[The trick]

Sorry, trick, but I can't use you project. I doesn't work and I can't figure it out because everything is in Russian and I can't read the message boxes so I don't know whats going on

Ok. I've made the other example. This example contain two projects: DLL and Exe.
DLL allows to subclass the different windows in other process as well. In order to start work you should call Initialize function. This function should return True if success. Further you can call SetSubclass for subclassing. This function takes a window handle at first parameter and a callbakc function at the second parameter. The callback function should have the following format:

Public Function UserWndProcProto( _
                ByVal hwnd As Long, _
                ByVal uMsg As Long, _
                ByRef wParam As Long, _
                ByRef lParam As Long, _
                ByRef defCall As Boolean) As Long

defCall determines whether the default procedure should be called or not. You can change wParam, lParam and defCall to change behavior of window.
In order to delete a subclassing you should call RemoveSubclass function. You should pass the same parameters you passed to SetSubclass.
I attach the sources both DLL and EXE. All the sources is commented in English (sorry for my English) i think you should understand the sense. This dll may contains some bugs because i don't test it enough.
Some explanation.
This dll uses a communication window for message exchanging. It registered special window class and create an instance of this window class. It uses file-mapping for data exchanging and a mutex object for synchronization. When you subclass a window in other process it installs a WH_CALLWNDPROC hook in order to load HookDLL to the needed process. Further it send the special global window message to the subclassed window that contains an information about sublcalssing. This message is intercepted by CallWndProc function in the needed process and it installs the subclassing in other process by SetWindowSubclass function. Every message passed to the subclassed window now goes thru WndProc procedure. Now it is necessary pass the messages to the our application. I decide to ensure the data exchange through a shared memory. In order to ensure atomic access to the shared memory i use the mutex object. When the mutex is capured i can access to shared memory exclusively. It is helpful if you want to do many subclassing or use this dll in the many projects. Each message will be process separately. After the mutex is captured we can pass the message data to the our communication window through shared memory. When the message has been processed we check a DefCall field if it equals to true it calls the default window proc. Eventually it is release mutex in order to other processes can use the shared memory.

' modFunctions.bas  - DLL for global subclassing
' © Anatoly Krivous (The trick), 2016

Option Explicit

Private Const TRICKWNDCLASS           As String = "TrickGlobalSubclass"         ' // Window class name
Private Const TRICKMUTEX              As String = "TrickGlobalSubclassMutex"    ' // Mutex name
Private Const TRICKMAP                As String = "TrickGlobalSubclassMap"      ' // Mapping name

' // This structure describes a message that is passed to subclasser application
Public Type MESSAGESTRUCT
    pWndProc        As Long     ' // Address of user window procedure
    callDef         As Boolean  ' // Determine whether call the default window proc or not
    lResult         As Long     ' // Returned value
    lParam          As Long     ' // Same as in WndProc
    wParam          As Long
    message         As Long
    hwnd            As Long
End Type

Private WM_SUBCLASS     As Long     ' // The message identifier for subclassing
                                    ' // Description:
                                    ' // wParam - a hWnd for subclassing
                                    ' // lParam - a hWnd that receives a WM_COPYDATA message
                                    ' // Return value - status
Private WM_UNSUBCLASS   As Long     ' // The message identifier for remove subclassing
Private WM_NEWMESSAGE   As Long     ' // New message available
Private hWndReceiver    As Long     ' // Handle of receiver window
Private isInitialized   As Boolean  ' // Determine whether dll is initialized or not
Private isClassReg      As Boolean  ' // Determine whether window class is registered or not
Private hMutex          As Long     ' // Handle of mutex object
Private pMapping        As Long     ' // Address of shared memory
Private hMap            As Long     ' // Handle of mapping object
Private pWndProcAddress As Long     ' // Address of WndProc procedure

' // This is EntryPoint procedure
Public Function DllEntry( _
                ByVal hInstDll As Long, _
                ByVal fdwReason As Long, _
                ByVal lpvReserved As Long) As Long
                
    If fdwReason = DLL_PROCESS_ATTACH Then
        
        ' // Register global communication message
        WM_SUBCLASS = RegisterWindowMessage("WM_SUBCLASS")
        WM_UNSUBCLASS = RegisterWindowMessage("WM_UNSUBCLASS")
        WM_NEWMESSAGE = RegisterWindowMessage("WM_NEWMESSAGE")
        
        ' // Create mutex
        hMutex = CreateMutex(ByVal 0&, 0, TRICKMUTEX)
        ' // Create mapping object
        hMap = CreateFileMapping(INVALID_HANDLE_VALUE, ByVal 0&, PAGE_READWRITE, 0, &H100, TRICKMAP)
        
        If hMap Then
            ' // Map shared memory
            pMapping = MapViewOfFile(hMap, FILE_MAP_WRITE, 0, 0, &H100)
            
        End If
        
        ' // Get address of WndProc
        pWndProcAddress = GetAddr(AddressOf WndProc)
        
        ' // Patch function
        PatchFunc AddressOf UserWndProcProto

    ElseIf fdwReason = DLL_PROCESS_DETACH Then
        
        ' // Release resources
        If hWndReceiver Then
        
            DestroyWindow hWndReceiver
            hWndReceiver = 0
            
        End If
        
        If isClassReg Then
        
            UnregisterClass TRICKWNDCLASS, App.hInstance
            isClassReg = False
            
        End If
        
        If hMutex Then
            
            CloseHandle hMutex
            hMutex = 0
            
        End If
        
        If pMapping Then
            
            UnmapViewOfFile ByVal pMapping
            CloseHandle hMap
            
            hMap = 0
            pMapping = 0
            
        End If
        
        isInitialized = False
        
    End If
    
    DllEntry = 1
    
End Function

' // You should call this procedure before any subclassing
Public Function Initialize() As Boolean
    Dim cls As WNDCLASSEX
    
    ' // Check DLL initialization
    If WM_SUBCLASS = 0 Or _
       WM_UNSUBCLASS = 0 Or _
       WM_NEWMESSAGE = 0 Or _
       hMutex = 0 Or _
       pMapping = 0 Then Exit Function
    
    cls.cbSize = Len(cls)
    cls.hInstance = App.hInstance
    cls.lpfnWndProc = GetAddr(AddressOf ReceiverProc)
    cls.lpszClassName = StrPtr(TRICKWNDCLASS)
    
    ' // Register class
    If RegisterClassEx(cls) = 0 Then Exit Function
    
    isClassReg = True
    
    ' // Create a receiver window
    hWndReceiver = CreateWindowEx(0, TRICKWNDCLASS, vbNullString, 0, 0, 0, 0, 0, HWND_MESSAGE, 0, App.hInstance, ByVal 0&)
    
    If hWndReceiver = 0 Then Exit Function

    ' // Set status
    isInitialized = True
    Initialize = True
    
End Function

' // This is main function to subclass window
Public Function SetSubclass( _
                ByVal hwnd As Long, _
                ByVal pWndProc As Long) As Boolean
    Dim pid     As Long
    Dim tid     As Long
    Dim hHook   As Long
    
    ' // Check initialization
    If Not isInitialized Then Exit Function
    
    ' // Get thread identifier
    tid = GetWindowThreadProcessId(hwnd, pid)
    
    If tid = App.ThreadID Then
        ' // Local
        If SetWindowSubclass(hwnd, pWndProcAddress, pWndProc, ByVal hWndReceiver) Then
            SetSubclass = True
        End If
    
    Else
    
        ' // Install hook
        hHook = SetWindowsHookEx(WH_CALLWNDPROC, AddressOf CallWndProc, App.hInstance, tid)
       
        If hHook Then
            ' // Send message for subclassing
            ' // This message will be received in the CallWndProc procedure in the needed process
            SendMessage hwnd, WM_SUBCLASS, hWndReceiver, ByVal pWndProc
            ' // Success
            SetSubclass = True
    
            UnhookWindowsHookEx hHook
            
        End If
        
    End If
    
End Function

' // This is main function to unsubclass window
Public Function RemoveSubclass( _
                ByVal hwnd As Long, _
                ByVal pWndProc As Long) As Boolean
    
    If Not isInitialized Then Exit Function
    
    If SendMessage(hwnd, WM_UNSUBCLASS, pWndProc, ByVal 0&) Then
        RemoveSubclass = True
    End If
    
End Function

' // This is the callback hook procedure
Private Function CallWndProc( _
                 ByVal nCode As Long, _
                 ByVal wParam As Long, _
                 ByRef lParam As CWPSTRUCT) As Long
                
    If nCode = HC_ACTION Then
        
        ' // We can process a message
        Select Case lParam.message
        Case WM_SUBCLASS
            ' // Query for subclassing
            If SetWindowSubclass(lParam.hwnd, pWndProcAddress, lParam.lParam, ByVal lParam.wParam) Then
                ' // Increment library counter
                LoadLibrary App.EXEName
            End If
        End Select
        
    End If
    
    CallWndProc = CallNextHookEx(0, nCode, wParam, lParam)
    
End Function

' // This is new window subclass procedure
Private Function WndProc( _
                 ByVal hwnd As Long, _
                 ByVal Msg As Long, _
                 ByVal wParam As Long, _
                 ByVal lParam As Long, _
                 ByVal uIdSubclass As Long, _
                 ByVal dwRefData As Long) As Long
    Dim ret     As Long
    Dim data    As MESSAGESTRUCT
    
    Select Case Msg
    Case WM_SUBCLASS
        ' // Success
        ret = True
        
    Case WM_UNSUBCLASS
        ' // Unsubclass window
        RemoveWindowSubclass hwnd, pWndProcAddress, wParam

    Case Else
        ' // Try to capture mutex for exclusive access
        If WaitForSingleObject(hMutex, INFINITE) = WAIT_OBJECT_0 Then
            
            ' // Fill message struct
            data.pWndProc = uIdSubclass
            data.callDef = True
            data.hwnd = hwnd
            data.message = Msg
            data.lParam = lParam
            data.wParam = wParam
            data.lResult = 0
            
            ' // Copy to shared memory
            memcpy ByVal pMapping, data, LenB(data)
            
            ' // Send message to the caller application window
            If SendMessage(dwRefData, WM_NEWMESSAGE, hwnd, ByVal App.ThreadID) Then
            
                ' // Check if need to call the default procedure
                memcpy data, ByVal pMapping, LenB(data)
                
                If data.callDef Then
                    ret = DefSubclassProc(data.hwnd, data.message, data.wParam, data.lParam)
                Else
                    ret = data.lResult
                End If
                
            Else
                ret = DefSubclassProc(hwnd, Msg, wParam, lParam)
            End If
            
            ' // Release mutex
            ReleaseMutex hMutex
            
        Else
            
            ret = DefSubclassProc(hwnd, Msg, wParam, lParam)
            
        End If
        
    End Select
    
    ' // Return value
    WndProc = ret
    
End Function

' // This is TrickGlobalSubclass window procedure
Private Function ReceiverProc( _
                 ByVal hwnd As Long, _
                 ByVal uMsg As Long, _
                 ByVal wParam As Long, _
                 ByVal lParam As Long) As Long
    Dim data    As MESSAGESTRUCT
    
    ' // Check window message
    If uMsg = WM_NEWMESSAGE Then
        ' // Mutex already is captured therefore an access to shared data is an atomic
        ' // Get message data form shared memory
        memcpy data, ByVal pMapping, LenB(data)
        ' // Call user window proc by pointer
        data.lResult = UserWndProcProto(data.pWndProc, data.hwnd, data.message, data.wParam, data.lParam, data.callDef)
        ' // Copy parameters
        memcpy ByVal pMapping, data, LenB(data)
        ' // Success
        ReceiverProc = True
    Else
        ReceiverProc = DefWindowProc(hwnd, uMsg, wParam, lParam)
    End If
    
End Function

Private Function UserWndProcProto( _
                 ByVal pAddress As Long, _
                 ByVal hwnd As Long, _
                 ByVal uMsg As Long, _
                 ByRef wParam As Long, _
                 ByRef lParam As Long, _
                 ByRef defCall As Boolean) As Long
End Function

Private Function GetAddr( _
                 ByVal Addr As Long) As Long
    GetAddr = Addr
End Function

[jmsrickland]

OK, your example (post 14) works very nice but I do have a concern. On the app I am subclassing I get an error message everytime I close that app even though i have already closed the subclassing app. The error message occurs no matter how or when I close the subclassed app.

Error message box =

Project1.exe - Application Error

The instruction at "0x734b37f4" referenced memory at "0x00000000". The memory could not be "read".

Click on OK to terminate the program.

[The trick]

OK, your example (post 14) works very nice but I do have a concern. On the app I am subclassing I get an error message everytime I close that app even though i have already closed the subclassing app. The error message occurs no matter how or when I close the subclassed app.

Error message box =

Project1.exe - Application Error

The instruction at "0x734b37f4" referenced memory at "0x00000000". The memory could not be "read".

Click on OK to terminate the program.

Can you send this project?