Subclassing an external window

[Megatron]

I find this question comes up quite a bit. I posted a solution in the API forum, but thought it would nice to have one here as well.

This example will subclass "Notepad" and catch the WM_RBUTTONDOWN event. We will display our own custom messagebox, instead of the default context menu that usually pops up.

So that said, open Notepad, and don't close it until I tell you to.

Now, let's start off with the gross part: C coding. Start a new Win32 Dynamic Link Library project in C++.

Insert the following code into it. I've tried to comment all the way through so you know what's going on.

#include <windows.h>
HHOOK hHook;

////////////////////////////////////////////////////////////////////////////
// GetMsgProc
// 
// This callback function will be imported from our VB app
////////////////////////////////////////////////////////////////////////////
extern "C" __declspec(dllexport) LRESULT CALLBACK GetMsgProc(INT nCode, 
															   WPARAM wParam, 
															   LPARAM lParam)
{
	// The lParam parameter contains a structure that provides information 
	// about the message being sent. 
	MSG* pCwp = (MSG*)lParam;

	// Here, we find the window we want to subclass. In this example, I'll use the
	// typing area of notepad
	HWND hEdit = FindWindowEx( FindWindow("Notepad", NULL), 0, "Edit", NULL );
	
	// We want to catch the WM_RBUTTONDOWN event, and we only want to trap it
	// if it was sent to the window we found earlier
	if(  (pCwp->message == WM_RBUTTONDOWN) && (pCwp->hwnd=hEdit) )
	{
		// The message has been caught -- do stuff here.

		// NOTE: If you do not need to replace/intercept the message, rather
		// just "know when it happens" you can send a message back to your main
		// window, and have any processing done there.
		MessageBox( hEdit, "You clicked it", "Bingo", MB_OK );
		return 0;
	}

	// Call the next hook.. Don't miss this step
	return CallNextHookEx( hHook, nCode, wParam, lParam );

}

All we're really doing is writing our GetMsgProc callback function, and specifying that the function is for export. This allows external applications to import the procedure.

So once that's created, add the following code to your VB Project.

Private Declare Function LoadLibrary Lib "kernel32" Alias "LoadLibraryA" (ByVal lpLibFileName As String) As Long
Private Declare Function GetProcAddress Lib "kernel32" (ByVal hModule As Long, ByVal lpProcName As String) As Long
Private Declare Function SetWindowsHookEx Lib "user32" Alias "SetWindowsHookExA" (ByVal idHook As Long, ByVal lpfn As Long, ByVal hmod As Long, ByVal dwThreadId As Long) As Long
Private Declare Function UnhookWindowsHookEx Lib "user32" (ByVal hHook As Long) As Long
Private Declare Function FreeLibrary Lib "kernel32" (ByVal hLibModule As Long) As Long
Private Const WH_GETMESSAGE = 3
Private hHook As Long
Private hModule As Long
 
Private Sub Form_Load()
        
    'Receives the address of the GetMsgProc function, which will be imported
    Dim lpfn As Long
    
    'Loads our DLL into memory. Change the path to the path of your DLL
    hModule = LoadLibrary("C:\MyPath\Subclassdll.dll")
    
    If hModule <> 0 Then
        'Retrieve the address of the GetMsgProc function in our DLL
        lpfn = GetProcAddress(hModule, "_GetMsgProc@12")
        
        'Install the WH_GETMESSAGE hook
        If lpfn <> 0 Then hHook = SetWindowsHookEx(WH_GETMESSAGE, lpfn, hModule, 0)
    Else
        MsgBox "Cannot load module"
    End If
    
End Sub
 
Private Sub Form_Unload(Cancel As Integer)
    'Uninstall the hook, and free the module in use
    If hHook <> 0 Then UnhookWindowsHookEx (hHook)
    FreeLibrary hModule
End Sub

The comments should explian what's going on in the code. I also want to draw your attention to the "_GetMsgProc@12" line. This is the function we're importing, but notice that's not what we originally called it? C modified the function name when you compiled your DLL. I found out the 'true name' via Dependancy walker. (Note: All hook functions will have the same prefix and suffix, since they contain the exact same parameters -- but that's going a little off topic).

Now, run your VB app, and try right-clicking in notepad. Close your VB application via the close-button (i.e. not "End" command, or "Stop" button on the toolbar). Also, close the VB app before you close notepad.

Save your work often.

If you break any of these rules, you'll discover why so many people recommand against subclassing external applications.

(You may close Notepad now)

[Megatron]

Just to ramble on some more.

I mentioned that you also have the option of sending a message back to your VB app, to notify it that an event has taken place in the external application.

Let's revisit the following code segment in our C procedure

if(  (pCwp->message == WM_RBUTTONDOWN) && (pCwp->hwnd=hEdit) )
{
	// The message has been caught

	// Send Message back to VB. Change the window title to the title of your VB application

	HWND hForm = FindWindowEx(0,0,NULL, "SubclassTest");
	if( hForm )
	{
		SendMessage( hForm, (WM_USER+800), (WPARAM)hEdit, NULL );
	}
	//return 0;
}

It's a neat little trick. We make a custom message (WM_USER+ 800)
that represents when an event has taken place in Notepad. I passed the handle of the edit control for reference, but you can be even more generic by passing the entire MSG structure. This will notify us of all messages sent to the window, as well as thier wParam and lParam values.

Now, to process this message in VB, simply subclass it like you normally would, and catch this customized WM_USER+800 message.

So add the following code to a module in your VB app

Private Declare Function SetWindowLong Lib "user32.dll" Alias "SetWindowLongA" (ByVal hwnd As Long, ByVal nIndex As Long, ByVal dwNewLong As Long) As Long
Private Declare Function GetWindowLong Lib "user32.dll" Alias "GetWindowLongA" (ByVal hwnd As Long, ByVal nIndex As Long) As Long
Private Declare Function CallWindowProc Lib "user32.dll" Alias "CallWindowProcA" (ByVal lpPrevWndFunc As Long, ByVal hwnd As Long, ByVal msg As Long, ByVal wParam As Long, ByVal lparam As Long) As Long
Private Const GWL_WNDPROC As Long = -4
Private lPrevProc As Long
Private Const WM_USER As Long = &H400
 
Public Sub SubClassWnd(ByVal hwnd As Long)
    lPrevProc = GetWindowLong(hwnd, GWL_WNDPROC)
    Call SetWindowLong(hwnd, GWL_WNDPROC, AddressOf WinProc)
End Sub
 
Public Sub UnSubclassWnd(ByVal hwnd As Long)
    SetWindowLong hwnd, GWL_WNDPROC, lPrevProc
End Sub
 
Public Function WinProc(ByVal hwnd As Long, ByVal uMsg As Long, ByVal wParam As Long, ByVal lparam As Long) As Long
    If uMsg = (WM_USER + 800) Then
        Debug.Print "NOTEPAD WAS RIGHT-CLICKED"
        Exit Function
    End If
    
    WinProc = CallWindowProc(lPrevProc, hwnd, uMsg, wParam, lparam)
End Function

Pay attention to the WinProc procedure, and notice how we scan for our customized message.

(Since this is already a somewhat advanced topic, I'll assume you already know how to set up the rest of the internal subclassing, so I won't go into detail there)

[RobDog888]

Very interesting, hmm. Nice work Megatron!

So if we wanted to add a new menu item to the standard right
click menu in Notepad, we could just replace the msgbox and add
GetMenu/GetSubMenu/GetMenuItemCount/AppendMenu API calls
to add a custom menu item?

Subclassing in vb would need to catch the WM_MENUCOMMAND
message and parse the parameter info to determine if our menu
item was clicked. Is this correct thinking?

[Megatron]

Precisely. Once you have it subclassed, you can create new menu items just like you would in VB (VB-API)

This question also brought up something I want to draw your attention to: The first example above actually isn't "subclassing," rather we're using hooks.

This next example will demonstrate the use of subclassing and hooks. We first use a hook to catch the message sent to our window, as per the previous example. But now we'll take it a small step further and actually subclass this window so that we have an actual window procedure to work with.

So this will produce a new (and useless) context menu for notepad.

#include <windows.h>

HHOOK hHook;
HMENU hMenu;
WNDPROC lPrevProc;

// We will use these identifiers 
const ID_FOO = 101;
const ID_BAR = 201;
const ID_CONTEXT = 301;

// Create a new popup menu for notepad
void initMenus()
{
	hMenu = CreatePopupMenu();
	AppendMenu( hMenu, MF_STRING, ID_CONTEXT, "Context");
	AppendMenu( hMenu, MF_STRING, ID_FOO, "Foo");
	AppendMenu( hMenu, MF_STRING, ID_BAR, "Bar");
}

// Subclassed callback procedure
LRESULT CALLBACK WinProc(HWND hWnd, UINT uMsg, WPARAM wParam, LPARAM lParam)
{

	// When the right-mouse is clicked, display our own pop-up menu
	if( uMsg == WM_RBUTTONDOWN )
	{
		POINT pt;
		pt.x = LOWORD(lParam);
		pt.y = HIWORD(lParam);
		ClientToScreen( hWnd, &pt );
		TrackPopupMenu( hMenu, TPM_LEFTALIGN, pt.x, pt.y, 0, hWnd, NULL );
		return 0;
	}

	// Handle each menu-item click
	else if( uMsg == WM_COMMAND )
	{
		switch( LOWORD(wParam) )
		{
			case ID_CONTEXT:
				MessageBox(hWnd, "Context", "Menu", MB_OK);
				return 0;
			case ID_FOO:
				MessageBox(hWnd, "Foo", "Menu", MB_OK);
				return 0;
			case ID_BAR:
				MessageBox(hWnd, "Bar", "Menu", MB_OK);
				return 0;
		}
	}

	// Unsubclass the window when it closes
	else if( uMsg == WM_CLOSE )
	{
		SetWindowLong( hWnd, GWL_WNDPROC, (LONG)lPrevProc );
		return 0;
	}

	return CallWindowProcA( lPrevProc, hWnd, uMsg, wParam, lParam);
}


extern "C" __declspec(dllexport) LRESULT CALLBACK GetMsgProc(INT nCode, WPARAM wParam, LPARAM lParam)
{

	// bSubclass was added to this procedure to ensure that we only subclass the window
	// once. Otherwise bad things can happen

	MSG* pCwp = (MSG*)lParam;
	static bool bSubclass = false;

	HWND hEdit = FindWindowEx( FindWindow("Notepad", NULL), 0, "Edit", NULL );

	if(  (pCwp->message == WM_RBUTTONDOWN) && (pCwp->hwnd=hEdit) && (bSubclass == false) )
	{
		initMenus();
		lPrevProc = (WNDPROC)GetWindowLong(hEdit, GWL_WNDPROC );
		SetWindowLong( hEdit, GWL_WNDPROC, (LONG)WinProc);
		bSubclass = true;
	}

	return CallNextHookEx( hHook, nCode, wParam, lParam );

}

Note, that instead of displaying messageboxes, you can actually re-route the event back to VB via a customized window message (i.e. WM_USER+500), for greater flexibility. My second example, above, shows you the basics of this.

[moeur]

One problem you'll run into with this nice technique is when you try to pass data back to your VB program. Pointers have to be handled in a special way.

One way I've used (and would suggest here) is instead of sending your subclassed VB app a WM_USER message, send it an exact copy of the message you recieved in the DLL. This way, the OS handles the marshalling of pointers for you.

So that you don't confuse your VB app with these messages, you could send them to an invisible control that isn't expecting any messages anyway.

[EJ12N]

wow awesome job megatron just what i wanted to learn you are god! lol
very nice indeed Thanks alot!

One problem you'll run into with this nice technique is when you try to pass data back to your VB program. Pointers have to be handled in a special way.

One way I've used (and would suggest here) is instead of sending your subclassed VB app a WM_USER message, send it an exact copy of the message you recieved in the DLL. This way, the OS handles the marshalling of pointers for you.

So that you don't confuse your VB app with these messages, you could send them to an invisible control that isn't expecting any messages anyway.

so you mean instead of this...

SendMessage( hForm, (WM_USER+800), (WPARAM)hEdit, NULL );

do this right...

SendMessage( hForm, (WM_RBUTTONDOWN), (WPARAM)hEdit, NULL );

[moeur]

No, more like this

 SendMessage(hForm, WM_RBUTTONDOWN, pCwp->wParam, pCwp->lParam);

This way your VB APP gets an exact copy of the message.

...

[moeur]

I've posted some code in this thread that shows you how to do this. As long as I was at it I also added code to show you how you can change the data received in a posted message.

-Smokin' H.Upmann

...

[xera]

ok, I know this thread is old, but there is a tiny typo, that might get some people confused

in this line:

if(  (pCwp->message == WM_RBUTTONDOWN) && (pCwp->hwnd=hEdit) )

I think it should be == and not =
Like so:

if(  (pCwp->message == WM_RBUTTONDOWN) && (pCwp->hwnd == hEdit) )

Lol, in it's current state, it crashes any app you right click on.

[xera]

ok, I know this thread is old, but I was going through this post again, and noticed a tiny typo, that might get some people confused...

in this line:

if(  (pCwp->message == WM_RBUTTONDOWN) && (pCwp->hwnd=hEdit) )

I think it should be == and not =
Like so:

if(  (pCwp->message == WM_RBUTTONDOWN) && (pCwp->hwnd == hEdit) )

Lol, in it's current state, it crashes almost every app you right click on.