Tutorial 20: Window Subclassing

In this tutorial, we will learn about window subclassing, what it is and how to use it to your advantage.
Download the example here.

Theory:

If you program in Windows for some time, you will find some cases where a window has nearly the attributes you need in your program but not quite. Have you encountered a situation where you want some special kind of edit control that can filter out some unwanted text? The straightforward thing to do is to code your own window. But it's really hard work and time-consuming. Window subclassing to the rescue.
In a nutshell, window subclassing allows you to "take over" the subclassed window. You will have absolute control over it. Let's take an example to make this clearer. Suppose you need a text box that accepts only hex numbers. If you use a simple edit control, you have no say whatsoever when your user types something other than hex numbers into your text box, ie. if the user types "zb+q*" into your text box, you can't do anything with it except rejecting the whole text string. This is unprofessional at least. In essence, you need the ability to examine each character the user typed into the text box right at the moment he typed it.
We will examine how to do that now. When the user types something into a text box, Windows sends WM_CHAR message to the edit control's window procedure. This window procedure resides inside Windows itself so we can't modify it. But we can redirect the message flow to our own window procedure. So that our window procedure will get first shot at any message Windows sends to the edit control. If our window procedure chooses to act on the message, it can do so. But if it doesn't want to handle the message, it can pass it to the original window procedure. This way, our window procedure inserts itself between Windows and the edit control. Look at the flow below: Now we put our attention on how to subclass a window. Note that subclassing is not limited to controls, it can be used with any window.
Let's think about how Windows knows where the edit control's window procedure resides. A guess?......lpfnWndProc member of WNDCLASSEX structure. If we can replace this member with the address of our own window procedure, Windows will send messages to our window proc instead.
We can do that by calling SetWindowLong. hWnd = handle of the window to change the value in the WNDCLASSEX structure
nIndex == value to change. dwNewLong = the replacement value.
So our job is easy: We code a window proc that will handle the messages for the edit control and then call SetWindowLong with GWL_WNDPROC flag, passing along the address of our window proc as the third parameter. If the function succeeds, the return value is the previous value of the specified 32-bit integer, in our case, the address of the original window procedure. We need to store this value for use within our window procedure.
Remember that there will be some messages we don't want to handle, we will pass them to the original window procedure. We can do that by calling CallWindowProc function. lpPrevWndFunc = the address of the original window procedure.
The remaining four parameters are the ones passed to our window procedure. We just pass them along to CallWindowProc.

Code Sample:

.386
.model flat,stdcall
option casemap:none
include \masm32\include\windows.inc
include \masm32\include\user32.inc
include \masm32\include\kernel32.inc
include \masm32\include\comctl32.inc
includelib \masm32\lib\comctl32.lib
includelib \masm32\lib\user32.lib
includelib \masm32\lib\kernel32.lib

WinMain PROTO :DWORD,:DWORD,:DWORD,:DWORD
EditWndProc PROTO :DWORD,:DWORD,:DWORD,:DWORD

.data
ClassName  db "SubclassWinClass",0
AppName    db "Subclassing Demo",0
EditClass  db "EDIT",0
Message  db "You pressed Enter in the text box!",0

.data?
hInstance  HINSTANCE ?
hwndEdit dd ?
OldWndProc dd ?

.code
start:
    invoke GetModuleHandle, NULL
    mov    hInstance,eax
    invoke WinMain, hInstance,NULL,NULL, SW_SHOWDEFAULT
    invoke ExitProcess,eax

WinMain proc hInst:HINSTANCE,hPrevInst:HINSTANCE,CmdLine:LPSTR,CmdShow:DWORD
    LOCAL wc:WNDCLASSEX
    LOCAL msg:MSG
    LOCAL hwnd:HWND
    mov   wc.cbSize,SIZEOF WNDCLASSEX
    mov   wc.style, CS_HREDRAW or CS_VREDRAW
    mov   wc.lpfnWndProc, OFFSET WndProc
    mov   wc.cbClsExtra,NULL
    mov   wc.cbWndExtra,NULL
    push  hInst
    pop   wc.hInstance
    mov   wc.hbrBackground,COLOR_APPWORKSPACE
    mov   wc.lpszMenuName,NULL
    mov   wc.lpszClassName,OFFSET ClassName
    invoke LoadIcon,NULL,IDI_APPLICATION
    mov   wc.hIcon,eax
    mov   wc.hIconSm,eax
    invoke LoadCursor,NULL,IDC_ARROW
    mov   wc.hCursor,eax
    invoke RegisterClassEx, addr wc
    invoke CreateWindowEx,WS_EX_CLIENTEDGE,ADDR ClassName,ADDR AppName,\
 WS_OVERLAPPED+WS_CAPTION+WS_SYSMENU+WS_MINIMIZEBOX+WS_MAXIMIZEBOX+WS_VISIBLE,CW_USEDEFAULT,\
           CW_USEDEFAULT,350,200,NULL,NULL,\
           hInst,NULL
    mov   hwnd,eax
    .while TRUE
        invoke GetMessage, ADDR msg,NULL,0,0
        .BREAK .IF (!eax)
        invoke TranslateMessage, ADDR msg
        invoke DispatchMessage, ADDR msg
    .endw
    mov eax,msg.wParam
    ret
WinMain endp

WndProc proc hWnd:HWND, uMsg:UINT, wParam:WPARAM, lParam:LPARAM
    .if uMsg==WM_CREATE
        invoke CreateWindowEx,WS_EX_CLIENTEDGE,ADDR EditClass,NULL,\
            WS_CHILD+WS_VISIBLE+WS_BORDER,20,\
            20,300,25,hWnd,NULL,\
            hInstance,NULL
        mov hwndEdit,eax
        invoke SetFocus,eax
        ;-----------------------------------------
        ; Subclass it!
        ;-----------------------------------------
        invoke SetWindowLong,hwndEdit,GWL_WNDPROC,addr EditWndProc
        mov OldWndProc,eax
    .elseif uMsg==WM_DESTROY
        invoke PostQuitMessage,NULL
    .else
        invoke DefWindowProc,hWnd,uMsg,wParam,lParam
        ret
    .endif
    xor eax,eax
    ret
WndProc endp

EditWndProc PROC hEdit:DWORD,uMsg:DWORD,wParam:DWORD,lParam:DWORD
    .if uMsg==WM_CHAR
        mov eax,wParam
        .if (al>="0" && al<="9") || (al>="A" && al<="F") || (al>="a" && al<="f") || al==VK_BACK
            .if al>="a" && al<="f"
                sub al,20h
            .endif
            invoke CallWindowProc,OldWndProc,hEdit,uMsg,eax,lParam
            ret
        .endif
    .elseif uMsg==WM_KEYDOWN
        mov eax,wParam
        .if al==VK_RETURN
            invoke MessageBox,hEdit,addr Message,addr AppName,MB_OK+MB_ICONINFORMATION
            invoke SetFocus,hEdit
        .else
            invoke CallWindowProc,OldWndProc,hEdit,uMsg,wParam,lParam
            ret
        .endif
    .else
        invoke CallWindowProc,OldWndProc,hEdit,uMsg,wParam,lParam
        ret
    .endif
    xor eax,eax
    ret
EditWndProc endp
end start

Analysis:

After the edit control is created, we subclass it by calling SetWindowLong, replacing the address of the original window procedure with our own window procedure. Note that we store the address of the original window procedure for use with CallWindowProc. Note the EditWndProc is an ordinary window procedure. Within EditWndProc, we filter WM_CHAR messages. If the character is between 0-9 or a-f, we accept it by passing along the message to the original window procedure. If it is a lower case character, we convert it to upper case by adding it with 20h. Note that, if the character is not the one we expect, we discard it. We don't pass it to the original window proc. So when the user types something other than 0-9 or a-f, the character just doesn't appear in the edit control. I want to demonstrate the power of subclassing further by trapping Enter key. EditWndProc checks WM_KEYDOWN message if it's VK_RETURN (the Enter key). If it is, it displays a message box saying "You pressed the Enter key in the text box!". If it's not an Enter key, it passes the message to the original window procedure.
You can use window subclassing to take control over other windows. It's one of the powerful techniques you should have in your arsenal.
[Iczelion's Win32 Assembly Homepage]