Feedback Search Top Backward Forward

Hooking: TSRs Invade PM

Written by Michael Shillingford



Back in the Jurassic days of PC computing, a few nifty programs such as Sidekick, screen savers and keyboard extenders rose up as handy addons to a basic DOS system, giving new life to an aging OS. These little wonders did their magic by a method called Terminate and Stay Resident (TSR) - once loaded, they lurked behind the scenes, hibernating until an event occurred that brought them to life, and occasionally, to a quick system death. Yes, these innocent lurkers had too much power, allowing whizbang action, but also spectacular lockups when tripping over other TSRs or the rather delicate OS under their feet.

The message-based design of OS/2 allows for a much more sophisticated interface into system activity. Indeed, a single PM function call allows an application to do all sorts of mischief to the operation of the rest of the system. This method, called a system hook, is the 90's version of the TSR.

But a single interface and a protected memory system does not mix into an "anyone-can-do-this" program. Like it's predecessor, writing solid system hooks is not an easy task, and frequent lockups of the crash-proof system will lie in the path of those who dare.

This article will introduce PM system hooks, giving tips along the way to allow you to get your hands dirty without having to stare at the boot logo all day long. Even if you are not interested in the guts, you might want to add the included sample program Xray ( to your arsenal of tools - it provides a simple window peeker that ferrets out the behind the scenes window info, not unlike PMTREE.

The Hook Function

PM activity is a beehive of message activity. As events occur, such as a mouse move or a timer elapsing, messages are generated that represent that activity, and the message is routed along a queue to the appropriate application. Imagine adding a T-junction to the middle of that queue, where you could observe those messages scurrying by. This is what a PM system hook does. It attaches itself to the message queue, allowing one to view the messages, alter them, or remove them before they reach the destination.

The function to enable a hook is:

    WinSetHook(HAB hab, HMQ hmqType, LONG hookType, PFN hookProc, HMODULE hmDll);

And to remove the hook, use the same parameters:

    WinReleaseHook(hab, hmqType, hookType, hookProc, hmDll);

Let's look at the parameters:

HAB hab
Anchor block of the calling application.
HMQ hmqType
Two types of handles for the message queue: "NULLHANDLE" if the hook is system wide, or "HMQ_CURRENT" if the hook applies only to the current thread of the calling application.
LONG hookType
There are many types of hooks. Our example will look just at two of them, the input message hook "HK_INPUT", and the send message hook "HK_SENDMSG".
PFN hookProc
This is the address that points to the applications hook procedure. The parameters of this function depend on the hookType, and are usually pointers to a message structure, a destination window handle and anchor block.
The handle of the module that contains the hook procedure. If this procedure is within the EXE, then "NULLHANDLE" is used (and hmqType must be HMQ_CURRENT). In most cases, the procedure is in a DLL, so this would be the value returned by DosQueryModuleHandle() or DosLoadModule().

Why DLLs are Used for Hooks

You might be wondering how an application can get access to data (i.e. messages) that belong to other applications. Doesn't each application have its own private memory block for data, not accessible by other applications? Well, yes they do, but by placing the hook function in a DLL, that DLL now attaches it self to any program that is receiving messages that the hook is also looking at. In effect, the DLL is now part of every running process that uses message queues. DLLs allow sharing of both code and data between executables.

So, if you plan to use hooks on a system wide basis, then place the hooks in a DLL. If you plan to use hooks only within your own application, then the hooks can reside anywhere (EXE or DLL).

Why Atoms Must be Used for Custom Messages

When creating a custom message (as our example program does), one method is to use WM_USER. For example:

    #define WMU_MYMESSAGE   (WM_USER+1)

However, this will quickly fail if we look for this message id in the Input or Send message hooks. Why? There is only one message queue, and if two or more applications use WM_USER+1, then when this message is trapped in the hook, we have no way of determining where that message originated from. What we need is a way of creating a unique message id that wouldn't be used by any other application. The 'partial' solution to this is atoms. 'Partial' because an atom we generate will be unique in the atom table, but still might be the same value as a WM_USER+n used in another application. We will look at one solution to this later on in the Input Hook section below.

Atoms are ULONGs, generated by the operating system, based on a string that we pass when we create an atom. However, that atom is unique only if we use the system atom table (and not a private atom table), and we pass a unique string to the create atom function. In this article, creating a unique string is done by combining: exe_name + desc + randon_number. For example:

            "XrayUpdateDisplay43654"    and

Since the atoms are generated at runtime, they are stored in a global variable, where they can be used later on in the hook function. The function to create an atom is:

    ATOM atomId = WinAddAtom(HATOMTBL atomTable, CHAR * szUniqueName);

In the example Xray program, atoms are created in XrayInit():

HATOMTBL hatomtblSystem;    // handle to system atom table
XrayInit(HWND hwndClient) {
    hatomtblSystem = WinQuerySystemAtomTable(); // get handle to system atom table
    srand((UINT)hwndClient);                    // set unique seed for rand
    // create unique msg string
    // create unique msg string
      // Global.wmu_UpdateDisplay and .wmu_FloatToTop are defined as ATOM
    Global.wmu_UpdateDisplay = WinAddAtom(hatomtblSystem, Global.szUpdateDisplay);
    Global.wmu_FloatToTop = WinAddAtom(hatomtblSystem, Global.szFloatToTop);
    .... enable hooks ...

and destroyed in XrayKill():

    ... disable hooks ...
    WinDeleteAtom(hatomtblSystem, Global.wmu_UpdateDisplay);
    WinDeleteAtom(hatomtblSystem, Global.wmu_FloatToTop);

Note that the atoms must be created BEFORE hooks are enabled, and destroyed AFTER hooks are disabled. Actual use of the atom/user messages is shown in the next two sections.

The Input Message Hook (HK_INPUT)

There are many type of hooks - for looking at the system message queue, for recording and playback of keyboard/mouse activity, help and lockup hooks, etc. (see your PM documentation for the full list). In our example, we're just interested in looking at the system message queue, so the input message and send message hooks are described here.

The input message hook allows us to view messages before they are posted (WinPostMsg()) to a window procedure. Since our hook procedure is passed a pointer to the message, we can also alter the parameters of that message, or we can process the message ourselves. If we process the message, we return TRUE. If not, we return FALSE, and the message is passed on to the next hook or to the final destination.

The message pointer we get has this layout:

typedef struct {
    HWND    hwnd;       // the window that's getting this msg
    ULONG   msg;        // WM_*
    MPARAM  mp1;        // parameters for the message
    MPARAM  mp2;
    ULONG   time;       // message time
    POINTL  ptl;        // pointer position when message was generated
    ULONG   reserved;
    } QMSG;

Here's a snip of the input hook from xraydll.c, the sample program:

BOOL EXPENTRY XrayHookInput(        // HK_INPUT
HAB hab,        // anchor block of window getting msg
PQMSG pQmsg,    // pointer to the message
USHORT fs       // PM_REMOVE (removed from queue) or
                // PM_NOREMOVE (not removed from queue)
    static HWND hwndLast;

    switch(pQmsg->msg) {

        case WM_MOUSEMOVE:
                    if(pQmsg->hwnd != hwndLast) {
                        hwndLast = pQmsg->hwnd;
                    break;                  // pass this msg on

                    // if this is our custom msg, then mp1 = window handle;
                    // mp2 = hab
                    // to double check if this is our msg, match client
                    // hwnds and verify params
                    if(pQmsg->msg == Global.wmu_UpdateDisplay
                        && pQmsg->hwnd == hwndXrayClient
                        && WinIsWindow((HAB)pQmsg->mp2,
                                       (HWND)pQmsg->mp1) == TRUE) {
                        UpdateDisplay((HWND)pQmsg->mp1, (HAB)pQmsg->mp2);
                        return TRUE;        // msg processed, return TRUE
    return FALSE;                           // msg not processed if FALSE

To enable this hook (XrayInit() in xraydll.c):

        if(DosQueryModuleHandle("xraydll", &hmXrayDll))
            return FALSE;
        habXray = WinQueryAnchorBlock(hwnd);
        WinSetHook(habXray, NULLHANDLE, HK_INPUT, (PFN)XrayHookInput, hmXrayDll);

This input hook processes two messages, WM_MOUSEMOVE and Global.wmu_UpdateDisplay. The former is a standard system message, generated whenever the mouse is moved. The latter is a custom user message (a system atom), which we will see is generated in the send message hook, described in the next section below.

In WM_MOUSEMOVE, we check to see if the mouse pointer is now over a new window. If so, we call UpdateDisplay() to display the new data from that window. We then return FALSE, to allow the message to continue down the queue.

In Global.wmu_UpdateDisplay, this is Xrays own message, posted from the send message hook. Since we process it here, we return TRUE to tell the system to remove it from the input queue. Since there is a small chance that our atom will have the same number as another applications WM_USER+n, we double check that this message is being sent to our own client window (hwndXrayClient) and that the passed parameters are valid with WinIsWindow().

The Send Message Hook (HK_SENDMSG)

The send message hook allows us to view messages before they are sent (WinSendMsg()) to a window procedure. Since our hook procedure is passed a pointer to the message, we can also alter the parameters of that message. However, we cannot process this message - it will always be sent.

The message pointer we get has this layout:

typedef struct {
    MPARAM  mp2;    // parameters to the message
    MPARAM  mp1;
    ULONG   msg;    // WM_*
    HWND    hwnd;   // the window that's getting this msg
    ULONG   model;  // PM_MODEL_1X or PM_MODEL_2X

Note it's completely reversed from the layout in the input hook procedure. Obviously written by someone else who has a reverse polish notation calculator on their desk. Here's a snip of the send message procedure from xraydll.c:

HAB habAnchor,          // same anchor block
PSMHSTRUCT psmh,        // similar message pointer
BOOL fInterTask         // TRUE if sent between tasks, FALSE if within a task
    switch(psmh->msg) {
        case WM_MENUSELECT:         // mp2 has the hwnd of the menu window
                    WinPostMsg(hwndXrayClient, Global.wmu_UpdateDisplay,
                               psmh->mp2, 0);
                    WinPostMsg(hwndXrayClient, Global.wmu_FloatToTop, 0, 0);

This send message hook processes two standard system messages. Whenever a new menu item is selected (WM_MENUSELECT), we post a message to ourselves to update our display. NEVER use WinSendMsg() in this hook - or PM will crash. Although this message is posted to the client window of Xray (hwndXrayClient), it will be processed in the input hook (see above). The reason it's not processed here is because this hook is really only for modifying a message, not for other tasks. Generally speaking, it is unsafe to do any kind of processing here, so we simply post messages to be handled later in a safer area (the input hook or better yet, our own client procedure).

The second message, WM_WINDOWPOSCHANGED, is used by Xray to keep itself on top of all windows. The custom user message (.wmu_FloatToTop) is processed in the client procedure (xray.c).

Sharing Data Between DLL, EXE, and Other Processes

When creating a DLL, a .DEF file is used to indicate, among other things, how the data area is to be handled. Since in this example program we want just one data area that is shared by all processes, we have this in the .DEF file: DATA SINGLE SHARED

When using shared data between the DLL and the EXE, a simple export statement doesn't work. You could use a named data segment or dynamically allocate a block of shared memory. Another work around used was setting a pointer to the data area in the DLL. In the EXE, this is initialized with (xray.c):

GLOBALS *Global;            // pointer to global data in the dll
Global = XraySetGlobals();  // initialize global data pointer

And in the DLL (xraydll.c):

GLOBALS Global;             // global data in the dll
    return &Global;   // allow client access to the dlls global data

A word of caution - using DATA SINGLE may cause problems if you use any of the C runtime library functions in the dll code. Try linking in the multithread libraries with the dll, or use DATA MULTIPLE with a shared memory block for globals.

Tips to Prevent Trips

Since every message in the system may be going through the hook, the hook code MUST BE BOMBPROOF. Badly coded, or worse, badly designed hooks can really mess things up big-time. Further, playing with messages before they can be handled by the intended window may interefere with an applications normal functioning (e.g. hooking WM_CONTEXTMENU may prevent the application from displaying its menus).

Get in and out quickly - try and do all or most of your processing in a standard window procedure, and not in the hook function. If necessary, subclass the window, then post messages to this subclassed procedure.

Never use WinSendMsg() in the SendMsgHook(), and it probably shouldn't be used in other hooks either. Use WinPostMsg() to signal events, or pipes/queues.

Use HMQ_CURRENT first in the WinSetHook() function, and thoroughly test your hook code on your own EXE. When it crashes, at least only your application dies, and not PM.

Use DosExitList() to ensure you release the hooks if your process is killed. For example, in main() we call XrayKill() before a normal closing - XrayKill() removes the hooks enabled on startup. If the process is killed for some reason, this function (XrayKill) might never be called. To ensure it does, we use DosExitList that allow proper cleanup in unusual deaths. In Xray.c:

    DosExitList(EXLST_EXIT, (PFNEXITLIST)ExeTrap);
    // trap exceptions to ensure hooks released
    DosExitList(EXLST_ADD, (PFNEXITLIST)ExeTrap);
    XrayInit()          // enable hooks
    ... main window creation and msg loop...
    XrayKill();         // release hooks upon normal closing
    ... standard closeup

Never use auto variables in the DLL if you pass the address of it to the EXE. For example, if another process is currently executing the hook code in the DLL:

    char *buf = "some data";
    WinSendMsg(hwndMyExeClient, WM_USER, buf,...);

Then the buf data is not accessible from the EXE, since the auto variable currently resides on the stack of a different process.

Which messages occur in the Input hook, and which occur in the Send Message hook? No set rule - either place a probe in both hooks to see where it occurs, or better yet, use PMSPY to find out - it places either an S (Send) or an I (Input) before the message to indicate which hook it passed thru. PMSPY was written by Microsoft (thanks Bill!), and is available as at hobbes/os2/dev16. A newer 32 bit version of PMSPY is also available on the Developers Connection CD.

Remember that when the hook code is executing, the window handle (hwnd) in the message is usually for a window in a different process, and not your process. If you plan to make changes to that hwnd (such as subclass), post a message to that hwnd, rather than your own client window handle. When you trap this message in the input hook, you can be assured that the hwnd is owned by the current process, and you can subclass it or make changes.

When your program crashes, you might find that you cannot make/build a new DLL, since it is still loaded in memory. I know of no trick to "kill" this locked DLL - if anyone does please let me know. In the meantime, rebooting will get rid of it. This is a real gotcha sometimes - you make changes to the DLL code, compile cleanly, but don't look at the results of the link, and blindly run the EXE. It will run, but it is still using the old DLL, still loaded in memory. Your changes have "vanished".

XRAY - A Sample Hook Program

Now that you have some background on using hooks, take a look at the source code for Xray ( Xray is a simple window peeker utility, like PMTREE ( at hobbes/wpsutil). It displays window information of the window that is under the mouse pointer, as well as parent window info. If the current window is a menu, you can use the up/down cursor keys to hilight another menu item to see its menu ID. If running Warp 4 - take a look at WarpCenter and its pull down "menus" - you might be surprised!

To reposition or close Xray, press the right mouse button when over the Xray window for a popup menu of options.

For some ideas to get you started, hooks have been used to create screen savers, provide keyboard recording/playback, provide simplified bubble help, and extend the Desktop (Xit, DragText, LtClock).

Source tested under Borland C 2.0 (xraybor.mak) and IBM C 3.0 (xrayibm.mak). The source code is free to use as you please, but as always, Mr. Phelps, if you or any of your hook programs blow up, I will disavow any knowledge of it.