Implementing Flyout Menus in OS/2

By Mike La Martina

[Click flyout.zip to download this article along with the source code samples. --Ed. ]

My granddaughter was watching me use OS/2 one day and she asked if the menus were broken. I asked what she meant and she explained that on her computer, Windows NT 4.0, the menus automatically opened when she moved the mouse over them. I told her that in OS/2 the menus were not supposed to open automatically. When she asked why not, I said that automatic menus were not CUA (Common User Access architecture) behaviour.

I told her I could make the menus open automatically. Thus was born the fly-out menu project. The goal of this project was to add the fly-out menus to an application without modifying the behaviour of the entire system. Some people don't like fly-out menus, so that should be a user selectable feature of an application. I also wanted the implementation to be as simple as possible.

Since I had no idea how to do this at first, I hit the documentation. Where had I seen fly-out menus in OS/2 before? The WarpCenter does this when the config.sys specifies SET MENUSFOLLOWPOINTER=ON. But I don't have the source for WarpCenter. Then I remembered that PC/2 has something called Dynamic Menus that also do this, and I do have the source for PC/2, thanks to Roman Stangl. The PC/2 implementation consists of a System Queue hook, which was beyond what I wanted to do.

From the PC/2 code I saw what to do:
 * 1) Subclass the Menu.
 * 2) Process the WM_MOUSEMOVE message in the new MenuProc.

WM_MOUSEMOVE processing.
The idea is to identify the menu item that is under the mouse pointer's location. The (x, y) coordinates of the mouse are in mp1. The rectangle of each menu item must be identified and it must be determined which rectangle is under the mouse.

To find the rectangles of the menu items, we first send the menu a MM_QUERYITEMCOUNT message to determine how many menu items exist for this menu. Then we loop through the menu items sending MM_ITEMFROMPOSITION to obtain the ID of each menu item. When a valid ID is returned MM_QUERYITEMRECT is sent to the menu for that ID to find its rectangle.

Now that we have the mouse position and the menu item rectangle, the WinPtInRect API will tell if the point is within the rectangle. If it is, we must process this menu item.

That means we send the MM_SELECTITEM message to the menu for this menuitem's ID. But we are not done yet. The menu item may be a submenu. If it is, we want to select the submenu's entry as well. To find out if a menu item is a submenu, we must access the item's MENUITEM structure. The MM_QUERYITEM message returns the contents of the MENUITEM for a given ID. If the menu item has the MIS_SUBMENU style, we find the window handle of the submenu from hwndSubMenu in the MENUITEM structure. Then the submenu item identity is found by another MM_ITEMFROMPOSITION this time sent to the hwndSubMenu, for the first submenu item at position 0. If the ID is valid, that item is selected. Since the item may itself be a submenu, its MENUITEM structure is retrieved and examined to see if it has the MIS_SUBMENU style. The iterative evaluation of menu items for submenus is contained within a while loop that tests for the MIS_SUBMENU style. When the while loop is broken, a static variable sLastId is set to the sCurrentId. The purpose of sLastId is to prevent reprocessing the same menu item. If the mouse has not moved to a different menu item, some overhead is avoided by an early exit from the processing.

At this point the WM_MOUSEMOVE logic is complete. I wrapped the WM_MOUSEMOVE logic in a window procedure that will be used to subclass WC_MENU.

Menu Subclassing.
The Main Menu window handle must be known before the WC_MENU can be subclassed. In my test application I use FCF_MENU to load a menu template from its resource file. Ordinarily, the menu's window handle is not exposed in the code. The WinWindowFromID API is used with ID FID_MENU to obtain the Menu window handle. Then WinSubclassWindow was used to replace the menu window procedure with my own. I installed this code in the test application and compiled it. The initial result was that when the mouse pointer moved over the main menu bar, the main menu item under the mouse position did indeed fly out, but as the mouse moved over the menu items nothing happened. The debugger showed the WM_MOUSEMOVE messages were not being received by my window procedure when the mouse moved over the submenu items. I then realized that I needed to subclass every menu, and I had just done the main one.

A Subclassing strategy.
I changed the subclassing logic from a single call to the WinSubclassWindow API to a routine that subclassed the menu whose handle is passed to it, and then loops through the menus items looking for submenus. When one is found, the subclassing routine recursively invokes itself to subclass that submenu. When I replaced the subclassing call in the test program with a call to this recursive routine, the submenus were all subclassed and multilevel menu fly-out was achieved. Part of the goal was achieved. I could cause menus to fly-out. Now I had to make it user controllable, which meant I had to be able to turn off the subclassing.

UnSubclassing.
Another routine was created to remove the subclassing. It too is a recursive routine that is the same as the Subclassing routine, except it replaces my menu procedure with the original window procedure. To test the controllability of the fly-out menus, I made two menu items that called the Subclass and UnSubclass routines. These routines seemed to work until I tried accidentally tried to subclass the menus when they were already subclassed. That caused an abnormal termination. Now what to do? I could disable the menu item once subclassing was in effect, but that is not a self-contained solution. It avoids a problem, but the subclassing code is still broken.

State flags seemed in order. First I tried a simple flag to say subclassing was in effect, and I checked it in the subclassing routine, and turned it on after subclassing the main menu. Because of the recursive nature of the routine, this flag prevented subclassing beyond the main menu and I was back to where I started. I added a second flag to represent Main Menu processing. Things were getting ugly. Rather than upset the elegance of the recursion, I created wrappers for the Subclass and UnSubclass routines, and put state checking in the wrapper functions. This allowed me to eliminate the flags and use the field that holds the original WC_MENU window procedure address. If it is zero, subclassing cannot be in effect. It becomes non-zero when updated by the subclassing routine.

The test application was run successfully with this code, and the fly-out menus were under user control. I then added fly-out menus to the IBM VACPP 3.0 sample code in IBMCPP\SAMPLES\TOOLKIT\PM\CONTROLS directory. I copied the .obj file to that directory, and added a call to FlymenuOn in the sty_main.c code. I also had to add code to get the main menu window handle as indicated above. After a recompile I had fly-out menus in that program as well. I added the support to several other samples I had on my computer as well. As long as I subclassed every submenu everything worked fine.

I do not know if this is the best way to achieve the stated goal, but it does seem to work. This code is simple, and user controllable.

When I showed my fly-out menus to my granddaughter, she was unimpressed. When I asked why not she told me that she didn't like them that much. But now there is a choice.