Feedback Search Top Backward Forward

Adventures in Kernel Debugging

Written by David C. Zimmerli



[My apologies for the poor formatting job. If anyone knows how to do this better, while keeping the normal article margin, and not breaking up the output lines, please contact me at]

This article discusses the use of the OS/2 Kernel Debugger (KDB), an essential tool for anyone interested in the internal workings of OS/2, or with a need to debug a device driver or file system driver.

In the following two sections, I will give the general procedure for setting up and using KDB. For more specifics, please refer to the KDEBUG.INF file supplied with KDB, and to the fine 4-volume OS/2 Debugging Handbook from IBM (Part Nos. SG24-4640,41,42, and 43). This latter series is a long-overdue treasure trove of information about OS/2 internal data structures, in addition to being a complete reference on the use of KDB, the Dump Formatter, and System Trace Facility. The CD-Rom accompanying the first volume has most of the text of the series, along with the complete set of debug kernels for all versions and fixpack levels of OS/2. The series can be ordered in the U.S. by calling IBM at 1 (800) 879-2755.

Next, I will go through several examples of using KDB to analyze applications, device drivers and pieces of the kernel. These will serve to illustrate some insights into the system we can gain from KDB, as well as some shortcuts and tricks for navigating the often cryptic interface of this program.

A word of warning before we begin: KDB is a powerful tool, but must be respected as such. You have the ability to write to any area of system memory or execute any code, without regard for privilege levels or kernel states. Thus, there is no such thing as "crash protection" when using KDB. It goes without saying (or ought to) that you should not leave any important data stored in volatile RAM while using KDB, especially if you plan to try anything new or tricky with the system.

Setting up KDB

KDB uses two machines, the Machine Under Test (affectionately known as the MUT) and the debug terminal. These can be located at the same physical site, or they can be linked remotely through a modem connection.

The debug terminal can be any machine with a COM port and capable of acting as an ANSI terminal. This includes any PC or Mac running terminal emulation software, as well as dedicated "dumb terminals" which can be had second-hand for as little as $10. Given a choice, though, it's probably better to use a computer, in order to capture log files and take advantage of terminal emulation programs which are specialized for KDB use.

The MUT is, of course, your OS/2 machine. For remote debugging, you need modems on both ends; for local (on-site) debugging, you only need a "null modem cable", available at most computer stores for $7-$10, to connect the COM ports of the two machines. By default, KDB outputs data on COM2 of the MUT, and you can of course configure your debug terminal to use whichever COM port you wish.

The KDB distribution can be obtained either from the Developer Connection CD-ROMs or the CD-ROM supplied with the Debugging Handbook. Each version and fixpack level of OS/2 has its own set of KDB files (they are produced in the same build which creates the OS/2 system itself), and it is crucial that you use the right files for your installation.

The KDB files consist of "debug versions" of the system files OS2LDR and OS2KRNL and some of the system DLLs. To install KDB, rename OS2LDR and OS2KRNL to OS2LDR.RTL and OS2KRNL.RTL respectively (you may need to un-hide these files first using "attrib -h"). Then copy in the KDB versions. Make a new directory for the debug DLLs-- say \OS2\DEBUG\DLL, copy in the debug DLLs, and add this path to the beginning of your LIBPATH statement.

Also-- and this is one of the most useful aspects of KDB-- symbol files are supplied for the kernel and important system DLLs. These files have the extension .SYM and base names matching the associated executable modules, e.g. OS2KRNL.SYM, PMWIN.SYM, DOSCALL1.SYM, and so on. To use these, just copy them to the same directories where the associated DLLs reside, and they will be loaded automatically when the system starts. You can also create SYM files for your own programs.

Finally, you may need to make some small CONFIG.SYS changes (to enable debug output for PMDD.SYS, for example) and edit your KDB.INI file (if you use other than the default comm configuration). Some versions of KDB include a REXX script to automate the above process for you.

When you have made these changes on the MUT and re-booted, you should get a large number of mysterious informational messages and assorted hieroglyphics on the debug terminal as the system starts. To freeze the MUT, type Ctrl-C on the debug terminal. You should see a display that looks something like the following:

eax=00000000 ebx=7b22002b ecx=80010013 edx=7ba0fc0c esi=7ba093c0 edi=ffffffff
eip=fff46572 esp=00006688 ebp=00006688 iopl=2 -- -- -- nv up ei pl zr na pe nc
cs=0170 ss=0030 ds=0168 es=0168 fs=0000 gs=0000  cr2=111f3302  cr3=001f6000
0170:fff46572 833d7c1ff1ff00 cmp     dword ptr [_ptcbPriQReady (fff11f7c)],+00
and you are now officially in Kernel-Debug-Land.

Basic KDB commands

Anyone who has used the DEBUG utility supplied with MS-DOS-- or a similar low-level debugger-- will feel right at home with the first few KDB commands. However, KDB provides a great deal more functionality than DEBUG, commensurate with the vast improvements of OS/2 over DOS. KDB also operates in the much more complex world of 80386 protected mode-- so if you are hazy on the mechanics of segment selectors, descriptor tables, and virtual addressing, this is a good time to brush up.

As with DEBUG, the "d" (Dump) command displays memory in hex format, while the "u" (Unassemble) command produces a disassembly listing of executable code. Each of these commands takes as an optional argument a memory address, specified as a selector:offset pair, a linear address, or a symbol from the symbol files mentioned above. For example:

##d 400:0
0400:00000000 53 41 53 20 16 00 68 01-1e 00 20 00 2c 00 5c 00 SAS ..h... .,.\.
0400:00000010 6e 00 74 00 80 00 08 00-00 00 18 00 00 01 f0 09 n.t...........p.
0400:00000020 59 09 00 00 a8 04 00 00-00 00 c8 00 e0 2a f1 ff Y...(.....H.`*q.
0400:00000030 f8 3e f1 ff 34 2b f1 ff-f5 b7 f0 ff 98 88 f0 ff x>q.4+q.u7p...p.
0400:00000040 1c 27 f1 ff f0 33 f1 ff-18 4b f0 ff 7c 30 f1 ff .'q.p3q..Kp.|0q.
0400:00000050 68 30 f1 ff 1c 22 f1 ff-fc 24 f1 ff 30 00 fc 3e h0q.."q.|$q.0.|>
0400:00000060 f1 ff 1a 7b f0 ff fe 07-e0 ff 02 08 e0 ff 00 00 q..{p.~.`...`...
0400:00000070 00 00 c8 07 b0 1f 8e fe-c0 00 20 04 e0 05 a8 00 ..H.0..~@. .`.(.

##u _pgSwitchContext
%fff474e4 8b154c7cf0ff       mov     edx,dword ptr [_pPTDACur (fff07c4c)]
%fff474ea a13427f1ff         mov     eax,dword ptr [_pgpPageDir (fff12734)]
%fff474ef f680bc01000001     test    byte ptr [eax+000001bc],01
%fff474f6 7467               jz      %fff4755f
%fff474f8 8bb280000000       mov     esi,dword ptr [edx+00000080]
%fff474fe 8b8680ff0600       mov     eax,dword ptr [esi+0006ff80]
%fff47504 83e041             and     eax,+41          ;'A'
%fff47507 83f841             cmp     eax,+41          ;'A'
%fff4750a 7415               jz      %fff47521
%fff4750c 8b86c0ff0600       mov     eax,dword ptr [esi+0006ffc0]
%fff47512 83e041             and     eax,+41          ;'A'
%fff47515 83f841             cmp     eax,+41          ;'A'

The first command above dumps 80h bytes of memory starting at 400:0, which happens to be the start address of the "System Anchor Segment". This area of memory, identified by the "eye-catcher" bytes SAS, contains pointers to the system's Per-Task Data Areas, Virtual Memory Object Blocks, and so forth. It acts similarly to the SysVars data area in MS-DOS and serves as a "home base" for the important kernel data structures.

The second command above displays the first few instructions of the kernel's context switching routine. The string "_pgSwitchContext" is a symbol from the file OS2KRNL.SYM.

The "r" (Register) command displays the contents of the 80386 registers. By default, this command is executed automatically whenever we break into KDB, whether by typing Ctrl-C at the debug terminal or by hitting an execution breakpoint.

The "e" (Enter) command enables us to modify memory contents directly. Needless to say, modifying memory is a perilous business and it's best not to do too much of it during your first explorations of KDB.

The next few commands allow us to step through code execution in a controlled fashion. The "t" (Trace) command single-steps through code, one assembly instruction at a time. The "p" (Proceed) command does the same thing, with this exception: when a CALL instruction is encountered, the "t" command will step into the sub-routine indicated, returning to the KDB prompt at the first instruction of the subroutine, whereas the "p" command will execute the entire subroutine and return to KDB at the instruction following the CALL.

The "bp" (BreakPoint) command lets us set breakpoints which will activate the KDB prompt automatically when they are reached in the flow of execution. The "g" (Go) command is KDB's way of saying "let 'er rip"-- it resumes normal execution at the current CS:EIP, thus re-activating the MUT until the next breakpoint is hit.

Finally, let's look at a few commands that have no counterpart in MS-DOS DEBUG. The ".p" command lists all the current processes and threads in the system, along with their priorities, execution states, and other data guaranteed to make a hacker's eyes light up.


[ ... ]
 Slot  Pid  Ppid Csid Ord  Sta Pri  pTSD     pPTDA    pTCB     Disp SG Name
*0009# 0016 000b 0016 0001 rdy 031f 7b21f000 7ba7c020 7ba0fc0c 1eb8 14 describe
 0035  0016 000b 0016 0002 rdy 0300 7b277000 7ba7c020 7ba146fc 1eb8 14 describe
 0036  0016 000b 0016 0003 blk 0300 7b279000 7ba7c020 7ba148b0 1eb8 14 describe
 0037  0016 000b 0016 0004 blk 0300 7b27b000 7ba7c020 7ba14a64      14 describe
 000a  0003 0000 0003 0001 blk 0200 7b221000 7ba7c84c 7ba0fdc0      00 lanmsgex
 000c  0010 0000 0010 0001 blk 0200 7b225000 7ba7d8a4 7ba10128      00 epwmux
 000d  0006 0000 0006 0001 blk 0200 7b227000 7ba7e0d0 7ba102dc      00 epwrout
 0019  0006 0000 0006 0002 blk 0200 7b23f000 7ba7e0d0 7ba1174c      00 epwrout
 000e  0007 0000 0007 0001 blk 0200 7b229000 7ba7e8fc 7ba10490      00 lsdaemon
 000f  0008 0000 0008 0001 blk 021f 7b22b000 7ba7f128 7ba10644      00 logdaem
 0010  0009 0000 0009 0001 blk 080b 7b22d000 7ba7f954 7ba107f8 1cf0 00 landll
 0011  000a 0000 000a 0001 blk 0200 7b22f000 7ba80180 7ba109ac      00 vnrminit
 002a  0012 000b 0012 0003 blk 0200 7b261000 7ba82a5c 7ba13440      11 pmshell
 002b  0012 000b 0012 0004 blk 0200 7b263000 7ba82a5c 7ba135f4      11 pmshell
 002c  0012 000b 0012 0005 blk 0200 7b265000 7ba82a5c 7ba137a8      11 pmshell
 0028  0013 0000 0013 0001 blk 0200 7b25d000 7ba811d8 7ba130d8      00 epwmux
 0031  0015 000b 0015 0001 blk 0200 7b26f000 7ba83ab4 7ba1402c 1eb8 13 ssaver
 0034  0014 000b 0014 0001 blk 0200 7b275000 7ba83288 7ba14548 1eb8 12 cmd
[ ... ]

The ".lm" command lists the current executable modules loaded in the system. This includes .EXE, .DLL, and .SYS files.


[ ... ]
hmte=0631 pmte=%fecd561c mflags=06903150 c:\describe\describe.exe
hmte=05ff pmte=%fe9698b8 mflags=06903152 c:\ssaver13\ssaver.exe
hmte=05da pmte=%fe91874c mflags=06903152 c:\os2\cmd.exe
hmte=0353 pmte=%fe8fea68 mflags=06903140 c:\ibmlan\netprog\vnrminit.exe
hmte=063a pmte=%fe96a418 mflags=4698b186 c:\describe\decrt.dll
hmte=061a pmte=%fe969ca8 mflags=0698b188 c:\os2\dll\moncalls.dll
hmte=02da pmte=%fe969c48 mflags=0698b189 c:\ssaver13\ssdll.dll
hmte=060a pmte=%fe969908 mflags=0698b189 c:\ssaver13\dll\emx.dll
hmte=048c pmte=%fe8d8360 mflags=0698b185 c:\muglib\dll\netoem.dll
hmte=0463 pmte=%fe8d83b0 mflags=0698b186 c:\muglib\dll\mailslot.dll
hmte=046e pmte=%fe8e2efc mflags=0698b18a c:\muglib\dll\netspool.dll
hmte=0441 pmte=%fe959ca4 mflags=0698b186 c:\muglib\dll\netapi.dll
hmte=0431 pmte=%fe8deb98 mflags=8698b19a c:\os2\dll\pmmle.dll
hmte=042e pmte=%fe8dec00 mflags=0698b188 c:\os2\dll\fka.dll
hmte=0410 pmte=%fe8ded6c mflags=0698b188 c:\os2\dll\dspres.dll
hmte=0409 pmte=%fe8dee98 mflags=0698b188 c:\os2\dll\8514_32.dll
hmte=0137 pmte=%fe8f5f88 mflags=0698b194 c:\os2\dll\display.dll
hmte=03d4 pmte=%fecd6da4 mflags=0698b198 c:\os2\dll\bvhwndw.dll
hmte=03d2 pmte=%fecd6e2c mflags=0698b188 c:\os2\dll\moucalls.dll
hmte=038c pmte=%fe8fdbe0 mflags=c698b196 c:\os2\dll\pmshapi.dll
hmte=0384 pmte=%fe8fdc58 mflags=8698b198 c:\os2\dll\pmgpi.dll
hmte=037c pmte=%fe8fe49c mflags=c698b194 c:\os2\debug\dll\pmgre.dll
hmte=036b pmte=%fe8fe5f4 mflags=8698b194 c:\os2\debug\dll\pmwin.dll
hmte=0364 pmte=%fe8fe674 mflags=c698b194 c:\os2\dll\pmwp.dll
hmte=0345 pmte=%fe8febc0 mflags=0608b188 c:\ibmcom\dll\acslan.dll
hmte=0322 pmte=%fe903f5c mflags=0698b188 c:\os2\dll\nampipes.dll
hmte=02bf pmte=%fe915e88 mflags=0691b180 c:\os2\mdos\vflpy.sys
hmte=02be pmte=%fe8d922c mflags=0691b180 c:\os2\mdos\vpic.sys
hmte=02bd pmte=%fe948a70 mflags=0691b180 c:\os2\mdos\vdma.sys
hmte=02bc pmte=%fe948b5c mflags=0691b180 c:\os2\mdos\vbios.sys
hmte=00d4 pmte=%fe8f4e18 mflags=0698b188 c:\os2\dll\bvh8514a.dll
hmte=00d0 pmte=%fe8f4f48 mflags=0698b198 c:\os2\dll\bvhvga.dll
hmte=00c2 pmte=%fe8e0eb0 mflags=0698b198 c:\os2\dll\quecalls.dll
hmte=00bd pmte=%fe8e0f04 mflags=0698b198 c:\os2\dll\sesmgr.dll
hmte=00bc pmte=%fe8e0f64 mflags=0698b188 c:\os2\dll\viocalls.dll
hmte=00bb pmte=%fe8f1ad0 mflags=0698b188 c:\os2\dll\bmscalls.dll
hmte=00ba pmte=%fe8f1b24 mflags=0698b188 c:\os2\dll\bkscalls.dll
hmte=00b9 pmte=%fe8f1ca4 mflags=0698b188 c:\os2\dll\bvscalls.dll
hmte=00b7 pmte=%fe8f1d5c mflags=0698b188 c:\os2\dll\nls.dll
hmte=00b3 pmte=%fe8f1da8 mflags=8698b194 c:\os2\dll\os2char.dll
hmte=00b2 pmte=%fe8f1f08 mflags=0698b188 c:\os2\dll\kbdcalls.dll
hmte=00b1 pmte=%fe8f1f5c mflags=0698b188 c:\os2\dll\msg.dll
hmte=00a7 pmte=%fe8ddf18 mflags=8698b594 c:\os2\dll\doscall1.dll
hmte=0079 pmte=%fff0c35e mflags=0002b180 mvdm.dll
hmte=0006 pmte=%fff0b7f5 mflags=0000b980 doscalls.dll
hmte=066a pmte=%fe96b0b0 mflags=0698b1c8 c:\describe\dll\describ6.dll
hmte=0669 pmte=%fe96acd4 mflags=0698b1c8 c:\describe\dll\describ3.dll
hmte=0407 pmte=%fe8f5d80 mflags=0698b1c8 c:\os2\dll\times.fon
hmte=0405 pmte=%fe8f5d58 mflags=0698b1c8 c:\os2\dll\helv.fon
hmte=0403 pmte=%fe8f5e18 mflags=0698b1c8 c:\os2\dll\courier.fon
hmte=0401 pmte=%fe8f4da8 mflags=0698b1c8 c:\os2\dll\sysmono.fon
hmte=03f8 pmte=%fe8f5f08 mflags=4698b1d5 c:\os2\dll\pmatm.dll
hmte=02ab pmte=%fe8f4700 mflags=0608f1c9 c:\ibmcom\protocol\netbios.os2
hmte=0149 pmte=%fe915ee8 mflags=0628a1c9 c:\ibmlan\netprog\netwksta.200
hmte=0148 pmte=%fe915fb0 mflags=0608f1c9 c:\ibmlan\netprog\rdrhelp.200
hmte=013c pmte=%fe912fa8 mflags=0608f1c9 c:\ibmcom\protocol\netbeui.os2
hmte=013a pmte=%fe90bf58 mflags=0608f1c9 c:\network\viny16.os2
hmte=0131 pmte=%fe8fee70 mflags=0608f1c9 c:\os2\log.sys
hmte=0127 pmte=%fe8fef1c mflags=0608f1c9 c:\os2\com.sys
hmte=0126 pmte=%fe8fdcc8 mflags=0608f1c9 c:\os2\mouse.sys
hmte=0125 pmte=%fe8fdd80 mflags=0608f1c9 c:\os2\pointdd.sys
hmte=011b pmte=%fe8def8c mflags=8608f1ca c:\os2\pmdd.sys
hmte=011a pmte=%fe8fbf84 mflags=0608f1c9 c:\os2\dos.sys
hmte=0119 pmte=%fe8fcfd0 mflags=0608f1c9 c:\os2\testcfg.sys
hmte=00e2 pmte=%fe8def10 mflags=0608f1c9 c:\ibmcom\protman.os2
hmte=00de pmte=%fe8f4c50 mflags=0608f1c9 c:\ibmcom\lanmsgdd.os2
hmte=00dd pmte=%fe8f4d00 mflags=0608f1c9 c:\ibmcom\protocol\lanpdd.os2
hmte=00ce pmte=%fe8f4fa0 mflags=0628a1c9 c:\os2\cdfs.ifs
hmte=00cc pmte=%fe8e0d0c mflags=0608f1c9 c:\os2\os2cdrom.dmd
hmte=009f pmte=%fe8e0fb8 mflags=0408e1c9 c:\os2scsi.dmd
hmte=009e pmte=%fe8defb8 mflags=0408e1c9 c:\os2dasd.dmd
hmte=009b pmte=%fe8ddfb8 mflags=0408e1c9 c:\lms206.add
hmte=0098 pmte=%fe8dcf84 mflags=0408e1c9 c:\aha154x.add
hmte=0096 pmte=%fe8dbf98 mflags=0408e1c9 c:\ibm1flpy.add
hmte=0090 pmte=%fe8d9e60 mflags=0408e1c9 c:\print01.sys
hmte=008f pmte=%fe8d9ec4 mflags=0408e1c9 c:\kbd01.sys
hmte=008e pmte=%fe8d9f30 mflags=0408e1c9 c:\screen01.sys
hmte=008d pmte=%fe8d9fb8 mflags=0408e1c9 c:\clock01.sys
This concludes our whirlwind tour of the KDB command set. There's quite a bit more to know here, but rather than duplicate too much of IBM's documentation, I want to move on to some practical applications of KDB techniques.

Starting an application under control of KDB

In this section and the next, we're going to take a look at the PSTAT.EXE utility supplied with OS/2. This will illustrate tracing through an application under the control of KDB, as well as tracing into the kernel code which services one of the Dos* function calls.

Naturally, if we're debugging a program we wrote, or one for which we have the source code, we would probably be using a higher level debugger such as IPMD. But if we don't have the source, or if we want to trace into a Win*, Dos*, etc. API call, we need the full firepower of KDB. So the first question is, how do we start an application such as PSTAT.EXE under the control of KDB?

This turns out to be a surprisingly challenging problem! The difficulty is that we have no breakpoints or symbols in the PSTAT program, and we can't set any breakpoints using "bp" because the program isn't loaded yet. We could start PSTAT on the MUT, then quickly run over to the debug terminal and type Ctrl-C to break in. But clearly this would (at best) get us to a random point in PSTAT's execution, not to the beginning where we presumably want to be. Surely there must be a better method.

One way to attack this problem is to combine KDB with some file-analysis utilities such as EXEHDR and the share-ware disassembler utility IDA by Ilfak Guilfanov. Using either of these utilities on PSTAT.EXE, we learn that the execution entry point of this module is at (for example) offset 1970h of code object 1. Then, using our exhaustive knowledge of the OS/2 executable file format (gleaned from the LXEXE.DOC file on Hobbes) we can trace through the module's Object Table and Object Page Table to locate the absolute offset within PSTAT.EXE of this first instruction-- in my case, offset 1f50h. Finally, we use our favorite hex editor, such as Bennett Baker's HexEdit/2, to change the byte at this offset (first making a note of its original contents) to the byte CCh, which is the Intel opcode for INT 3.

By this method, we have effectively "poked" a breakpoint into PSTAT.EXE at the exact point where we want to start tracing. When we run the thus modified version of PSTAT on the MUT, the INT 3 instruction will cause control to be transferred to KDB as soon as we hit PSTAT's entry point. We can then use the KDB "e" command to restore the original byte at this location.

While this approach works, it has several drawbacks. It is cumbersome to have to use several different utilities on the MUT before even running the program to be debugged-- especially when these involve changing the .EXE file in question. We also have to remember to change back the .EXE to the way it was (or restore it from a backup copy) when we're finished debugging. And if we're debugging a MUT at a remote location, using a modem connection, with a lay user on the other end, we're going to have a wonderful time trying to explain all the above steps to them. There are patch programs which partly automate this process, but it's still not a very elegant solution.

The way out of this dilemma requires some knowledge of the OS/2 program loader. It turns out that there is a kernel data block, the Swappable Module Table Entry (SMTE), which is allocated for each executable module in the system. This block contains, inter alia, the start address of the program, which is where we want to break. Of course, this block is not set up in memory until shortly before the program begins to run. However, the kernel routine DOSLIBIDISP-- and the associated symbol in OS2KRNL.SYM-- is encountered at just the right point in this process, so we can begin by setting a breakpoint there.

So we first break in with Ctrl-C, then set the breakpoint and return control to the MUT:

(We need to make sure no other programs are running on the MUT which might be spawning programs. Otherwise we might hit this breakpoint in a context which doesn't interest us.)

We then start PSTAT on the MUT, and encounter the breakpoint:

eax=0000001f ebx=00000814 ecx=00005690 edx=0000f834 esi=00002000 edi=000006eb
eip=00000294 esp=00003690 ebp=00000000 iopl=2 -- -- -- nv up ei pl nz na po nc
cs=d01f ss=0017 ds=0017 es=0000 fs=150b gs=0000  cr2=00040030  cr3=001f6000
d01f:00000294 b80100         mov     ax,0001          ;br0
And now, using the ".lm" command, we can see that the module has been loaded.

hmte=06eb pmte=%fe96d308 mflags=06903150 c:\os2\pstat.exe
[ ... ]
The "pmte" value is a pointer (the % sign indicates a linear address in KDB syntax) to the Module Table Entry (MTE), the main control block accompanying the PSTAT.EXE module. Let's look at the first 10h bytes of this structure:

##d %fe96d308 l 10
%fe96d308 02 00 eb 06 78 6c 48 fd-38 d3 96 fe b8 bc 96 fe ..k.xlH}8S.~8<.~
(The "l" parameter is an optional argument to the "d" command, and specifies the length of the desired memory dump). Referring to Volume IV of the Debugging Handbook, p. 165, we see that offsets 4-7 in this structure are a pointer to the SMTE for this module. So we continue following the trail by dumping memory at this address:

##d %fd486c78 l 10
%fd486c78 04 00 00 00 01 00 00 00-70 19 00 00 02 00 00 00 ........p.......
A few pages later in the Debugging Handbook, we find the SMTE layout. Offset 4-11 are the initial Object#:EIP of the module-- in this case, 1:1970. Since OS/2 executable images are loaded into memory starting at linear address %00010000, the entry point is at %00011970, or, written as an LDT selector:offset pair, 000f:1970. So we can finally set a breakpoint at the start of PSTAT:

##bp f:1970
Page not present: 000f:00001970
Whoops! We've one hurdle still to overcome. In virtual memory manager parlance, the code and data pages for this module have been "committed" but not yet "allocated". This means that no physical RAM has yet been set aside for these virtual addresses, but virtual page structures have been set up in the kernel so that when we attempt to access these pages, page faults will be generated and physical RAM will be allocated. Fortunately, KDB provides the ".i" command for just this situation. This command forces the associated pages to be read in so that we can set breakpoints within them.

##.i f:1970
##bp f:1970
eax=0000001f ebx=00000814 ecx=00005690 edx=0000f834 esi=00002000 edi=000006eb
eip=00001970 esp=00003690 ebp=00000000 iopl=2 rf -- -- nv up ei pl nz na po nc
cs=000f ss=0017 ds=0017 es=0000 fs=150b gs=0000  cr2=00011970  cr3=001f6000
000f:00001970 fc             cld                      ;br1
And we're at the beginning of PSTAT.EXE. One last caveat: if we're debugging a program which was compiled from C code-- as is the case with PSTAT-- we are now in the C startup code, not in main(). The point at which the C startup code calls main() can be generally recognized as similar to the following:

##u f:19c2
000f:000019c2 ff368702       push    word ptr [0287]     ;  char *envp[]
000f:000019c6 ff368502       push    word ptr [0285]     ;  char *argv[]
000f:000019ca ff368302       push    word ptr [0283]     ;  int argc
000f:000019ce e81be7         call    00ec   ; main()

Tracing into a kernel API call

Next, we will look at how to trace into API calls made by an application.

In general, OS/2 applications run at ring 3 in the Intel 80386 architecture, and many of the familiar Win*, Dos* calls (WinCreateStdWindow, etc.) are also implemented in "user space DLLs", i.e. code that runs at ring 3. Thus, invoking these APIs requires nothing more than an ordinary "near", or intra-segment, call in the 32-bit flat programming model. Furthermore, IBM has graciously provided symbols to help us break at the entry points to most of these functions. The symbols start with "WIN32" or "DOS32" to emphasize that they are in 32-bit code, though this 32-bit code is often a wrapper for older 16-bit code internally.

So suppose, for example, we want to see the locations where the OS/2 Jigsaw game is making calls to WinCreateStdWindow. We set the breakpoint...

and run JIGSAW on the MUT.

eax=00023309 ebx=00000000 ecx=00000000 edx=00010001 esi=00000000 edi=00000000
eip=1a233532 esp=000232d8 ebp=00023314 iopl=2 -- -- -- nv up ei pl zr na pe nc
cs=005b ss=0053 ds=0053 es=0053 fs=150b gs=0000  cr2=11560112  cr3=001f6000
005b:1a233532 55             push    ebp              ;br0
To verify that we did indeed break here as a result of a call from JIGSAW, we use the ".p" command with the "*" option-- this instructs KDB to display only the currently active thread.

##.p *
 Slot  Pid  Ppid Csid Ord  Sta Pri  pTSD     pPTDA    pTCB     Disp SG Name
*0038# 0038 000b 0038 0001 run 0300 7b27d000 7ba83ab4 7ba14c18 1a10 10 jigsaw
We can then look at the top of the stack...

##dd ss:esp l 4
0053:000232d8  00010209 00000001 80000000 00023310
to see the return address-- in this case %00010209. (The extra "d" after the "d" command instructs KDB to dump memory in doubleword format, rather than the default byte format.) To see the call itself, we can subtract 10h from this address and unassemble:

##u %000101f9
%000101f9 fc                 cld
%000101fa 50                 push    eax
%000101fb 6800000080         push    80000000
%00010200 6a01               push    +01
%00010202 b009               mov     al,09
%00010204 e82933221a         call    WIN32CREATESTDWINDOW (%1a233532)
%00010209 83c424             add     esp,+24          ;'$'
%0001020c a30c100200         mov     dword ptr [0002100c],eax
%00010211 833d0c10020000     cmp     dword ptr [0002100c],+00
%00010218 0f94c0             sete    al
%0001021b 25ff000000         and     eax,000000ff
%00010220 0bc0               or      eax,eax
Calls to 16-bit legacy API functions-- other than those implemented in the kernel itself-- are no more complicated than the example above. The only difference is that the control transfer is a "far", or inter-segment call: even though the implementation code is still in user space, CS must be re-loaded with a tiled LDT selector since we're moving into 16-bit code. The symbols for these calls (e.g. VIOGETMODE) of course do not have the "32" infix in them. And, when examining the stack after the call has been made, we must remember to use "d ss:sp" rather than "d ss:esp" lest we overflow the 16-bit stack segment. With these hints, the reader should have no trouble, for example, finding the call to VioGetMode within PSTAT.EXE.

It's worth noting that a few of the Dos* API calls documented in the "Control Program Programming Reference" do not have symbols associated with them. You will get an "Expression error", for example, if you try to set a breakpoint on DosStartSession (or DOS16STARTSESSION or DOS32STARTSESSION). This is because this API call is implemented in a separate DLL by the name of SESMGR.DLL, and, to my knowledge, IBM has not released a symbol file for this module. All we have to go on are the names of the (mostly undocumented) entry points embedded in the DLL itself. It would be possible, as an interesting project, to use information about the *.SYM and *.DLL file formats to write a utility which would generate a symbol file containing at least these entry points.

Finally, we must consider calls to API functions implemented in ring 0 code. This includes most of the familiar Dos* calls such as DosOpen and DosMove. These functions are implemented in a module called DOSCALLS.DLL, which, unlike most DLLs, does not exist as a separate file-- it is linked right into the OS2KRNL file itself. Since this code runs at ring 0, applications must pass through an 80386 "call gate" in order to access it.

The following disassembly shows a call by PSTAT.EXE to the undocumented DosQProcStatus function in DOSCALLS.DLL.

##u f:1306
000f:00001306 ff368e16       push    word ptr [168e]
000f:0000130a ff368c16       push    word ptr [168c]
000f:0000130e ff7608         push    word ptr [bp+08]
000f:00001311 9a00003316     call    1633:0000
When you see a call to a segment:offset address whose offset is 0, as here, you can be fairly sure you are dealing with a call gate. You can be certain by using the KDB "dg" (Dump GDT) command with the selector in question:

##dg 1633
1633  CallG32 Sel:Off=0158:0000442c     DPL=3 P  DWC=2
To see which API call this is, we can use the "ln" (List Near symbols) command to see the symbol at the entry point of the routine:

##ln 158:442c
so the call is to DosQProcStatus, as advertised.

Tracing into this call,

eax=00000000 ebx=000034d0 ecx=000012b1 edx=000019d5 esi=000002da edi=000002da
eip=00001311 esp=000034c8 ebp=000034d6 iopl=2 rf -- -- nv up ei pl zr na pe nc
cs=000f ss=0017 ds=0017 es=0017 fs=150b gs=0000  cr2=0002168e  cr3=001f6000
000f:00001311 9a00003316     call    1633:0000
eax=00000000 ebx=000034d0 ecx=000012b1 edx=000019d5 esi=000002da edi=000002da
eip=0000442c esp=0000647c ebp=000034d6 iopl=2 -- -- -- nv up ei pl zr na pe nc
cs=0158 ss=0030 ds=0017 es=0017 fs=150b gs=0000  cr2=0002168e  cr3=001f6000
0158:0000442c 6a06           push    +06
##d ss:sp l 20
0030:0000647c 16 13 00 00 0f 00 00 00-01 40 00 00 3f 00 da 02 .........@..?.Z.
0030:0000648c c8 34 00 00 17 00 00 00-1c 64 00 00 48 45 a1 7b H4.......d..HE!{
We see that we have landed at the expected address. We also notice that the values of SS and ESP have changed: as part of the protection mechanism, the processor switches to a special ring 0 stack in passing through this call gate. The return address and parameters are on the new stack, as usual, but they are followed by the old values of SS:ESP-- here, 0017:34c8-- and the processor uses these to restore the ring 3 stack when returning from the call.

Now you can trace through kernel API calls to your heart's content. A word of warning, though: the rules of the game are somewhat different in ring 0. There are small but critical sections of code which simply cannot be traced due to KDB re-entrancy problems. In particular, if you try to trace a kernel call all the way back out to the application, you will at some point encounter a call to KMExitKModeEvents, the focal point of OS/2's multi-tasking machinery. You will not be able to trace far into this function, since it involves a potential context switch. Nor can you "p" (Proceed) past it, as the return may happen in a different thread. The easiest solution is to put a breakpoint in the application following the kernel call, and type "g" to allow the scheduler to take its course.

Watching a mouse click

The preceding sections concentrated on debugging the "top half" of the OS/2 system-- what is variously called "task time", or the "synchronous" part of the kernel's operations. But there is also much activity that bubbles up from the "bottom half" of the system, triggered by hardware events such as keypresses, mouse clicks, system timer ticks, and peripheral devices. As most programmers know, these events awaken the attention of the processor through a mechanism called IRQs, or Interrupt Requests. We will now look at a few examples of debugging IRQ handlers.

First, for those of you with a burning desire to understand exactly how a physical click of the left mouse button results in a WM_BUTTON1DOWN message appearing in the System Input Queue, we will take a look at how to explore this process.

Just as "dg" and "dl" dump entries from the Global and Local Descriptor Tables, respectively, the "di" KDB command displays the Interrupt Descriptor Table. This is how it looks on my system:

0000  TrapG32 Sel:Off=0170:fff48da0     DPL=0 P
0001  IntG32  Sel:Off=0170:fff48e4c     DPL=3 P
0002  TaskG   Sel:Off=1e38:00000000     DPL=0 P
0003  IntG32  Sel:Off=0170:fff49008     DPL=3 P
0004  TrapG32 Sel:Off=0170:fff49094     DPL=3 P
0005  TrapG32 Sel:Off=0170:fff490a0     DPL=0 P
0006  TrapG32 Sel:Off=0170:fff490ac     DPL=0 P
0007  TrapG32 Sel:Off=0170:fff49144     DPL=0 P
0008  TaskG   Sel:Off=0088:00000000     DPL=0 P
0009  TrapG32 Sel:Off=0170:fff49194     DPL=0 P
000a  TrapG32 Sel:Off=0170:fff491a4     DPL=0 P
000b  TrapG32 Sel:Off=0170:fff491ac     DPL=0 P
000c  TrapG32 Sel:Off=0170:fff491b4     DPL=0 P
000d  TrapG32 Sel:Off=0170:fff491bc     DPL=0 P
000e  TrapG32 Sel:Off=0170:fff4946c     DPL=0 P
000f  TrapG32 Sel:Off=0170:fff49474     DPL=0 P
0010  TrapG32 Sel:Off=0170:fff4947c     DPL=0 P
0011  TrapG32 Sel:Off=0170:fff49484     DPL=0 P
0012  TrapG32 Sel:Off=0170:fff4948c     DPL=0 P
0013  TrapG32 Sel:Off=0170:fff4948c     DPL=0 P
0014  TrapG32 Sel:Off=0170:fff4948c     DPL=0 P
0015  TrapG32 Sel:Off=0170:fff4948c     DPL=0 P
0016  TrapG32 Sel:Off=0170:fff4948c     DPL=0 P
0017  TrapG32 Sel:Off=0170:fff4948c     DPL=0 P
0018  TrapG32 Sel:Off=0170:fff4948c     DPL=0 P
0019  TrapG32 Sel:Off=0170:fff4948c     DPL=0 P
001a  TrapG32 Sel:Off=0170:fff4948c     DPL=0 P
001b  TrapG32 Sel:Off=0170:fff4948c     DPL=0 P
001c  TrapG32 Sel:Off=0170:fff4948c     DPL=0 P
001d  TrapG32 Sel:Off=0170:fff4948c     DPL=0 P
001e  TrapG32 Sel:Off=0170:fff4948c     DPL=0 P
001f  TrapG32 Sel:Off=0170:fff4948c     DPL=0 P
0021  TrapG   Sel:Off=0000:00000000     DPL=3 P
0050  IntG32  Sel:Off=0170:ffef9ca4     DPL=0 P
0051  IntG32  Sel:Off=0170:ffef9cc0     DPL=0 P
0052  IntG32  Sel:Off=0170:ffef9cdc     DPL=0 P
0053  IntG32  Sel:Off=0170:ffef9cf8     DPL=0 P
0054  IntG32  Sel:Off=0170:ffef9d14     DPL=0 P
0055  IntG32  Sel:Off=0170:ffef9d30     DPL=0 P
0056  IntG32  Sel:Off=0170:ffef9d4c     DPL=0 P
0057  IntG32  Sel:Off=0170:ffef9d68     DPL=0 P
0070  IntG32  Sel:Off=0170:ffef9d9c     DPL=0 P
0071  IntG32  Sel:Off=0170:ffef9dd0     DPL=0 P
0072  IntG32  Sel:Off=0170:ffef9dec     DPL=0 P
0073  IntG32  Sel:Off=0170:ffef9e08     DPL=0 P
0074  IntG32  Sel:Off=0170:ffef9e24     DPL=0 P
0075  IntG32  Sel:Off=0170:ffef9e40     DPL=0 P
0076  IntG32  Sel:Off=0170:ffef9e5c     DPL=0 P
0077  IntG32  Sel:Off=0170:ffef9e78     DPL=0 P
In OS/2, IRQs 0 through 7 are mapped to interrupt numbers 50h through 57h respectively, while IRQs 8 through 15 correspond to interrupts 70h through 77h. (One of the problems with MS-DOS, corrected in OS/2, was that IRQs 0 through 7 were mapped to interrupt numbers 8 through 15. This sometimes caused conflicts with Intel's pre-assigned exception interrupt numbers.)

My mouse is connected to COM1, which in general will be mapped to IRQ4, or interrupt 54h. According to the listing above, interrupt 54h will be dispatched to the address 0170:ffef9d14. We could put a breakpoint at that address and start tracing. There is, however, a shortcut which will take us directly to the mouse driver code. All IRQs are handled by a routine in the kernel known, appropriately enough, as intIRQRouter. Referring again to volume IV of the Debugging Handbook, pp. 210-211, we see that there are kernel structures known as IRQIs (IRQ Information Blocks) and that the global symbol airqi points to an array of these structures. Let's have a look:

##d airqi
%fff08664 d8 2f 91 fe 00 00 00 00-b0 9e 8d fe 01 00 00 00 X/.~....0..~....
%fff08674 00 00 00 00 02 00 02 00-00 00 00 00 03 00 00 00 ................
%fff08684 ec ef 8f fe 04 00 00 00-e0 df 8d fe 05 00 00 00 lo.~....`_.~....
%fff08694 c4 bf 8d fe 06 00 00 00-00 00 00 00 07 00 00 00 D?.~............
%fff086a4 e0 9f 8d fe 08 00 00 00-54 59 91 fe 09 00 00 00 `..~....TY.~....
%fff086b4 00 00 00 00 0a 00 00 00-b0 cf 8d fe 0b 00 00 00 ........0O.~....
%fff086c4 00 00 00 00 0c 00 00 00-ec 6f cd fe 0d 00 08 00 ........loM~....
%fff086d4 00 00 00 00 0e 00 00 00-00 00 00 00 0f 00 00 00 ................
The IRQI struct is 8 bytes long, so the one corresponding to IRQ4 is at %fff08684. The first doubleword of this struct-- %fe8fefec-- points to the device driver (or chain of device drivers) which are "camped out" on this interrupt. We follow the scent:

##d %fe8fefec l 10
%fe8fefec 00 00 00 00 96 0a 80 09-78 09 04 00 00 00 00 00 ........x.......
This data has the format of a DIRQ (Device IRQ) block, the next struct in the Debugging Handbook. We deduce that the corresponding device driver has an interrupt handler at 0980:0a96, and a data segment selector of 0978. And since a device driver's data segment always begins with its device header structure, we can get a few more clues by examining the memory at 0978:0. As a special treat, KDB provides a command to format memory known to be a device header: the ".d dev" command.

##.d dev 978:0
          DevNext: 0968:0000
          DevAttr: c980
         DevStrat: 0473
           DevInt: 01ec
          DevName: MOUSE$
        DevProtCS: 0980
        DevProtDS: 0978
        DevRealCS: 0000
        DevRealDS: 0000
So IRQ4 is handled by a driver with a device name of MOUSE$ (surprise!) and a strategy entry point of 0980:0473. The DevInt address is the entry point of the IDC (Inter-Device Communication) routine, not the interrupt handler.

Which module in the system "owns" code selector 0980, and the associated data selector 0978? That is, which module has its code and data loaded in these segments? If we're able to take an educated guess, we can use the ".lmo" command, which displays a module's segments and memory objects, to verify it. This command takes, as an optional parameter, either a module's hMTE (handle to a Module Table Entry), or its name, sans file extension.

##.lmo `mouse`
hmte=0126 pmte=%fe8fdcc8 mflags=0608f1c9 c:\os2\mouse.sys
seg  sect psiz vsiz hob  sel  flags
0001 0011 01f1 1430 0000 0978 8d49 data iter prel rel
0002 0032 1c9a 1c9a 0000 0980 ad60 code shr prel rel
0003 01ff 1ac3 1ac4 0000 0988 ad60 code shr prel rel
The values in the "sel" column match the selectors we're looking for, so IRQ4 is indeed being handled by the code in MOUSE.SYS. If we really didn't know which module owned these selectors, we could use ".lmo" without any arguments to list all modules and their owned memory objects. This would produce many screens of output, but we could capture this output and search it for the selectors we want; a crude but effective approach.

We will now leave this intriguing line of research and move on to our final topic. For readers who are interested in investigating this subject further, I will offer a few guideposts. The parts of MOUSE.SYS involving I/O port access cannot be productively traced, because they depend on real-time access to the COM port. The next stop on the journey is PMDD.SYS, a driver with the driver name SINGLEQ$; MOUSE.SYS calls PMDD.SYS at its IDC entry point to report mouse events. On the Developer Connection CD-ROMs, IBM provides source code for MOUSE.SYS and a symbol file for PMDD.SYS. The symbol at the IDC entry point for PMDD.SYS is SingleQEP.

Debugging a network driver

Finally, we will use KDB to partially dissect an NDIS MAC network driver used with Warp Connect. The example file is VINY16.OS2, which contains the driver for ComTree Technology's VinyLan network card.

As is well known to system programmers, device drivers operate in three different modes: initialization, kernel, and interrupt. The kernel and interrupt modes correspond, respectively, to the "top half" and "bottom half" of the kernel mentioned earlier. The initialization mode, and the corresponding kernel start-up code (in the kernel segment DOSINITR3CODE and part of DOSCODE), arguably constitute a "third half" of the system; but this code is de-allocated and vanishes from virtual memory before applications begin to run.

Suppose, then, we want to watch the initialization sequence of this network driver. As in the example of running an application, we could rummage through the disk file and poke a CCh (INT 3) opcode into the appropriate place-- in this case, at the beginning of the strategy routine. We would then reboot the MUT and get control just as an INIT request is being passed to the driver. Also as before, programmers with refined taste will wish for a more elegant and "KDB pure" means of accomplishing this result.

Fortunately, KDB provides a convenient way of getting control during system initialization: just hold down the space bar on the debug terminal as the MUT is rebooting. We get a display similar to the following:

eax=00000168 ebx=00000170 ecx=fff40000 edx=7c3e0158 esi=00000400 edi=fff00ef1
eip=0000bdd0 esp=00005f08 ebp=00000433 iopl=3 -- -- -- nv up di pl nz ac po cy
cs=1100 ss=0030 ds=0400 es=0140 fs=0000 gs=0000  cr2=7bd24ffd  cr3=001f6000
1100:0000bdd0 6661           popad
The routine syiInitializeDeviceDriver sets up the request block for the INIT command and passes it to the driver. So we put a breakpoint there, and run until we hit it:

##br e syiInitializeDeviceDriver
Debug register hit
eax=00000001 ebx=ffe30648 ecx=00000000 edx=ffe00000 esi=ffe3005a edi=ff1bffff
eip=00001648 esp=00000fc8 ebp=00000fe0 iopl=3 rf -- -- nv up ei pl nz na po nc
cs=014b ss=0017 ds=0140 es=0433 fs=0000 gs=0000  cr2=ffe3e000  cr3=001f6000
014b:00001648 c8060000       enter   0006,00          ;br0
Here we have used a new type of breakpoint command, the "br e" command. This command is usually interchangeable with "bp", but it uses the special 80386 debug registers, rather than poking a CCh opcode at the breakpoint. Most KDB users prefer "br e" command as being less invasive and avoiding the risk of leaving stray CCh bytes around if program flow takes an unexpected turn.

Tracing into syiInitializeDeviceDriver for a few instructions, and experimenting a bit, we find that on entry to this function, the word at DS:0976 points to the device header of the device about to be initialized. We display the header:

##.d dev (wo(ds:978)):(wo(ds:976))
          DevNext: 0648:001e
          DevAttr: 2980
          DevStrat: 209c
          DevInt: 0000
          DevName: CD-ROM1$
          DevProtCS: 0650
          DevProtDS: 0648
          DevRealCS: 0000
          DevRealDS: 0000
and see that this is not the driver we want. (The "wo" function, part of KDB's elaborate expression evaluator, takes the WOrd at the address specified.) So we could just keep typing "g" and dumping the header as above until we find the device name of our network driver, namely VINY16$$.

But a typical OS/2 system may have 20, 30, or more drivers loaded. If ours is towards the end, we're going to get awfully tired of typing the above expression, especially if the debug terminal doesn't have a command recall feature. Is there a better way?

As it turns out, the breakpoint commands take an optional command argument, in single quotes, to determine what happens when the breakpoint is hit. There is also a "j" command for conditional execution whose format is as follows:

j condition "command"
By combining these features, we can construct a breakpoint which will automatically "g" (resume execution) if the first letter of the device name (at offset 0ah in the device header) is not the character 'V'. Our new breakpoint is set up thus:

##br e syiInitializeDeviceDriver 'j by( (wo(ds:978)):(wo(ds:976)) + 0a) != 56 "g"'
And now:

Symbols linked (pmdd)
Debug register hit
eax=00000001 ebx=fd050a48 ecx=fe8f0000 edx=00060000 esi=000015bc edi=fe8fffff
eip=00001648 esp=00000fc8 ebp=00000fe0 iopl=3 rf -- -- nv up ei pl nz na po nc
cs=014b ss=0017 ds=0140 es=0433 fs=0000 gs=0000  cr2=fc5d4000  cr3=001f6000
014b:00001648 c8060000       enter   0006,00          ;br0
##.d dev (wo(ds:978)):(wo(ds:976))
          DevNext: 0a48:ffff
          DevAttr: 8080
          DevStrat: 0000
          DevInt: 0000
          DevName: VINY16$$
          DevProtCS: 0a50
          DevProtDS: 0a48
          DevRealCS: 0000
          DevRealDS: 0000
Finally, we put a breakpoint on the strategy routine of VINY16$$:

##br e a50:0
Debug register hit
eax=00000004 ebx=fd0534ca ecx=fe8f0000 edx=00060000 esi=000015bc edi=fe8f0000
eip=00000000 esp=00000f94 ebp=00000fc6 iopl=3 rf -- -- nv up ei pl zr na pe nc
cs=0a53 ss=0017 ds=0a48 es=0140 fs=0000 gs=0000  cr2=fc5d4000  cr3=001f6000
0a53:00000000 268a4702       mov     al,byte ptr es:[bx+02] ;br1     es:34cc=00
and we're at the start of the INIT command.

Notice that this code is running at application, not system privilege level; the DOSINITR3CODE segment, as its name implies, exists to call device drivers' initialization routines at ring 3. This is the only case of code located in kernel space (i.e. above 512 MB in virtual address) running at ring 3; all other such code runs at ring 0. Also notice that the value of IOPL in the register dump is 3; it was always 2 in previous sections. This is to allow the network driver's INIT routine to check for the presence of the network card by manipulating I/O ports.


In my opinion, what KDB lacks in clarity and intuitiveness, it more than makes up for with the control and insight it provides into the OS/2 system. Along with several other tools in the programmer's arsenal, such as hex editors, executable-file dump utilities, and disassemblers, it gives us a way to learn about parts of OS/2 that are simply not documented in any public materials.

I welcome any feedback and comments on this article. Contact me at:

Happy debugging!

Copyright © 1996, David C. Zimmerli. All rights reserved.