CPGuide - Kernel Debugger Communications Protocol

Reprint Courtesy of International Business Machines Corporation, © International Business Machines Corporation

Control Program Programming Guide and Reference

Introduction to the Control Program Control Program Functions Keyboard Functions Mouse Functions Video Functions Data Types Errors Debugging Kernel Debugger Communications Protocol Device I/O Dynamic Linking Error Management Exception Management Extended Attributes File Management File Names File Systems Generic IOCtl Commands Memory Management Message Management National Language Support Pipes Program Execution Control Queues Semaphores Timers Notices Glossary

The Kernel Debugger is essentially a replacement OS/2 Kernel module that contains a built-in debugger component. The kernel debugger can be used to halt system execution, inspect and alter memory and registers, and display system control blocks. The kernel debugger is described in detail in The OS/2 Debugging Handbook - Volume II, IBM publication number SG24-4641.

Generally the kernel debugger communicates over a serial communications link with a terminal emulator program running on another machine. This allows a user to debug a problem by issuing kernel debugger commands in the emulator program and seeing the results displayed on the debug console.

To automate the debugging process, or to provide a high-level language debugging environment, this interaction with the kernel debugger could instead be handled by a program running on the other machine. This debug engine would interact with the user, convert the user's debugging request to a series of kernel debugger commands, issue them, and then present the response from the kernel debugger back to the user in a user-friendly format.

Communications between the debug engine and the kernel debugger can proceed in one of two modes:

raw (dumb TTY) mode

ASCII characters are sent to the kernel debugger one at a time. The kernel debugger echoes each character. A carriage return (^M or 0x0d) ends a line. The kernel debugger returns data to the debug engine one ASCII character at a time.

packet mode

Packets are sent to the kernel debugger. A packet consists of a fixed sized header followed by zero or more bytes of data. The kernel debugger returns data to the debug engine in packets.

Raw Mode

Any kernel debugger command may be sent in raw mode. Debug engines that communicate in packet mode may wish to use raw mode to issue a .B command to set the communication rate for the serial connection.

To enter raw mode from packet mode, or to get the kernel debugger's attention while the system is running and enter raw mode, the debug engine should send a break character (^C or 0x03) and wait for the kernel debugger to issue a prompt.

Packet Mode

To enter packet mode from raw mode, or to get the kernel debugger's attention while the system is running and enter packet mode, the debug engine should send the KDP break character (0x1f). If the system was running, the kernel debugger will respond with an event packet containing a CVK_RET_ASYNC event. If the system was quiesced, the kernel debugger will not respond at all.

Packet Format

The format of a packet is as follows:

   ┌──────┬───────────────────────┬─────────...────────────┬──────┐
   │ 0x1d │ 10-byte packet header │ packet body (optional) │ 0x1e │
   └──────┴───────────────────────┴─────────...────────────┴──────┘

The packet header and the packet body, if present, are bitstuffed. The data is treated as a stream of bits and is broken into seven-bit chunks. Each chunk is put into the seven low-order bits of a byte and the high order bit of the byte is set. The bitstuffed data is padded at the end with zero bits to the next byte boundary. For example, the header 00009540 0000ac57 when bitstuffed becomes 808092d4 808081ac abc0.

Packet Header Format

The header, prior to bitstuffing, contains a 4-byte logical ID field, a 2-byte length field, and a 2-byte checksum. The checksum of n bytes of data is computed as follows:

  unsigned char  data[ ];
  unsigned short checksum = 0xa1e8;
  for (i = 0; i < n; i++)
  {
    checksum += data[i];
    checksum  = (checksum << 3) + (checksum >> 13);
  }

Packet Data Format

All multibyte items are presumed to appear in little-endian order. Thus, the checksum computed for the header 00009540 0000 is 0x57ac; when stored in the header the low order byte (0xac) appears first.

The header length field contains the number of bytes of data in the packet body before the packet body is bitstuffed. If the header length field is zero, there is no packet body. If a packet body is present, it includes a 2-byte checksum that is not accounted for in the header length field. For example, if the header length field is 0x12, there are actually 20 bytes in the unbitstuffed packet body.

The logical ID field takes one of the following forms (sequence numbers are one byte long):

CVK_HDR_DATA - 0x8000                          |
               0x4000 if "flast" flag is set   |
              ((index number & 0x3f) << 8) |
               sequence number

CVK_HDR_ACK - (0x4000 | ((index number & 0x3f) << 8) | sequence number) << 16

CVK_HDR_NACK- (0xc000 | ((index number & 0x3f) << 8) | sequence number) << 16

Maximum Packet Size

The maximum packet size for a bitstuffed packet is defined by CVK_PACKET_MAXSIZE as 623. This is derived by:

  Start byte      1
  Header         10
  Data          611
  End Byte        1

General Considerations

The debug engine initiates all transactions with the kernel debugger, normally by sending a command while the kernel debugger is waiting to receive one. In this case, the kernel debugger expects to receive a CVK_HDR_DATA packet and will discard any CVK_HDR_ACK or CVK_HDR_NACK packets it receives. (The kernel debugger also discards a packet if its header cannot be unbitstuffed or has a bad checksum. The kernel issues a CVK_HDR_NACK packet containing the sequence number and index number from the original CVK_HDR_DATA packet if the packet's body cannot be unbitstuffed or has a bad checksum.)

Once the kernel receives a valid CVK_HDR_DATA packet, it extracts the sequence number and index number and uses them to construct a CVK_HDR_ACK packet, which it returns to the debug engine. (The flast flag in the CVK_HDR_DATA packet is ignored.) The kernel debugger then performs the action requested by the command and returns the result.

In general, the result is returned in a single CVK_HDR_DATA packet whose sequence number matches the sequence number contained in the debug engine's original command packet, whose index number is zero, and whose flast flag is FALSE. After the result is transmitted, the kernel debugger waits for a response from the debug engine. The kernel is expecting either a CVK_HDR_ACK packet whose sequence number and index number match those sent in the result _or_ a CVK_HDR_DATA packet (containing the next command). If the kernel debugger receives any other response, it resends the CVK_HDR_DATA packet containing the result.

If the debug engine sends a KDP break character while the victim machine is running, either to initiate a transaction or to regain control after the victim machine has resumed execution, the kernel debugger responds with a CVK_HDR_DATA packet whose sequence number matches the sequence number from the CVK_HDR_DATA packet that caused the system to resume. There is no such packet when the kernel debugger responds to the first KDP break sent by the debug engine. The sequence number in that case contains garbage.

The kernel debugger does not generate replies for some commands, such as reboot, and the replies to commands that cause the victim machine to resume execution, such as resume or step, are not sent until an event such as a breakpoint, module load, or break signal from the debug engine, has caused the victim machine to quiesce.

The kernel debugger responds somewhat differently to the CVK_CMD_RAW command, which is used to issue arbitrary kernel debugger commands while in packet mode. Each line in the response is returned in a separate CVK_HDR_DATA packet whose sequence number matches the sequence number in the CVK_CMD_RAW command's header. The index number in the first reply packet is 0; the index number increases by 1 in each successive reply packet (and wraps from 63 to 0). The debug engine should return a CVK_HDR_ACK packet with the appropriate sqeuence number and index number after each reply packet is received.

The kernel debugger does not manipulate or increment sequence numbers and uses them only to generate ACKs and NACKs and to match ACKs with replies. A debug engine could use the same sequence number for every request, but this is not recommended.

Kernel Debugger Packet Responses

┌──────────────────────────────┬──────────┬──────────────────────────────┐
│Event                         │Code      │Description                   │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│CVK_RET_SUC                   │0x0000    │Success                       │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│CVK_BAD_COMMAND               │0x0002    │Unrecognized command          │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│CVK_RET_PAGEIN                │0xffef    │Discarded page reloaded       │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│CVK_RET_TEND                  │0xfff0    │Task died                     │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│CVK_RET_TNEW                  │0xfff1    │Task created                  │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│CVK_RET_ASYNC                 │0xfff5    │Asynchronous halt (break)     │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│CVK_RET_LIB                   │0xfff8    │Module loaded                 │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│CVK_RET_GPF                   │0xfff9    │General protection fault      │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│CVK_RET_KIL                   │0xfffa    │Module unloaded               │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│CVK_RET_NMI                   │0xfffb    │Non-maskable interrupt        │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│CVK_RET_BPT                   │0xfffc    │Software breakpoint (INT3)    │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│CVK_RET_TBT                   │0xfffd    │Single step                   │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│CVK_RET_ERR                   │0xffff    │Failure                       │
└──────────────────────────────┴──────────┴──────────────────────────────┘

Events Reported by the Kernel Debugger

The following is a summary of the data reported when the kernel debugger sends an event in packet mode:

CVK_RET_GPF Events

┌──────────────────────────────┬──────────┬──────────────────────────────┐
│Event                         │Code      │Description                   │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│KDP_T_DIVIDE                  │  0       │Divide by zero exception      │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│KDP_T_INTO                    │  4       │Overflow Interrupt (INTO      │
│                              │          │instruction)                  │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│KDP_T_BOUND                   │  5       │Bounds check (BOUND           │
│                              │          │instruction)                  │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│KDP_T_INVALID_OPCODE          │  6       │Invalid operation             │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│KDP_T_EXTENSION               │  7       │Coprocessor not available     │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│KDP_T_DOUBLE_EXCEPTION        │  8       │Double exception              │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│KDP_T_EXTENSION_SEG_OVERRUN   │  9       │Coprocessor segment overrun   │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│KDP_T_INVALID_TSS             │ 10       │Invalid TSS                   │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│KDP_T_SEG_NOT_PRESENT         │ 11       │Segment not present           │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│KDP_T_STACK_SEG               │ 12       │Stack exception               │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│KDP_T_GP_FAULT                │ 13       │General protection fault      │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│KDP_T_PAGE_FAULT              │ 14       │Page fault                    │
└──────────────────────────────┴──────────┴──────────────────────────────┘

Fields returned in cvkcmd_s are as follows:

• Cmd CVK_RET_GPF
• Value Event code from the table above.
• OffV Linear address in CS:(E)IP at time of event.
• SegV Slot number of thread.
• MTE MTE entry of executable running in process.
• PID PID of process that generated the event.
• TID TID of thread that generated the event.
• DBit Flags from CS selector.
• Reg Registers at time of event.
• MemCache Not used.

CVK_RET_TBT Events

A CVK_RET_TBT event is generated when a single step occurs or when a debug trap, such as an event triggered by one of the hardware debug registers, occurs.

┌──────────────────────────────┬──────────┬──────────────────────────────┐
│Event                         │Code      │Description                   │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│KDP_T_SSTEP                   │  1       │Single step or debug trap     │
└──────────────────────────────┴──────────┴──────────────────────────────┘

Fields returned in cvkcmd_s are as follows:

• Cmd CVK_RET_TBT
• Value Not used.
• OffV Linear address in CS:(E)IP at time of event.
• SegV Slot number of thread.
• MTE MTE entry of executable running in process.
• PID PID of process that generated the event.
• TID TID of thread that generated the event.
• DBit Flags from CS selector.
• Reg Registers at time of event.
• MemCache Not used.

CVK_RET_BPT Events

┌──────────────────────────────┬──────────┬──────────────────────────────┐
│Event                         │Code      │Description                   │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│KDP_T_BREAKPOINT              │  3       │Software breakpoint           │
└──────────────────────────────┴──────────┴──────────────────────────────┘

Fields returned in cvkcmd_s are as follows:

• Cmd CVK_RET_BPT
• Value Not used.
• OffV Linear address in CS:(E)IP at time of event.
• SegV Slot number of thread.
• MTE MTE entry of executable running in process.
• PID PID of process that generated the event.
• TID TID of thread that generated the event.
• DBit Flags from CS selector.
• Reg Registers at time of event.
• MemCache Not used.

CVK_RET_NMI Events

┌──────────────────────────────┬──────────┬──────────────────────────────┐
│Event                         │Code      │Description                   │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│KDP_T_NMI                     │  2       │Non-maskable interrupt        │
└──────────────────────────────┴──────────┴──────────────────────────────┘

Fields returned in cvkcmd_s are as follows:

• Cmd CVK_RET_NMI
• Value Not used.
• OffV Linear address in CS:(E)IP at time of event.
• SegV Slot number of thread.
• MTE MTE entry of executable running in process.
• PID PID of process that generated the event.
• TID TID of thread that generated the event.
• DBit Flags from CS selector.
• Reg Registers at time of event.
• MemCache Not used.

CVK_RET_SUC Events

┌──────────────────────────────┬──────────┬──────────────────────────────┐
│Event                         │Code      │Description                   │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│KDP_T_SUCCESS                 │100       │When last CVK_CMD_RAW response│
│                              │          │is sent.                      │
└──────────────────────────────┴──────────┴──────────────────────────────┘

• Cmd CVK_RET_SUC

CVK_RET_ASYNC Events

┌──────────────────────────────┬──────────┬──────────────────────────────┐
│Event                         │Code      │Description                   │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│KDP_T_ASYNC_TRAP              │101       │KDP break received.           │
└──────────────────────────────┴──────────┴──────────────────────────────┘

Fields returned in cvkcmd_s are as follows:

• Cmd CVK_RET_ASYNC
• Value Not used.
• OffV Linear address in CS:(E)IP at time of event.
• SegV Slot number of thread.
• MTE MTE entry of executable running in process.
• PID PID of process that generated the event.
• TID TID of thread that generated the event.
• DBit Flags from CS selector.
• Reg Registers at time of event.
• MemCache Not used.

CVK_RET_LIB and CVK_RET_KIL Events

┌──────────────────────────────┬──────────┬──────────────────────────────┐
│Event                         │Code      │Description                   │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│KDP_T_LINK                    │102       │Module loaded                 │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│KDP_T_UNLINK                  │103       │Module unloaded               │
└──────────────────────────────┴──────────┴──────────────────────────────┘

Fields returned in cvkcmd_s are as follows:

• Cmd CVK_RET_LIB or CVK_RET_KIL
• Value MTE handle of module in question.
• OffV Not used.
• SegV Slot number of thread.
• MTE MTE entry of executable running in process.
• PID PID of process that generated the event.
• TID TID of thread that generated the event.
• DBit Not used.
• UCHAR NAME[ ] Null-terminated full pathname of module in question (immediately follows DBit and overlays Reg and MemCache)

CVK_RET_TNEW and CVK_RET_TEND Events

┌──────────────────────────────┬──────────┬──────────────────────────────┐
│Event                         │Code      │Description                   │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│KDP_T_TASK_CREATE             │104       │Task create                   │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│KDP_T_TASK_END                │105       │Task died                     │
└──────────────────────────────┴──────────┴──────────────────────────────┘

Fields returned in cvkcmd_s are as follows:

• Cmd CVK_RET_TNEW or CVK_RET_TEND
• Value Not used.
• OffV Not used.
• SegV Not used.
• MTE Not used.
• PID PID of process that was created or died.
• TID 1 (Primary TID in process)
• DBit Not used.
• Reg Not used.
• MemCache Not used.

CVK_RET_PAGEIN Events

┌──────────────────────────────┬──────────┬──────────────────────────────┐
│Event                         │Code      │Description                   │
├──────────────────────────────┼──────────┼──────────────────────────────┤
│KDP_T_PAGEIN                  │106       │Discarded page reloaded       │
└──────────────────────────────┴──────────┴──────────────────────────────┘

Fields returned in cvkcmd_s are as follows:

• Cmd CVK_RET_PAGEIN
• Value Not used.
• OffV Address of the page in question.
• SegV Not used.
• MTE MTE handle of the module that contains the page in question.
• PID Not used.
• TID Not used.
• DBit Not used.
• Reg Not used.
• MemCache Not used.

Kernel Debugger Packet Commands

The following table lists the kernel debugger packet commands:

Command	Code	Description	CVK_CMDSIZE	CVK_RETSIZE
CVK_CMD_RMEM	1	Read memory.	18	20
CVK_CMD_RREG	3	Read registers.	18	24
CVK_CMD_WMEM	4	Write memory.	20	6
CVK_CMD_WREG	6	Write registers.	20 + sizeof(RegSa_struc)	2
CVK_CMD_RUN	7	Resume execution.	6	0
CVK_CMD_KILL	8	Reboot victim machine.	2	0
CVK_CMD_STEP	9	Single step.	2	0
CVK_CMD_NUMTOBASE	13	Get object/segment information.	14	14
CVK_CMD_LIBNAME	16	Get module information.	6	6
CVK_CMD_RAW	20	Perform kernel debugger command.	6
CVK_CMD_DBIT	22	Get selector information.	20
CVK_CMD_RSTEP	23	Range step.	10	0
CVK_CMD_SCANMTE	24	Scan module table.	2	6
CVK_CMD_SCANTCB	25	Scan thread control blocks.	6	10
CVK_CMD_SEL2LIN	26	Convert selector:offset to linear address.	18	6
CVK_CMD_LIN2SEL	27	Convert linear address to selector:offset.	18	12
CVK_CMD_OBJCOUNT	28	Get number of objects/segments in module.	6	6
CVK_CMD_SCANOBJ	29	Scan object/segment table.	14	10
CVK_CMD_SELINFO	30	Get selector information.	18	20
CVK_CMD_RNPX	31	Read NPX state.	18	128
CVK_CMD_WNPX	32	Write NPX state.	128	60
CVK_CMD_ENA	33	Enable optional features.	6	2
CVK_CMD_DIS	34	Disable optional features.	6	2
CVK_CMD_PIREG	35	Register for PAGEIN notification.	14	2
CVK_CMD_PIDRG	36	Deregister for PAGEIN notification.	14	2

Each command is described below, along with the input parameters and the output values. Parameters are usually passed in a cvkcmd_s If a given field in the cvkcmd_s structure is not listed, its value is not used.

There are constants called CVK_CMDSIZE_xxx and CVK_RETSIZE_xxx which provide the size of the command and the size of the response from each command, where xxx is substituted with the last part of the command name. For CVK_CMD_PIDRG, the constants would be called CVK_CMDSIZE_PIDRG and CVK_RETSIZE_PIDRG. The values for these constants are provided in the last two rows of the table above.