CPGuide - Kernel Debugger Communications Protocol

Reprint Courtesy of International Business Machines Corporation, © International Business Machines Corporation

Control Program Programming Guide and Reference
Introduction to the Control Program Control Program Functions Keyboard Functions Mouse Functions Video Functions Data Types Errors Debugging Kernel Debugger Communications Protocol Device I/O Dynamic Linking Error Management Exception Management Extended Attributes File Management File Names File Systems Generic IOCtl Commands Memory Management Message Management National Language Support Pipes Program Execution Control Queues Semaphores Timers Notices Glossary

The Kernel Debugger is essentially a replacement OS/2 Kernel module that contains a built-in debugger component. The kernel debugger can be used to halt system execution, inspect and alter memory and registers, and display system control blocks. The kernel debugger is described in detail in The OS/2 Debugging Handbook - Volume II, IBM publication number SG24-4641.

Generally the kernel debugger communicates over a serial communications link with a terminal emulator program running on another machine. This allows a user to debug a problem by issuing kernel debugger commands in the emulator program and seeing the results displayed on the debug console.

To automate the debugging process, or to provide a high-level language debugging environment, this interaction with the kernel debugger could instead be handled by a program running on the other machine. This debug engine would interact with the user, convert the user's debugging request to a series of kernel debugger commands, issue them, and then present the response from the kernel debugger back to the user in a user-friendly format.

Communications between the debug engine and the kernel debugger can proceed in one of two modes:

raw (dumb TTY) mode: ASCII characters are sent to the kernel debugger one at a time. The kernel debugger echoes each character. A carriage return (^M or 0x0d) ends a line. The kernel debugger returns data to the debug engine one ASCII character at a time.
packet mode: Packets are sent to the kernel debugger. A packet consists of a fixed sized header followed by zero or more bytes of data. The kernel debugger returns data to the debug engine in packets.

Raw Mode

Any kernel debugger command may be sent in raw mode. Debug engines that communicate in packet mode may wish to use raw mode to issue a .B command to set the communication rate for the serial connection.

To enter raw mode from packet mode, or to get the kernel debugger's attention while the system is running and enter raw mode, the debug engine should send a break character (^C or 0x03) and wait for the kernel debugger to issue a prompt.

Packet Mode

To enter packet mode from raw mode, or to get the kernel debugger's attention while the system is running and enter packet mode, the debug engine should send the KDP break character (0x1f). If the system was running, the kernel debugger will respond with an event packet containing a CVK_RET_ASYNC event. If the system was quiesced, the kernel debugger will not respond at all.

Packet Format

The format of a packet is as follows:

┌──────┬───────────────────────┬─────────...────────────┬──────┐
│ 0x1d │ 10-byte packet header │ packet body (optional) │ 0x1e │
└──────┴───────────────────────┴─────────...────────────┴──────┘

The packet header and the packet body, if present, are bitstuffed. The data is treated as a stream of bits and is broken into seven-bit chunks. Each chunk is put into the seven low-order bits of a byte and the high order bit of the byte is set. The bitstuffed data is padded at the end with zero bits to the next byte boundary. For example, the header 00009540 0000ac57 when bitstuffed becomes 808092d4 808081ac abc0.

Packet Header Format

The header, prior to bitstuffing, contains a 4-byte logical ID field, a 2-byte length field, and a 2-byte checksum. The checksum of n bytes of data is computed as follows:

  unsigned char  data[ ];
  unsigned short checksum = 0xa1e8;
  for (i = 0; i < n; i++)
  {
    checksum += data[i];
    checksum  = (checksum << 3) + (checksum >> 13);
  }

Packet Data Format

All multibyte items are presumed to appear in little-endian order. Thus, the checksum computed for the header 00009540 0000 is 0x57ac; when stored in the header the low order byte (0xac) appears first.

The header length field contains the number of bytes of data in the packet body before the packet body is bitstuffed. If the header length field is zero, there is no packet body. If a packet body is present, it includes a 2-byte checksum that is not accounted for in the header length field. For example, if the header length field is 0x12, there are actually 20 bytes in the unbitstuffed packet body.

The logical ID field takes one of the following forms (sequence numbers are one byte long):

CVK_HDR_DATA - 0x8000                        |
               0x4000 if "flast" flag is set |
               ((index number & 0x3f) << 8)  |
               sequence number

CVK_HDR_ACK - (0x4000 | ((index number & 0x3f) << 8) | sequence number) << 16

CVK_HDR_NACK- (0xc000 | ((index number & 0x3f) << 8) | sequence number) << 16

Maximum Packet Size

The maximum packet size for a bitstuffed packet is defined by CVK_PACKET_MAXSIZE as 623. This is derived by:

  Start byte      1
  Header         10
  Data          611
  End Byte        1

General Considerations

The debug engine initiates all transactions with the kernel debugger, normally by sending a command while the kernel debugger is waiting to receive one. In this case, the kernel debugger expects to receive a CVK_HDR_DATA packet and will discard any CVK_HDR_ACK or CVK_HDR_NACK packets it receives. (The kernel debugger also discards a packet if its header cannot be unbitstuffed or has a bad checksum. The kernel issues a CVK_HDR_NACK packet containing the sequence number and index number from the original CVK_HDR_DATA packet if the packet's body cannot be unbitstuffed or has a bad checksum.)

Once the kernel receives a valid CVK_HDR_DATA packet, it extracts the sequence number and index number and uses them to construct a CVK_HDR_ACK packet, which it returns to the debug engine. (The flast flag in the CVK_HDR_DATA packet is ignored.) The kernel debugger then performs the action requested by the command and returns the result.

In general, the result is returned in a single CVK_HDR_DATA packet whose sequence number matches the sequence number contained in the debug engine's original command packet, whose index number is zero, and whose flast flag is FALSE. After the result is transmitted, the kernel debugger waits for a response from the debug engine. The kernel is expecting either a CVK_HDR_ACK packet whose sequence number and index number match those sent in the result _or_ a CVK_HDR_DATA packet (containing the next command). If the kernel debugger receives any other response, it resends the CVK_HDR_DATA packet containing the result.

If the debug engine sends a KDP break character while the victim machine is running, either to initiate a transaction or to regain control after the victim machine has resumed execution, the kernel debugger responds with a CVK_HDR_DATA packet whose sequence number matches the sequence number from the CVK_HDR_DATA packet that caused the system to resume. There is no such packet when the kernel debugger responds to the first KDP break sent by the debug engine. The sequence number in that case contains garbage.

The kernel debugger does not generate replies for some commands, such as reboot, and the replies to commands that cause the victim machine to resume execution, such as resume or step, are not sent until an event such as a breakpoint, module load, or break signal from the debug engine, has caused the victim machine to quiesce.

The kernel debugger responds somewhat differently to the CVK_CMD_RAW command, which is used to issue arbitrary kernel debugger commands while in packet mode. Each line in the response is returned in a separate CVK_HDR_DATA packet whose sequence number matches the sequence number in the CVK_CMD_RAW command's header. The index number in the first reply packet is 0; the index number increases by 1 in each successive reply packet (and wraps from 63 to 0). The debug engine should return a CVK_HDR_ACK packet with the appropriate sequence number and index number after each reply packet is received.

The kernel debugger does not manipulate or increment sequence numbers and uses them only to generate ACKs and NACKs and to match ACKs with replies. A debug engine could use the same sequence number for every request, but this is not recommended.

Kernel Debugger Packet Responses

Event	Code	Description
CVK_RET_SUC	0x0000	Success
CVK_BAD_COMMAND	0x0002	Unrecognized command
CVK_RET_PAGEIN	0xffef	Discarded page reloaded
CVK_RET_TEND	0xfff0	Task died
CVK_RET_TNEW	0xfff1	Task created
CVK_RET_ASYNC	0xfff5	Asynchronous halt (break)
CVK_RET_LIB	0xfff8	Module loaded
CVK_RET_GPF	0xfff9	General protection fault
CVK_RET_KIL	0xfffa	Module unloaded
CVK_RET_NMI	0xfffb	Non-maskable interrupt
CVK_RET_BPT	0xfffc	Software breakpoint (INT3)
CVK_RET_TBT	0xfffd	Single step
CVK_RET_ERR	0xffff	Failure

Events Reported by the Kernel Debugger

The following is a summary of the data reported when the kernel debugger sends an event in packet mode:

CVK_RET_GPF Events

Event	Code	Description
KDP_T_DIVIDE	0	Divide by zero exception
KDP_T_INTO	4	Overflow Interrupt (INTO instruction)
KDP_T_BOUND	5	Bounds check (BOUND instruction)
KDP_T_INVALID_OPCODE	6	Invalid operation
KDP_T_EXTENSION	7	Coprocessor not available
KDP_T_DOUBLE_EXCEPTION	8	Double exception
KDP_T_EXTENSION_SEG_OVERRUN	9	Coprocessor segment overrun
KDP_T_INVALID_TSS	10	Invalid TSS
KDP_T_SEG_NOT_PRESENT	11	Segment not present
KDP_T_STACK_SEG	12	Stack exception
KDP_T_GP_FAULT	13	General protection fault
KDP_T_PAGE_FAULT	14	Page fault

Fields returned in cvkcmd_s are as follows:

Cmd CVK_RET_GPF
Value Event code from the table above.
OffV Linear address in CS:(E)IP at time of event.
SegV Slot number of thread.
MTE MTE entry of executable running in process.
PID PID of process that generated the event.
TID TID of thread that generated the event.
DBit Flags from CS selector.
Reg Registers at time of event.
MemCache Not used.

CVK_RET_TBT Events

A CVK_RET_TBT event is generated when a single step occurs or when a debug trap, such as an event triggered by one of the hardware debug registers, occurs.

Event	Code	Description
KDP_T_SSTEP	1	Single step or debug trap

Fields returned in cvkcmd_s are as follows:

Cmd CVK_RET_TBT
Value Not used.
OffV Linear address in CS:(E)IP at time of event.
SegV Slot number of thread.
MTE MTE entry of executable running in process.
PID PID of process that generated the event.
TID TID of thread that generated the event.
DBit Flags from CS selector.
Reg Registers at time of event.
MemCache Not used.

CVK_RET_BPT Events

Event	Code	Description
KDP_T_BREAKPOINT	3	Software breakpoint

Fields returned in cvkcmd_s are as follows:

Cmd CVK_RET_BPT
Value Not used.
OffV Linear address in CS:(E)IP at time of event.
SegV Slot number of thread.
MTE MTE entry of executable running in process.
PID PID of process that generated the event.
TID TID of thread that generated the event.
DBit Flags from CS selector.
Reg Registers at time of event.
MemCache Not used.

CVK_RET_NMI Events

Event	Code	Description
KDP_T_NMI	2	Non-maskable interrupt

Fields returned in cvkcmd_s are as follows:

Cmd CVK_RET_NMI
Value Not used.
OffV Linear address in CS:(E)IP at time of event.
SegV Slot number of thread.
MTE MTE entry of executable running in process.
PID PID of process that generated the event.
TID TID of thread that generated the event.
DBit Flags from CS selector.
Reg Registers at time of event.
MemCache Not used.

CVK_RET_SUC Events

Event	Code	Description
KDP_T_SUCCESS	100	When last CVK_CMD_RAW response is sent.

Cmd CVK_RET_SUC

CVK_RET_ASYNC Events

Event	Code	Description
KDP_T_ASYNC_TRAP	101	KDP break received.

Fields returned in cvkcmd_s are as follows:

Cmd CVK_RET_ASYNC
Value Not used.
OffV Linear address in CS:(E)IP at time of event.
SegV Slot number of thread.
MTE MTE entry of executable running in process.
PID PID of process that generated the event.
TID TID of thread that generated the event.
DBit Flags from CS selector.
Reg Registers at time of event.
MemCache Not used.

CVK_RET_LIB and CVK_RET_KIL Events

Event	Code	Description
KDP_T_LINK	102	Module loaded
KDP_T_UNLINK	103	Module unloaded

Fields returned in cvkcmd_s are as follows:

Cmd CVK_RET_LIB or CVK_RET_KIL
Value MTE handle of module in question.
OffV Not used.
SegV Slot number of thread.
MTE MTE entry of executable running in process.
PID PID of process that generated the event.
TID TID of thread that generated the event.
DBit Not used.
UCHAR NAME[ ] Null-terminated full pathname of module in question (immediately follows DBit and overlays Reg and MemCache)

CVK_RET_TNEW and CVK_RET_TEND Events

Event	Code	Description
KDP_T_TASK_CREATE	104	Task create
KDP_T_TASK_END	105	Task died

Fields returned in cvkcmd_s are as follows:

Cmd CVK_RET_TNEW or CVK_RET_TEND
Value Not used.
OffV Not used.
SegV Not used.
MTE Not used.
PID PID of process that was created or died.
TID 1 (Primary TID in process)
DBit Not used.
Reg Not used.
MemCache Not used.

CVK_RET_PAGEIN Events

Event	Code	Description
KDP_T_PAGEIN	106	Discarded page reloaded

Fields returned in cvkcmd_s are as follows:

Cmd CVK_RET_PAGEIN
Value Not used.
OffV Address of the page in question.
SegV Not used.
MTE MTE handle of the module that contains the page in question.
PID Not used.
TID Not used.
DBit Not used.
Reg Not used.
MemCache Not used.

Kernel Debugger Packet Commands

The following table lists the kernel debugger packet commands:

Command	Code	Description	CVK_CMDSIZE	CVK_RETSIZE
CVK_CMD_RMEM	1	Read memory.	18	20
CVK_CMD_RREG	3	Read registers.	18	24
CVK_CMD_WMEM	4	Write memory.	20	6
CVK_CMD_WREG	6	Write registers.	20 + sizeof(RegSa_struc)	2
CVK_CMD_RUN	7	Resume execution.	6	0
CVK_CMD_KILL	8	Reboot victim machine.	2	0
CVK_CMD_STEP	9	Single step.	2	0
CVK_CMD_NUMTOBASE	13	Get object/segment information.	14	14
CVK_CMD_LIBNAME	16	Get module information.	6	6
CVK_CMD_RAW	20	Perform kernel debugger command.	6
CVK_CMD_DBIT	22	Get selector information.	20
CVK_CMD_RSTEP	23	Range step.	10	0
CVK_CMD_SCANMTE	24	Scan module table.	2	6
CVK_CMD_SCANTCB	25	Scan thread control blocks.	6	10
CVK_CMD_SEL2LIN	26	Convert selector:offset to linear address.	18	6
CVK_CMD_LIN2SEL	27	Convert linear address to selector:offset.	18	12
CVK_CMD_OBJCOUNT	28	Get number of objects/segments in module.	6	6
CVK_CMD_SCANOBJ	29	Scan object/segment table.	14	10
CVK_CMD_SELINFO	30	Get selector information.	18	20
CVK_CMD_RNPX	31	Read NPX state.	18	128
CVK_CMD_WNPX	32	Write NPX state.	128	60
CVK_CMD_ENA	33	Enable optional features.	6	2
#CVK_CMD_DIS	34	Disable optional features.	6	2
CVK_CMD_PIREG	35	Register for PAGEIN notification.	14	2
CVK_CMD_PIDRG	36	Deregister for PAGEIN notification.	14	2

Each command is described below, along with the input parameters and the output values. Parameters are usually passed in a cvkcmd_s If a given field in the cvkcmd_s structure is not listed, its value is not used.

There are constants called CVK_CMDSIZE_xxx and CVK_RETSIZE_xxx which provide the size of the command and the size of the response from each command, where xxx is substituted with the last part of the command name. For CVK_CMD_PIDRG, the constants would be called CVK_CMDSIZE_PIDRG and CVK_RETSIZE_PIDRG. The values for these constants are provided in the last two rows of the table above.

CVK_CMD_RMEM

Read Memory.

Parameters

Cmd CVK_CMD_RMEM
Value Number of bytes to read. (<= CVK_MEMCACHE_SIZE)
OffV Offset of data to read.
SegV Selector or segment index for data to read. 0 if OffV is a linear address, otherwise setting of PE in CR0 and VM in EFLAGS of interrupted thread determine whether SegV is selector or selector index.
TID Slot of thread in whose context address is to be used. 0 is used for regions in globally shared areas.

Returns

Value Number of bytes actually read.
MemCache[Value] Data read.

CVK_CMD_RREG

Read Registers.

Parameters

Cmd CVK_CMD_RREG
TID Slot of thread in whose registers are desired.

Returns

Reg Registers for thread.
ULONG PC (immediately following Reg) Linear address corresponding to CS:(E)IP.

CVK_CMD_WMEM

Write Memory.

Parameters

Cmd CVK_CMD_WMEM
Value Number of bytes to write. (<= CVK_MEMCACHE_SIZE)
OffV Offset of data to write.
SegV Selector or segment index for data to write. 0 if OffV is a linear address, otherwise setting of PE in CR0 and VM in EFLAGS of interrupted thread determine whether SegV is selector or selector index.
TID Slot of thread in whose context address is to be used. 0 is used for regions in globally shared areas.
MemCache[Value] Data to write.

Returns

Value Number of bytes actually written.

CVK_CMD_WREG

Write Registers.

Parameters

Cmd CVK_CMD_WREG
TID Slot of thread in whose registers are to be changed.
Reg Registers for thread. Only the following registers are affected: EAX, EBX, ECX, EDX, EDI, ESI, EBP, ESP, EIP, CS, DS, ES, FS, GS, SS, and EFLAGS.

Returns

Cmd Return code

CVK_CMD_RUN

Resume Execution.

Parameters

Cmd CVK_CMD_RUN
Value Indicates action kernel debugger should take after resuming execution.
-CVK_CMD_RUN:Kernel debugger is not to expect further commands or generate further events. All optional features enabled with the CVK_CMD_ENA command are disabled. In other words, this command detaches the debug engine from the kernel debugger.

Anything else:Kernel debugger sends notification when an event occurs.

Returns

None. See description above in Value for action taken by system under debug.

CVK_CMD_STEP

Single step.

Parameters

None.

Returns

None. System under debug sends event notification when an event occurs.

CVK_CMD_NUMTOBASE

Get object / Segment Information.

Parameters

Cmd CVK_CMD_NUMTOBASE
Value Object/segment index for segment about which information is desired.
MTE MTE handle for module about which information is desired.

Returns

Value Flags from object/segment table entry for segment. (ote_flags or ste_flags)
OffV Base address for object. (0 for segments.)
SegV Selector for object/segment.

CVK_CMD_LIBNAME

Get Module Information.

Parameters

Cmd CVK_CMD_LIBNAME
Value MTE handle for module about which information is desired.

Returns

Value Length of full pathname of module, including trailing null character. Returns 0 if SMTE or name is not resident.
UCHAR NAME[Value] Null-terminated full pathname of module (immediately follows Value)

CVK_CMD_RAW

Perform Kernel Debugger Command.

Parameters

Cmd CVK_CMD_RAW
OffV Null-terminated text of command to perform (overlays OffV)

Returns

Cmd Indicates whether this is the last response.

CVK_CMD_RAW More response packets coming.

CVK_RET_SUC Last response packet.

Value Length of command response.
OffV Null-terminated response to command (overlays OffV)

Note: This command may return multiple responses. The last event packet response will have Cmd set to CVK_RET_SUC instead of CVK_CMD_RAW.

CVK_CMD_DBIT

Get Selector Information.

Parameters: Cmd CVK_CMD_DBIT; SegV Selector for which information is desired.
Returns: DBit Flags from selector's descriptor.

CVK_CMD_RSTEP

Range Step.

Parameters

Cmd CVK_CMD_RSTEP
OffV Offset at which to stop.

Returns

None. System under debug steps until CS selector changes or until (E)IP becomes >= OffV value, then issues a single step event.

CVK_CMD_SCANMTE

Scan Module Table.

Parameters

Cmd CVK_CMD_SCANMTE

Returns

Cmd CVK_RET_LIB if information was returned, otherwise returns CVK_RET_SUC if end of module table reached.
Value MTE handle of module if Cmd = CVK_RET_LIB.
UCHAR NAME[Value] Null-terminated full pathname of module (immediately follows Value)

Not present if SMTE or name is not paged in. Packet length is CVK_RETSIZE_SCANMTE in this case.
The module table scan ignores resource modules, which contain only resource objects and segments, and forwarder modules, which contain no objects or segments.

CVK_CMD_SCANTCB

Scan thread control blocks.

Parameters

Cmd CVK_CMD_SCANTCB
Value Slot number of thread control block to start with. Specify 0 to start at first occupied slot. Specify -1 to start at currently scheduled slot.

Returns

Value Slot number of next occupied thread control block. 0 if no more TCBs.
OffV Number of information blocks returned in current response.

struct {
  USHORT slot;   /* Slot index for thread */
  USHORT PID;    /* PID that contains thread */
  USHORT TID;    /* Thread ordinal */
  USHORT MTE;    /* MTE handle for module running in process
                    that contains thread */
  USHORT flags;  /* Flags for thread
                    Bit 0 (low-order bit)  If ON, indicates
                         thread was running when system stopped.
                    Bits 1-15 are unused  */
  }  TIBInfo

Information returned immediately following OffV.

CVK_CMD_SEL2LIN

Convert selector:offset to linear address.

Parameters

Cmd CVK_CMD_SEL2LIN
OffV Offset
SegV Selector
TID Thread that linear address should be in context of.

Returns

Value Linear address

CVK_CMD_LIN2SEL

Convert linear address to selector:offset.

Parameters

Cmd CVK_CMD_LIN2SEL
Value Linear address
TID Slot of thread that desired linear address is to be in context of. (Currently ignored. Linear address must be in global region and mapped by interrupted thread's CS, DS, ES, or SS.)

Returns

OffV Offset
SegV Selector

CVK_CMD_OBJCOUNT

Get number of objects/segments in module.

Parameters

Cmd CVK_CMD_OBJCOUNT
Value MTE handle for module about which information is desired.

Returns

Value Number of objects/segments in module. 0 returned if the module is a forwarder module, which contains no objects or segments, or a resource module, which contains only resource objects and segments.

CVK_CMD_SCANOBJ

Scan object/segment table

Parameters

Cmd CVK_CMD_SCANOBJ
Value Object/segment table tindex of entry to start with. Specify 0 to start at the beginning.
MTE MTE handle for module about which information is desired.

Returns

Value Object/segment table index of next entry. 0 is returned if no more entries.
OffV Number of information blocks returned in current response.

 struct {
    ULONG  base;   /* Base address of object/segment */
    ULONG  size;   /* Size in bytes of object/segment in memory */
    ULONG  flags;  /* Flags for object/segment:
                      bit 0 (low-order bit)
                         ON  - 16-bit segment
                         OFF - 32-bit segment
                      bit 1
                         ON  - data
                         OFF - code
                      bits 2-31 are unused */
 } ObjInfo;

Information block returned immediately follows OffV

CVK_CMD_SELINFO

Get selector information.

Parameters

Cmd CVK_CMD_SELINFO
SegV Selector
TID Slot of thread desired selector is to be in context of.

Returns

Value Limit of segment in bytes.
OffV Base address of segment.
DBit Flags from descriptor.

CVK_CMD_RNPX

Read NPX state.

Parameters

Cmd CVK_CMD_RNPX
TID Slot of thread whose NPX state is desired.

Returns

Cmd Return code.

0 NPX state read and returned

1 NPX is being emulated.

2 Invalid slot.

3 Thread has no NPX state.

4 State swapped out.

Reg Image of NPX state as saved by FSAVE instruction.

CVK_CMD_WNPX

Write NPX state

Parameters

Cmd CVK_CMD_WNPX
TID Slot of thread whose NPX state is to be set.
Reg Image of NPX state (restored by FRESTOR instruction)

Returns

Cmd Return code.

0 NPX state written successfully.

1 NPX is being emulated.

2 Invalid slot.

3 Thread has no NPX state.

4 State swapped out.

CVK_CMD_ENA

Enable optional features.

Parameters

Cmd CVK_CMD_ENA
Value Index of feature to enable.

0 CVK_RET_TNEW and CVK_RET_TEND notifications.

Returns

Cmd Return code.

Note: All optional features are disabled when a CVK_CMD_RUN command is issued with a Value of -CVK_CMD_RUN.

CVK_CMD_DIS

Disable optional features.

Parameters

Cmd CVK_CMD_DIS
Value Index of feature to disable.

0 CVK_RET_TNEW and CVK_RET_TEND notifications.

Returns

Cmd Return code.

Note: All optional features are disabled when a CVK_CMD_RUN command is issued with a Value of -CVK_CMD_RUN.

CVK_CMD_PIREG

Register for PAGEIN notification.

Parameters

Cmd CVK_CMD_PIREG
OffV Address of page.
MTE MTE handle of module that contains page.

Returns

Cmd Return code.

CVK_CMD_PIDRG

Deregister for PAGEIN notification.

Parameters

Cmd CVK_CMD_PIDRG
OffV Address of page.
MTE MTE handle of module that contains page.

Returns

Cmd Return code.