View source for DDTT: A New API Test Tool

''by [[Vince Rapp]]''
__TOC__

Got a new API set that you want to test, but don't want to spend your valuable test time writing a script parser? We have the answer for you! The Device Driver Test Tool (DDTT) provides an efficient environment to create, execute, and refine device driver test cases. But, don't let the device driver part scare you off. DDTT can and is being used for testing API sets that are implemented in DLLs, as well as device drivers.

DDTT is an extensible Functional Verification Test (FVT) tool originally developed to test OS/2 Base and Multimedia device drivers. Adding new API-dependent DLLs and Test Grammar files lets you quickly develop interfaces to new APIs. DLL files contain the API-specific interface (or "stub functions"). The Test Grammar files define Function and Parameter keywords that you can use in test case script files. Additionally, the Test Grammar files provide DDTT with run-time symbol resolution, so DDTT can call the stub functions referenced by the keywords in the test case script files.
[[Image:DDTT-Fig-1.gif|thumb|300px|''Figure 1. Overall DDTT Data Flow'']]

The test case script files contain actual test cases. Other API-level test tools have shown the advantages of placing test case content into script files, such as:
* Ease of test case creation and evolution
* Test case repeatability
DDTT lets the test case designer focus on the APIs under test by providing a basic grammar and parsing capability that does not have to be recoded and debugged with each new API tested.

DDTT was written in C++ using Borland's C++ for OS/2; but is being ported to IBM's C Set++. To date, we've used DDTT primarily to test procedural APIs; however the flexibility provided by the stub functions lets you also test object APIs.

==Using DDTT==
The first step in using DDTT to define the test functions and their parameters in a DDTT Grammar File. Sample Code1 defines a simple DDTT grammar for the imaginary CARINFO API.
<PRE>
*
*    DDTT Grammar file defined for the CARINFO API
*    (First character in a line = "*" then the line is a comment)
*    All DDTT Stub functions are defined in the DLL: CARINFO.DLL
*    Include the DDTT global function definitions:
*
@IMPORT GLOBAL.GRA
*
* Opens a new car.
OPEN_CAR           $DLL=CARINFO                   \
                   COLOR=STRING                   \
                   WEIGHT=NUM                     \
                   YEAR=NUM                       \
                   $EXPECTED                      \
                    CAR_HANDLE=NUM                \
                    APIRET_STATUS=NUM
*
* Change at least the color of an existing car.
CHANGE_CAR         $DLL=CARINFO                   \
                   $FUNC=_AlterCar                \
                   COLOR=STRING                   \
                   CAR_HANDLE=NUM                 \
                   $EXPECTED                      \
                    APIRET_STATUS=NUM
*
* Get a car and log its characteristics.
SHOW_CAR           $DLL=CARINFO                   \
                   $FUNC=_GetCar                  \
                   CAR_HANDLE=NUM                 \
                   $EXPECTED                      \
                     COLOR=STRING                 \
                     WEIGHT=NUM                   \
                     YEAR=NUM                     \
                     APIRET_STATUS=NUM
*
* Delete a car.
CLOSE_CAR          $DLL=CARINFO                   \
                   $FUNC=_DeleteCar               \
                   CAR_HANDLE=NUM                 \
                   $EXPECTED                      \
                   APIRET_STATUS=NUM

</PRE>
''Sample Code 1. DDTT Test Grammar File for CAR_API.GRA''

The DDTT Grammar File defines a set of Function Specifications for the API under test. In the simplest case, a one-to-one relationship exists between the API entry point under test and the DDTT stub function. This is the case for the CARINFO API; however, it is not a requirement. For example, if you always call more than one API as a group, then you can put these APIs in a single stub function. Likewise, if a single API is invoked under drastically different circumstances, then create multiple stub functions, one stub function for each circumstance. You can use different Function keywords for each stub function.

The first token in each DDTT function specification is the Function Keyword. The function keyword that is used in the test script files to call this function. All remaining tokens on the left side of the equal (=) sign within the function specification are Parameter Keywords. The right side of the equal sign contains the DDTT type identifier that indicates the type of the Parameter Keyword. Possible type identifiers are: STRING, NUM, ALNUM, ALPHA. The $DLL and $FUNC Parameter Keywords tell DDTT which DLL and which stub function to call when the Function Keyword is found in a test case script. All other Parameter Keywords occurring before the $EXPECTED token must be defined and be the specified type before DDTT will call this function at run-time.

==Built-In Scripting Functions==
Actual DDTT test cases are created in test script files. The default filename extension for these files is .SCR. DDTT provides several built-in control functions to coordinate test case execution. You can access these functions by importing GLOABL.GRA into your specific grammar file. Functions currently available are:
 @THREAD logfile
Starts a new thread and identifes the logfile to use for executing all DDTT commands up to the next @THREAD statement or the end of the test script file.
 @NEWALIAS alias grammar_file
Creates a new DDTT alias and establishes the device-specific grammar to use when parsing all other commands with the same alias name in this test script.
 @IMPORT import_filename
Inserts the content of the specified import_filename into this test case file.
 @LOOP loop_cnt ! [alias, numeric_param_keyword]
Executes the following block of one or more scripted functions up to the next @ENDLOOP token in a loop. Nested loops are permitted. The number of loop iterations is controlled in one of two methods. A single loop count positive numeric value can be specified. The other loop count syntax allows specification of a DDTT parameter keyword as the loop count.
 @LOG "string"
Logs the only parameter to the current thread's logfile specified with the previous @THREAD command. The log entry is time-stamped. If the string contains script variables, the script variables will be expanded before the text is placed in the log file. The log file is locked to prevent other DDTT threads from interspersing output. Output is prefaced with the current time stamp.
 @PAUSE seconds
Stops execution of the current thread for the specified number of seconds. Execution is automatically resumed.
 @IF $COND=num_param_keyword
Executes the next DDTT script statement if the numeric parameter keyword is defined and non-zero.
 alias SET param_name=value ...
Initializes, or reinitializes, the current value of DDTT parameter keyword to the string value. Multiple keyword parameters can be set in a one DDTT SET command. All keyword parameters are defined only in the specified DDTT alias.
 alias RESPONSE $PROMPT="message text" $PAUSE=seconds $RESPONSE=param_keyword
Prompts the user for input and returns the result in the DDTT parameter keyword specified in $RESPONSE. $PAUSE specifies the maximum time DDTT will wait for the user to begin entering input.
 alias DUMP $BUFFER=ddtt_buffer
Locks the output log file of the current alias, formats the ddtt_buffer in hexadecimal, and sends the data to the log file. Unlocks the log file.
 alias PAUSE $PAUSE=seconds
Pauses the alias' thread for $PAUSE seconds.
 alias OUT $KWD=param_keyword
Looks up the value of the parameter keyword specified in $KWD, and logs it as an ASCII string to the log file of the alias. The log file is locked.
 alias CLEAR $BUFFER=ddtt_buffer
Sets every byte in the DDTT data buffer, ddtt_buffer, to null.
 alias COMPARE $BUFFER1=ddtt_buffer $BUFFER2=ddtt_buffer
Compares the content of the two DDTT data buffers and logs the result, SUCCESS or ERROR, to the log file of the current thread.
 %Environment_Var%
Inserts the OS/2 Environment variable translation of the specified Environment_Var string.
 $(param_keyword)
Inserts the current value of the specified DDTT Parameter Keyword. 

==Automatic Error Checking==
DDTT grammar files can use the built-in $EXPECTED token to specify expected results of Parameter Keywords. The test script writer must specify expected values for the keyword parameters listed after the $EXPECTED token. To specify an expected value parameter keyword, the test case writer prefaces the keyword parameter with EV_. If the test case cannot predict the exact value and if the test script writer prefaced the parameter keyword with ET_, you can instruct DDTT to perform type checking on the resulting parameter keyword. DDTT automatically performs the appropriate error checking after you call the function. If expected parameter keywords are not specified in the DDTT script file or set by a previous stub function, then DDTT logs this fact as a warning message to the log file and continues to call the function under test.

When creating a DDTT test case, the test case designer calls the functions defined in the grammar using two DDTT defined objects:
* DDTT Thread
* DDTT Alias
Consider the DDTT test script in Sample Code 2, which you can use to test the NEWCAR API.
<PRE>
    @THREAD thread1.log    @NEWALIAS  my_car CAR_API.GRA
    my_car LOG $PROMPT="Starting testcase, thread 1..."
    my_car OPEN_CAR COLOR=GREEN                   \
                    WEIGHT=6700                   \
                    YEAR=1975                     \
                    EV_APIRET_STATUS=0
    my_car SHOW_CAR EV_COLOR=GREEN                \
                    EV_WEIGHT=6700                \
                    EV_YEAR=1975
    my_car CHANGE_CAR COLOR=LT_GREEN
    my_car SHOW_CAR  EV_COLOR=LT_GREEN
    my_car CLOSE_CAR
*
    @THREAD thread2.log
    @NEWALIAS  my_other_car CAR_API.GRA
    my_other_car LOG $PROMPT="Starting testcase, thread 2..."
    my_other_car SET COLOR=RED
    my_other_car OPEN_CAR WEIGHT=3400             \
                          YEAR=1992               \
                          COLOR=LT_RED            \
                          EV_APIRET_STATUS=0
    my_other_car SHOW_CAR EV_COLOR=LT_RED         \
                          EV_YEAR=1992            \
                          EV_WEIGHT=3400
    my_other_car CLOSE_CAR
    @NEWALIAS yet_another_car CAR_API.GRA
    yet_another_car SET YEAR=1958 WEIGHT=5300     \
                    EV_APIRET_STATUS=0
    yet_another_car OPEN_CAR COLOR="BRILANT BLUE"
    yet_another_car SHOW_CAR EV_COLOR="BRILANT BLUE"   \
                             EV_YEAR=1958              \
                             EV_WEIGHT=5300
    yet_another_car CLOSE_CAR
</PRE>
''Sample Code 2. Sample DDTT test case script for NEWCAR.SCR''

A DDTT test case script can define multiple threads of execution. NEWCAR.SCR creates two test threads. Inside test scripts, the token @THREAD identifies the start of a new thread. The @THREAD command's only parameter is the name of an output file (THREAD1.LOG and THREAD2.LOG) to receive all logs and error messages this thread generates.

DDTT parses the test case script and locates all @THREAD tokens. Test threads are created and DLLs are loaded. Finally, all test threads are started simultaneously.

Inside each DDTT thread, you can define multiple DDTT aliasnames. The first thread defines the alias, my_car; the second thread defines two aliasnames, my_other_car and yet_another_car. The aliasname, or alias, is created with the @NEWALIAS command. All test functions execute within one alias object. Upon instantiation, an alias object knows what test functions, required parameters, and the type of error checking is to be performed. DDTT acquires this information from the test grammar filename specified in the @NEWALIAS command.

The alias object also defines the scope of all DDTT parameter keywords defined within the alias. DDTT stub functions always are called with a single parameter, which is a reference to a keyword list (or collection) of all currently defined DDTT parameter keywords.

To round out the picture, Sample Code 3 contains a sample of what the DDTT function stub might look like that implements the OPEN_CAR and SHOW_CAR functions.
<PRE>
//****************************************************************
// Name: APIRET _export CreateCar( Kwd_List &param )
//
// Description: DDTT Stub Function for OPEN_CAR. The CreateCar
//              API is supposed to allocate storage for the new
//              car defined by the input parameters and return
//              its handle. This Stub stores the handle in the
//              DDTT keyword, CAR_HANDLE.
//
// Required Parameters:  COLOR=STRING, Color of the car
//                       WEIGHT=NUM, Weight of the car in pounds
//                       YEAR=NUM, Year the car was manufactured
//
// Returns:     APIRET_STATUS=NUM, 0 if everything worked OK.
//*****************************************************************
APIRET _export CreateCar( Kwd_List &param )
{
  APIRET apiret;
  CAR_HANDLE cHandle;

  // Note: this function is not called unless the required Parameter
  // Keywords are defined.  Thus no checking is required here.
  ISting color = param("COLOR");          // Get the keyword as an IString
  LONG weight = param.getInt("WEIGHT");   // Retrieve Weight and year as
  LONG year = param.getInt("YEAR");       // Integers

  apiret = NEW_API_CREATE_CAR( &cHandle
                               (char *)color,
                               weight,
                               year );
  // Save the API status,  DDTT will anunciate if the value does
  // not match the content of EV_APIRET_STATUS.
  param.set( "APIRET_STATUS", (IString)apiret );

  if( apiret == 0 )  // If it worked, save the car handle
      param.set( "CAR_HANDLE", (IString)(LONG)*cHandle );
  else               // Otherwise delete any existing car handle
      param.unset( "CAR_HANDLE" );

  // Always return 0 from the stub functions.  Any non-zero value
  // signals a DDTT internal processing error, an error message is
  // logged and no additional calls are executed.
  return 0;
}
//****************************************************************
// Name: APIRET _export GetCar( Kwd_List &param )
//
// Description: DDTT Stub Function for SHOW_CAR.  The GetCar API
//              is supposed to return copies of the characteristic
//              data associated with the input CAR_HANDLE.  This stub
//              dumps the returned information to the log and refreshes
//              the associated DDTT keywords.
//
// Required Parameters:  CAR_HANDLE=NUM, handle to car of interest
//
//
// Returns:   COLOR=STRING, Color of the car
//            WEIGHT=NUM, Weight of the car in pounds
//            YEAR=NUM, Year the car was manfactured
//            APIRET_STATUS=NUM, 0 if everything worked OK.
//*****************************************************************
APIRET _export GetCar( Kwd_List &param )
{
  APIRET apiret;
  CAR_HANDLE cHandle = param.getInt("CAR_HANDLE");
  char color(50);
  LONG weight;
  LONG year;

  apiret = NEW_API_GET_CAR( cHandle, color, &weight, &year );

  // Save the API status,  DDTT will annunciate if the value does
  // not match the content of EV_APIRET_STATUS.
  param.set( "APIRET_STATUS", (IString)apiret );

  // Save the output keywords, if there are errors, DDTT will
  // discover and log the errors.
  param.set( "COLOR", (IString)color );
  param.set( "WEIGHT", (IString)weight );
  param.set( "YEAR", (IString)year );

  // Can also log the results to this thread's log file
  {
     IString s1;
     s1 = "This car's handle = " + param("CAR_HANDLE");
     s1 += "\n\t COLOR is ";
     s1 += param("COLOR");
     s1 += "\n\t WEIGHT is ";
     s1 += param("WEIGHT");
     s1 += "\n\t Year is ";
     s1 += param("YEAR");
     param.files()->out1<<s1;
   }
}
</PRE>
''Sample Code 3. DDTT stub functions''

DDTT defines several C++ member functions for operating on alias objects. An example is the param.set member function that behaves the same as the DDTT global function SET. SET, or .set, associates a new value with the referenced parameter keyword. All parameter keywords are stored internally as IString objects. Below is list of the member functions available to operate on the KwdList (alias) objects passed into DDTT stub functions:
<PRE>
// Add or update a keyword-value pair on the list
set(IString kwd, IString value);
//Removes a keyword-value pair from the list
unset(IString kwd);
// Return the corresponding keyword's value as an IString
IString operator[](IString kwd);
// Return a sscanf'd int corresponding to the keyword
long int getInt(IString kwd);
// Return a sscanf'd unsigned corresponding to the keyword
unsigned long int getUnsigned(IString kwd);
// Return a sscanf'd pointer corresponding to the keyword
void * getPtr(IString kwd);
// Return true if the keyword is currently defined
BOOL isKeyword(IString kwd);
// Returns a pointer corresponding to the FILES keyword
FileInfo * files();
</PRE>

==Where to Find More==
To date, test grammars have been implemented for the following OS/2 Device Driver APIs:
* CD-ROM
* DASD
* SCSI
* Keyboard
* PCMCIA Socket Services
These test grammars, their associated test script files, and an extensible version of DDTT with full documentation and makefile is available on the current version of the Developer Connection Device Driver Kit for OS/2. It is also on the IBM InfoRom that was distributed during Fall COMDEX, as well as through the DDK DUDE support channel. Please see the directory of this newsletter for more information on the DUDE.

{{IBM-Reprint}}
[[Category:Developer Connection News]]