RMX-OS2: An In-Depth View - Part 2

From EDM2
Jump to: navigation, search
RMX-OS2: An In-Depth View
Part: 1 2 3 4

Written by Johan Wikman

Introduction

This is the second article in a series where I am describing RMX. RMX is a system for allowing OS/2 PM applications to run remotely, i.e., to run on one computer and be used an another. A somewhat more thorough overview of RMX is available in the previous issue of EDM/2.

One central feature of RMX is the replacement of the local procedure calls an application makes to PM with equivalent remote procedure calls over a network to a remote computer. In this article I will describe the low level transfer mechanism that is used for implementing the RPCs.

Remote Procedure Calls

Although most of you probably know what remote procedure calls are, I nevertheless provide here a brief introductory lesson. If RPCs are familiar you can simply skip this section.

Whenever we have two communicating processes, the data exchanged must somehow be transferred between them. Consider a situation where a client process asks a server process to add two integers and return the result back. Pseudo code for achieving that may look like:

  1. write both integers to a buffer
  2. send the buffer to server
  3. block while waiting for server to reply
  4. extract result from buffer

This works, but it would be tedious if the client process writer would have to write all that code (and its a lot more in reality when error checking, etc. is included) only to get two integers added. If this sequence is examined closer, it is pretty obvious that it resembles very much an ordinary function call. If we just wrap the pseudo code above in a function, a client process need not even be aware that the function is executed remotely.

Part2-rmx1.gif

The general procedure when a client calls an RPC is as follows: 1) the client calls a stub procedure, 2) the stub procedure marshals the arguments into suitable transfer format and sends them to the server, 3) the server unpacks the arguments and executes the actual function, 5) the server marshals the return value and sends it back to the client, 6) the client extracts the return value and returns it to the client process in an ordinary way.

So, my task in RMX was to replace every PM function with an RPC as described above. This was the-other-way-around compared with how RPCs usually are implemented. The usual procedure is to describe the RPCs in a special purpose language and then generate client and server stubs and C-prototypes from that description. But there was nothing I could do about that since PM API was already defined and had to provide an RPC implementation for it.

What is required for RPCs

In order to implement RPCs there are some fairly obvious primitives that are needed. A client and a server both need to send data to and read data from their peer. These primitives are not sufficient, however, as a connection - in whatever way it is implemented - between a client and a server does not come into being by itself. In practice we need to be able to:

  1. create a connection (server)
  2. place a connection into listening state (server)
  3. disconnect a connection (server)
  4. open a connection (client)
  5. close a connection (server/client)
  6. read data from a connection (server/client)
  7. write data to a connection (server/client)

As the client and server usually are running on different machines, the client also needs a way to find out the name of the server connection.

When I started working on RMX I had, of practical reasons, to implement the communications mechanism using named pipes. At the same time, however, I did not want to build the use of named pipes into RMX as it would have made it quite difficult to later start supporting TCP/IP or something else. So what I did, was to specify a communications API according to the needs of RMX and implement it using named pipes. As the communications DLL is dynamically loaded (based on the value of an environment variable) during start-up, it can easily be replaced by another DLL - e.g. implemented on top of TCP/IP - and no changes are needed in RMX. This is provided, of course, that the new DLL fulfils the requirements of the API exactly.

Recently I found the complete TCP/IP package on a Developers Connection CD I have, and after a few days of pondering (with the help of "Client/Server Programming with OS/2 2.1" by Orfali and Harkey - an excellent book) I managed to put together an TCP/IP based RMX communications DLL. So, my initial design turned out to work. <grin>

RMX Communications API

Ok, so here is the API. I only present the prototypes here and continue with the implementation on top of named pipes and TCP/IP in the next section.

RmxGetServiceName
This is used by the server to obtain the name to use when creating a connection and by the client to obtain the well-known entry point of the server.
RmxCreate
This is used by the server to create a connection. The name the server uses is always obtained from RmxGetServiceName(). The connection created must be blocking (calls to a empty connection must block the caller).
RmxCreateUnique
This is used by the server to create a new unique connection. The name the function uses for creating the connection is given back to the caller so that it - by some means - can pass it to a client.
RmxOpen
This is used by the client to open an existing connection.
RmxClose
This is used is by both the client and the server to close a connection. The server always first calls RmxDisConnect() and only then RmxClose().
RmxConnect
This is used by the server to place a connection into listening state. In practice it causes the server to block and wait for clients to connect.
RmxDisConnect
This is used by the server to acknowledge that the client has closed the connection.
RmxWrite
This is used by the server and the client to write data to the peer.
RmxRead
This is used by both the server and client to read data from the peer.

Error codes

As the idea was to be able to implement the RMX communications API on top of any real communications mechanism, I had to define a bunch of error codes that hopefully would suffice for all implementations. But in fact, as RMX treats most errors as fatal, a boolean would have been enough. For instance, if the connection to the display computer is broken, there is no other option but to terminate the application.

Error Code Description
RMXAPI_OK The function was successfully executed.
RMXERR_ENLARGE_BUFFER The provided buffer was too small. Enlarge and call again.
RMXERR_UNKNOWN_SERVICE The service requested is not known.
RMXERR_GENERAL An all purpose error code.
RMXERR_BAD_NAME The name is invalid or illegal.
RMXERR_BAD_NETWORK The network or the server is not running.
RMXERR_CONNECTION_NOT_CONNECTED The server has disconnected the connection.
RMXERR_CONNECTION_BUSY The connection is busy. Wait and retry.
RMXERR_INVALID_HANDLE The handle provided is no good.
RMXERR_OUT_OF_MEM Out of memory.
RMXERR_OUT_OF_RESOURCES Out of some critical resource.

How RMX uses the API

Until now, I've been talking solely about a client and a server although RMX in fact also consists of an engine. The purpose of RMX is to allow a PM application to run remotely. So, if we have a client application on computer A and want to use the application on computer B it means in practice that on B there must be a server that will execute the RPCs of the client application.

So far so good, but what if we want to run several application remotely? Should the same server provide services to all remote applications? It would work, although it would be a pain to keep all data in the server belonging to one remote application inaccessible from another remote applications.

Furthermore, would one of the remote applications cause the server to crash (it still does happen) then all remote applications would be affected. One solution is to have one server - with a well-known name - whose only task is to start a dedicated engine on behalf of a client's request. The engine creates a unique connection whose name is transferred over the server connection back to the client. The client closes the connection to the server and opens a connection to the engine.

Part2-rmx2.gif

Ok, so how is does the communication take place on function call level? An observant reader notices that RmxCreateUnique() is not mentioned at all, and that a function RmxGetUniqueName() seems to be used. The reason is that initially there was no RmxCreateUnique(), but the same functionality was obtained by first calling RmxGetUniqueName() and then RmxCreate(). While I was implementing the TCP/IP DLL I noticed that it is a lot simpler to combine the two. I just hadn't the time to update the picture.

Part2-rmx3.gif

The connection setup follows the following steps:

  1. When the server is started it calls RmxGetServiceName() to get the name it should listen to.
  2. The server creates the connection, using RmxCreate(), and issues RmxConnect() on the connection. This causes the server to block.
  3. When the client is started it calls RmxGetServiceName() to get the name of the server. It obtains the name of the host from the environment variable RMXDISPLAY (In the demo client application provided with this article, the hostname is given on the command line.)
  4. The client then opens a connection using RmxOpen(). This will cause the RmxConnect() function in the server to return.
  5. The server immediately issues a call to RmxRead(), which again will cause the server to block.
  6. The client sends the required connection setup info to the server using RmxWrite(). This will cause the RmxRead() call in the server to return.
  7. Immediately after the call to RmxWrite(), the client calls RmxRead() in order to get the reply from the server. This will cause the client to block.
  8. If the start-up parameters are acceptable, the server starts a new engine. The engine calls RmxGetUniqueName() to get a unique new name and uses RmxCreate() to create the connection.
  9. The engine passes the engine name back to the server which, using RmxWrite(), sends it back to the client. Meanwhile the engine calls RmxConnect() which will cause it to block.
  10. The client will now return from RmxRead(). It will immediately call RmxClose() to close the connection.
  11. The server calls RmxDisConnect() to close the connection instance.
  12. The client uses the value of the environment variable RMXDISPLAY and the returned engine name for opening a connection to the engine using RmxOpen().
  13. When the client calls RmxOpen() the engine will return from RmxConnect().
  14. The client enters a loop where it calls RmxWrite() to send requests to the engine and calls RmxRead() to receive replies. The engine will similarly enter a loop where it calls RmxRead() to get a request from a client, process the request and send a reply using RmxWrite().
  15. Although not shown in the figure, the client will eventually call RmxClose() to close the connection. The engine will detect this, exit its own loop, call RmxDisConnect() and RmxClose() and finally exit.

RMXPIPE.DLL

Ok, let's look at the named pipe implementation. All source is available in RMXCOMMS.ZIP delivered with this issue of EDM/2.

Types

One thing that has to be specified when named pipes are used is the size of the named pipe buffers. I chose to use buffers of 4096 bytes (one page). Bigger buffers would give better throughput when big chunks of data is being transferred - with the cost of more memory being used - but as in RMX mostly small chunks are being sent, 4K was just about right.

const ULONG SIZE_PAGE = 4096;
const ULONG SIZE_PIPE = SIZE_PAGE;

There is a need for a buffer outside the actual named pipes as well. As I wanted the entire data structure to reside within a (multiple if need be) page, I defined the buffer size using the pipe size.

     const ULONG SIZE_BUFFER = SIZE_PIPE - sizeof(HPIPE);

     struct CONNECTION
     {
	 HPIPE hPipe;
	 BYTE  bytes[SIZE_BUFFER];
     };

Now CONNECTION fits conveniently within a page. A user should not see this structure, however, as it will be different depending on the underlying communications mechanism. Therefore the only thing a user sees is a,

typedef void* HCONNECTION;

that can only be used as a handle.

ULONG RmxGetServiceName(PCSZ pcszService, ULONG* pulSize, PSZ pszName);

In the TCP/IP world there is standard way of finding server applications on remote hosts. In the directory /etc (\TCPIP\etc on OS/2) there is a file services that contains well-known entry points for different services. Unfortunately there is no similar convention with named pipes.

At first I had the RMX named pipe names hard coded in RMXPIPE.DLL but while writing this article I realised that hard coding the names effectively would mean that nobody else could use RMXPIPE.DLL directly but would have to modify it. Consequently I decided to implement a similar kind of functionality as the TCP/IP \etc\services.

As there is no obvious place in OS/2 to put a file like that I chose to require that it has to be in the same directory as RMXPIPE.DLL and that its name has to be RMXPIPE.DAT. Currently it looks like:

#$Header$
#
# This file associates official service names with a pipe name.
#
# The form for each entry is:
# <official service name>  <pipe name>
#
# RMXPIPE.DLL will automatically provide the initial required '\PIPE'
# of a name so it need not be explicitly provided.
#

rmxserver \rmx\server	   # RMX Server Process
rmxstarter	   \rmx\starter	  # RMX Starter Daemon
rmxtester \rmx\tester	   # RMX Test Server

The implementation of RmxGetServiceName() simply scans this file and returns the corresponding pipe name.

The server uses the returned name directly with RmxCreate(). The client cannot in general use the name directly with RmxOpen(), as the name uniquely identifies the service within the computer but not on the network.

The name of the host is required for uniquely identifying the service on the network. In RMX this is solved by requiring that the host name is provided in the environment variable RMXDISPLAY. If RMXDISPLAY is specified, RMX attempt to redirect the application to the remote host, if it is not, the application is run locally.

[C:\]set RMXDISPLAY=\\LOKE
[C:\]start app

The client application provided along this article takes the host as a command line argument.

ULONG RmxCreate(PCSZ pcszName, HCONNECTION* phConn);

RmxCreate() is used by a server process to create a connection. The only argument it takes is the name of the connection and a pointer to a variable where the connection handle will be returned. Most IPC mechanism can be configured in a million different ways - just look at the prototype for DosCreateNPipe() for instance - but as I only wanted to create connections of a specific kind there was no reason to have any additional arguments in the prototype of RmxCreate(). What RMX needs from a connection is:

  • blocking behaviour, reads on an empty connection blocks
  • write through, no buffering
  • read-write mode, possible to read and write using the same handle
  • unlimited instances, multiple instances with the same name can be created

Let's see how these requirements can be achieved with named pipes. The prototype of DosCreateNPipe() looks like:

    APIRET APIENTRY DosCreateNPipe(PCSZ	pszName,
					     PHPIPE pHpipe,
					     ULONG	openmode,
					     ULONG	pipemode,
					     ULONG	cbInbuf,
					     ULONG	cbOutbuf,
					     ULONG	msec);

The interesting arguments are openmode and pipemode. Openmode defines the mode in which the pipe is opened and pipemode defines the mode of the pipe. To get the behaviour I wanted I defined the open flags as:

const ULONG NP_OPEN_FLAGS = NP_NOWRITEBEHIND	 |
			  NP_NOINHERIT		 |
			  NP_ACCESS_DUPLEX;

and the pipe flags as:

const ULONG NP_PIPE_FLAGS = NP_WAIT		 |
			  NP_TYPE_BYTE		 |
			  NP_READMODE_BYTE	 |
			  NP_UNLIMITED_INSTANCES;

Using these constants the pipe can be created.

    ULONG 	rc;
    HPIPE 	hPipe;
    rc = DosCreateNPipe(pcszName,
			 &hPipe,
			 NP_CREATE_FLAGS,
			 NP_PIPE_FLAGS,
			 SIZE_PIPE,
			 SIZE_PIPE,
		 	 0);

As the HCONNECTION that will be returned from RmxCreate() is in fact a CONNECTION, some memory must also be allocated. As this takes place in a DLL I chose to allocate the memory using DosAllocMem() directly, the main reason being that I don't know how safe it is to call runtime library functions inside a DLL that will be used by multithreaded application, that are not necessarily even compiled with the same compiler.

ULONG 	rc;
PVOID 	pvMemory;

do
  rc = DosAllocMem(&pvMemory, sizeof(CONNECTION), PAG_READ | PAG_WRITE | PAG_COMMIT);
while (rc == ERROR_INTERRUPT);

CONNECTION *pconn = (PCONNECTION) pvMemory;

memset(pconn, sizeof(CONNECTION), 0);

pconn->hPipe = hPipe;

ULONG RmxCreateUnique(ULONG*	 pulSize,
		    PSZ		 pszName,
		    HCONNECTION* phConn);

RmxCreateUnique() is used by the engine to create a unique connection. The caller needs to provide a buffer sufficiently large where the used name will be returned. Internally the biggest question is how to create a unique name. A reasonably reliable method is to use the process id, thread ordinal and the time.

const CHAR  acUniqueName[]  = "\\PIPE\\RMXPIPE\\UNIQUE\\";
      PPIB	 ppib;
      PTIB 	 ptib;

DosGetInfoBlocks(&ptib, &ppib);
DATETIME dateTime;
DosGetDateTime(&dateTime);

     USHORT 	 pid	    = (USHORT) ppib->pib_ulpid,
        	 tord       = (USHORT) ptib->tib_ordinal,
	         seconds    = (USHORT) dateTime.seconds,     // <= 59
	         hundredths = (USHORT) dateTime.hundredths;  // <= 99

     sprintf(pszName, "%s%hu%hu%hu%hu", acUniqueName, pid, tord, seconds, hundredths);

and then create the named pipe in a loop that eventually either succeeds properly or fails completely.

     do
     {
	   GetUniqueName(achUniqueName);
 	   rc = RmxCreate(achUniqueName, phConn);
     }
     while (rc == RMXERR_CONNECTION_BUSY);

ULONG  RmxOpen(PCSZ pcszHost, PCSZ pcszPort, HCONNECTION* hConn);

RmxOpen() is used by a client process to open a connection. The arguments it takes are the name of the host and the port to open and a pointer to a variable where the connection handle will be returned. When named pipes are used the host name will be the name of a server and the port the name of a named pipe.

To get the qualified name we simply need to concatenate the host and the port names. In order to do that we need a buffer. Instead of first allocating a buffer for this purpose and then another for the actual connection handle we first allocate the connection handle buffer and use it for storing the string. The allocation is done pretty much the same way as in RmxCreate().

  CONNECTION 	 *pconn = (PCONNECTION) pvMemory;
  PSZ 	        pszName = (PSZ) pconn;

  pszName[0] = 0;

     if (pcszHost)
	 strcpy(pszName, pcszHost);
     strcat(pszName, pcszPort);

The host name can be NULL which effectively means that we attempt to open the pipe on the same computer. Once the name is built we can open the actual pipe.

In RMX this function will be called behind the scenes of an application which may be doing a lot of things, e.g. open a large number of files. Hence we make sure the function does not fail because there are no handles left.

     const ULONG NP_MODE_FLAGS = OPEN_FLAGS_WRITE_THROUGH |
				  OPEN_FLAGS_FAIL_ON_ERROR |
			          OPEN_FLAGS_NO_CACHE |
				  OPEN_FLAGS_SEQUENTIAL |
			          OPEN_SHARE_DENYNONE |
				  OPEN_ACCESS_READWRITE |
				  OPEN_FLAGS_NOINHERIT;

    HPIPE 	hPipe;
    ULONG 	rc;

    do
    {
	ULONG 	  ulActionTaken; // Not used.
 	rc = DosOpen(pszName, 
                    &hPipe,
    		     &ulActionTaken,
		     0L,
	             FILE_NORMAL,
		     FILE_OPEN,
		     NP_MODE_FLAGS,
		     0L);

	if (rc == ERROR_TOO_MANY_OPEN_FILES)
	{
	  LONG 	    lReqCount = 1; // We need one additional handle
	  ULONG	    ulCurMaxFH;	 // Not used.
	  DosSetRelMaxFH(&lReqCount, &ulCurMaxFH);
	}
    }
    while (rc == ERROR_TOO_MANY_OPEN_FILES);

Then we set the state of the pipe the same way as it was set in RmxCreate(). I am not entirely sure whether this is necessary or if the state is automatically set to the same as what was used when the pipe was created, but better safe than sorry.

    if (rc == NO_ERROR)
    {
	// The connection must be blocking.
	rc = DosSetNPHState(hPipe, NP_WAIT | NP_READMODE_BYTE);
	...

ULONG RmxClose(HCONNECTION hConn);

RmxClose() is simply implemented by deallocating the memory that was allocated in RmxCreate(), and closing the pipe

    CONNECTION	*pconn = (PCONNECTION) hConn;

    DosClose(pconn->hPipe);
    DosFreeMem(pconn);

ULONG RmxConnect(HCONNECTION hConn);

This function is mapped directly to DosConnect(). It blocks until a client opens the pipe.

    CONNECTION *pconn = (CONNECTION*) hConn;
    ULONG 	rc = 0;

    do
	rc = DosConnectNPipe(pconn->hPipe);
    while (rc == ERROR_INTERRUPT);

ULONG RmxDisConnect(HCONNECTION hConn);

Like RmxConnect() is mapped to DosConnect(), RmxDisConnect() is mapped to DosDisConnect().

    CONNECTION	*pconn = (CONNECTION*) hConn;

    DosDisConnectNPipe(pconn->hPipe);

ULONG RmxWrite(HCONNECTION hConn, PCBYTE pbBuffer,
		   ULONG  ulBytesToWrite);

Okay, now we are getting to the more interesting stuff. This functions takes a handle to a connection, a pointer to a buffer and an unsigned long specifying how many bytes to write. Contrary to usual write-functions, it does not return the number of bytes written. This is because in RMX, where this function is used for implementing RPCs, the function must either succeed or completely fail. Also, if the number of bytes to transfer is too large for the underlying implementation (in this case named pipes) the function must itself deal with the task of splitting the message into smaller packets.

     CONNECTION *pconn = (PCONNECTION) hConn;
     BYTE 	 *pbData = pconn->bytes;

The length of the entire message is sent in the first 4 bytes as an unsigned integer.

     *((ULONG*) pbData) = ulBytesToWrite;

     ULONG 	 ulBytesInBuffer = SIZE_BUFFER - sizeof(ULONG);
     ULONG 	 ulBytesToCopy   = ulBytesInBuffer < ulBytesToWrite ?
				 ulBytesInBuffer : ulBytesToWrite;

     memcpy(pbData + sizeof(ULONG), pbBuffer, ulBytesToCopy);

Okay, so what is happening here? After the size, we copy as much actual data as can fit into the data area of the connection handle. An alternative would be to first send the size, and then send the data directly out of the buffer provided by the caller. But that would mean that no matter how small a message, it would always be sent in at least two separate packets. In the code that follows note that during the first DosWrite(), pbData points to the connection buffer, but on subsequent calls it points directly to the buffer given by the caller.

ULONG 	 ulBytesLeft   = ulBytesToWrite + sizeof(ULONG),
	 ulBytesToSend = ulBytesToCopy  + sizeof(ULONG);
LONG 	 lBytesSent = -sizeof(ULONG);
ULONG 	 rc = RMXAPI_OK;

do
{
  ULONG  ulBytesSent;
  rc = DosWrite(hPipe, pbData, ulBytesToSend, &ulBytesSent);

 lBytesSent  += ulBytesToSend;
	 ulBytesLeft -= ulBytesToSend;

 if (ulBytesLeft != 0)
 {
     pbData = (PSZ)pbBuffer + lBytesSent;
     ulBytesToSend = SIZE_PIPE < ulBytesLeft ?
	              SIZE_PIPE : ulBytesLeft;
 }
}
while (ulBytesLeft != 0);

The funny looking plus and minus sizeof(ULONG) is there because the first message contains the size information which is not part of the actual message and must be excluded when we figure out whether data needs to be copied still. This ought to be cleaned up somwhat, I get nervous when signed and unsigned quantities are mixed in this fashion.

ULONG RmxRead(HCONNECTION hConn,  PBYTE pbBuffer, ULONG ulSize, ULONG* pulBytesRead);

This function takes a handle to a connection, a pointer to a buffer, an unsigned long that specifies the size of the buffer, and a pointer to an unsigned long where the size of the read message is returned. If the buffer is not big enough the function returns an error code indicating that the buffer must be enlarged and the required size is returned in the variable pointed to by pulBytesRead.

If the variable indicating the length is 0, this is a true new call to RmxRead().

CONNECTION	 *pconn = (PCONNECTION) hConn;
HPIPE 	 hPipe = pconn->hPipe;
ULONG 	 *pulLength = (ULONG*) pconn->bytes;

If it is 0, we read the size as a separate operation. We have to do that because we do not want to read to the size to the buffer provided by the caller, but to the buffer associated with the handle.

if (*pulLength == 0)
{
  ULONG     ulBytesLeft = sizeof(ULONG),
	     ulBytesRead = 0;
  do
  {
    PBYTE   pbData = ((PBYTE) pulLength) + ulBytesRead;
    DosRead(hPipe, pbData, ulBytesLeft, &ulBytesRead);
    ulBytesLeft -= ulBytesRead;
  }
  while (ulBytesLeft != 0);
  if (*pulLength > ulSize)
  {
    *pulBytesRead = *pulLength;
    return RMXERR_ENLARGE_BUFFER;
  }
}

Perhaps it is overkill to read 4 bytes in a loop like that, but at least we'll be certain that we get the data we are interested in. The loop for reading the actual message data looks pretty much the same. Note that pbData now points to the buffer provided by the caller.

 ULONG	 ulBytesLeft = *pulLength,
	 ulBytesRead = 0,
	 ulTotalBytesRead = 0;
 do
 {
   PBYTE pbData = pbBuffer + ulTotalBytesRead;
   rc = DosRead(hPipe, pbData, ulBytesLeft, &ulBytesRead);

   ulBytesLeft -= ulBytesRead;
   ulTotalBytesRead += ulBytesRead;
 }
 while (ulBytesLeft != 0);
 *pulBytesRead = *pulLength;

The length is set to 0 to indicate that the entire message has been read.

*pulLength = 0;

RMXTCPIP.DLL

Okay, now we begin to look at the equivalent TCP/IP implementation. When I initially specified the communications API I followed the named pipe model fairly closely. The reason was that I wanted it to be easy to implement the RMX API on top of them, but also because I had very little experience with TCP/IP and I didn't exactly know how they work. But, I was confident that it would later be possible to fit them into the RMX model.

If we look at named pipes and TCP/IP in a multithreaded client-server situation, the basic scenario can be pictured as (_beginthread() is called when a client opens the connection the server blocks on):

Part2-rmx4.gif

From this figure it is fairly obvious that there is not a direct mapping between named pipes and sockets. And while it is trivial to implement the RMX API using named pipes, sockets require a little bit more thought.

The main problem is the difference in how multiple connections with the same name is handled. With named pipes the server simply calls DosCreateNPipe() and DosConnect() multiple times. The name given to DosCreateNPipe() is each time the same. With sockets you first create a socket using socket(), bind() and listen(), and then you call accept() multiple times. Each call to accept() returns a new socket handle. Like most other things in computer science this can be solved by adding an extra level of indirection.

ULONG RmxGetServiceName(PCSZ pcszService, ULONG* pulSize, PSZ pszName);

Now that we are running on top of TCP/IP we can directly use the \tcpip\etc\services file. The file should contain entries like:

rmxserver	  4711/tcp
rmxstarter	  4712/tcp
rmxsecurity	  4713/tcp
rmxtester	  4714/tcp

The first column contains the public name of a server, and the second the port number and the protocol to be used. The port numbers are something that basically the world has to agree upon, so I just chose a few numbers that did not seem to be used by anybody. They are easily changed if they happen be in conflict with something else. The socket library directly provides functions for dealing with this file. As the port is returned in network format we have to convert it to host format using ntohs().

servent *s = getservbyname((PSZ) pcszService, "tcp");
if (!s)
  return RMXERR_UNKNOWN_SERVICE;
sprintf(pszName, "%d", ntohs((unsigned short) s->s_port));
ULONG RmxCreate(PCSZ pcszName, HCONNECTION* phConn);

RmxCreate() is used by a server process to create a connection. And as it is supposed to be called several times with the same name we cannot directly map it onto the usual socket creation functions. The first thing we do is to convert the given name to a port number (the user supposedly has called RmxGetServiceName() before this).

 int port = atoi(pcszPort);

 return CreateConnection(htons((unsigned short) port), phConn);

All work is done by CreateConnection() which is also used by RmxCreateUnique(). As we need to be able to detect whether the caller actually wants to create a new socket or if he only wants to have a new instance of the same socket we need to have an additional data structure for that purpose.

struct SOCKET
 {
   unsigned short port;	// The port number in network format.
   int 	   socket;      // The socket handle.
   unsigned       users;	// Number of users of this instance.
   SOCKET*        next;        // Pointer to next instance (or NULL).
   SOCKET*	   prev;        // Pointer to previous instance (or NULL).
 };

const ULONG SIZE_BUFFER = SIZE_CONNECTION - sizeof(SOCKET*) - sizeof(int);
struct CONNECTION
 {
   SOCKET* sharedSocket;
   int	   privateSocket;
   BYTE    bytes[SIZE_BUFFER];
 };

The first thng CreateConnection() does is to get an instance of SOCKET. I don't show it here (it's all available in the source code) but PickSOCKET() scans through all existing SOCKET instances and if it finds one instance where the port number is the same as the one requested, it increases the use count and returns it. Otherwise it creates a new instance.

 SOCKET *pSocket = PickSOCKET(port);
 if (pSocket->socket == 0)
   rc = InitializeSOCKET(pSocket);
 PCONNECTION pconn = AllocConnection();
 pconn->sharedSocket = pSocket;
 *phConn = pconn;

If it was an all new instance there is no real socket associated with it yet. The socket is created in InitializeSOCKET() that as its first action calls CreateSocket(). CreateSocket() creates the socket handle and deals with system call interruptions. It is in a separate function because it is also used by RmxOpen().

 static int CreateSocket()
  {
    int   serrno = 0,
	   socket = 0;

    do
      {
	 socket = ::socket(AF_INET, SOCK_STREAM, 0);
        if (socket < 0)
          serrno = sock_errno();
      }
    while (serrno == SOCEINTR);
    if (serrno != 0)
      return 0;
    return socket;
  }

The socket handle is just a handle, it cannot be used for anything yet. It must first be bound which means that a local address is assigned to it. If the specified port is 0, the system allocates a unique port on behalf of the caller.

int socket = CreateSocket();
sockaddr_in in;
     in.sin_family	= AF_INET;
     in.sin_port	= pSocket->port;
     in.sin_addr.s_addr = INADDR_ANY;

     ::bind(socket, (sockaddr*) &in,
     sizeof(sockaddr_in));

The operation is not complete yet. We need to have a connection request queue for the incoming requests. That is taken care of by listen().

::listen(socket, SOMAXCONN);

If the socket was created without an explicit port number, we ask the system what the actual port number is.

if (pSocket->port == 0)
{
  int size = sizeof(sockaddr_in);
  getsockname(socket, (sockaddr*) &in, &size);
  pSocket->port = in.sin_port; // Note, network order
}

  pSocket->socket = socket;

The call to AllocConnection() simply allocates an instance of CONNECTION and returns it. This completes the implemetation of RmxCreate(). Although not necessarily entirely obvious this arrangement allows RmxCreate() to be called multiple times using the same port number, yet the actual socket is created only once, on subsequent calls only its use-count is increased.

ULONG RmxCreateUnique(ULONG*	     pulSize,
		       PSZ	     pszName,
	               HCONNECTION*  phConn);

RmxCreateUnique() is implemented exactly like RmxCreate() except that the port number provided is zero. That will then cause the system to allocate some unique port.

CreateConnection(0, phConn); 
PCONNECTION pconn = (PCONNECTION) *phConn;
sprintf(pszPort, "%d", ntohs(pconn->sharedSocket->port)); 
ULONG  RmxOpen(PCSZ pcszHost, PCSZ pcszPort, HCONNECTION* hConn);

As the host and port names are expressed textually they must be converted into integers before we can use them. The port is simply converted using atoi(), the host needs a little bit more attention.

int port = atoi(pcszPort);

First we check whether the host has been specified in ordinary dotted decimal notation (e.g. 192.26.110.20).

unsigned long address = 0;
  if (pcszHost)
     {
	 address = inet_addr((PSZ) pcszHost);
	   if (address == (unsigned long) -1)
    {

If it hasn't we attempt to look it up from the hosts file, or from a nameserver if present (handled automatically by gethostbyname()).

hostent  *host = gethostbyname((PSZ) pcszHost);
address = *(unsigned long*) host->h_addr;
   }
  }

When we have the complete address, we can create and connect the socket. First we need the actual socket handle.

int socket = CreateSocket();

Just like when we were creating the connection this socket cannot be used for anything. It has to be connected to a real address. We use the port and address we just figured out. Again the port must be converted into network format. The address is in network format already when it is returned by inet_addr() or gethostbyname().

sockaddr_int	 sin;
sin.sin_family	 = AF_INET;
sin.sin_port        = htons((unsigned short) port);
sin.sin_addr.s_addr = address;

::connect(socket, (sockaddr*) &sin,
sizeof(sockaddr_in));

The only thing left is to allocate the connection handle.


     PCONNECTION
	 pconn = AllocConnection();

     pconn->privateSocket = socket;

     *phConn = pconn;

As the memory returned by AllocConnection() is set to 0, pconn->sharedSocket will also be NULL.


ULONG RmxClose(HCONNECTION hConn);

RmxClose() is used by both the client and the server to close a connection. The client will never have a shared socket, and the server should not - provided it has first (as it ought to) called RmxDisConnect() - have a private socket.

PCONNECTION pconn = (PCONNECTION) hConn;
if (pconn->sharedSocket)
  ReleaseSOCKET(pconn->sharedSocket);
  int socket = pconn->privateSocket;
  DosFreeMem(pconn);
  if (socket)
    ::soclose(socket);

ReleaseSOCKET() decrements the use count of the shared socket, and if it reaches 0, it closes the real socket and deletes the SOCKET instance.

ULONG RmxConnect(HCONNECTION hConn);

This function can be directly mapped to the socket call accept(). Accept() blocks the caller until a client opens a connection.

PCONNECTION pconn = (PCONNECTION) hConn;
int        socket = ::accept(pconn->sharedSocket->socket, 0, 0);
pconn->privateSocket = socket;

Accept() returns a unique socket handle that can be used for communicating with the client that opened the connection

ULONG RmxDisConnect(HCONNECTION hConn);

RmxDisConnect() is called by the server to close a specific client connection.

PCONNECTION pconn = (PCONNECTION) hConn;
     ::soclose(pconn->privateSocket);
     pconn->privateSocket = 0;

The shared socket is still there, so we can use the same connection handle for servicing a new client by calling RmxConnect().

ULONG RmxWrite	(HCONNECTION hConn, PCBYTE pbBuffer, ULONG ulBytesToWrite);
ULONG RmxRead 	(HCONNECTION hConn, PBYTE pbBuffer, ULONG ulSize, ULONG* pulBytesRead);

The socket implementation of RmxRead() and RmxWrite() is almost identical to the named pipe implementation. The main difference is that instead of using DosRead() and DosWrite() we use recv() and send().

RMXCOMMS.DLL

If you look at the makefiles of RMXPIPE.DLL and RMXTCPIP.DLL you'll see that I generate no import library from the DLLs. That is on purpose. The way I see it, these two DLLS are only an implementation of an API. Any program that uses the API should be able to use another DLL that implements the same API using another communications mechanism.

If an application is linked with an import library generated from RMXPIPE.DLL, for example, then that application is forever married to the DLL. If the application should use another DLL it must be relinked with an import library generated from the other DLL.

If the application is instead linked with RMXCOMMS.LIB which is generated from RMXCOMMS.DLL, the application need never be relinked in order to use a particular implementation of the RMX communications API.

RMXCOMMS.DLL provides exactly the same interface as RMXPIPE.DLL and RMXTCPIP.DLL, and is in fact built using the same header as they are. Its implementation is, however, quite different.

First a few types are defined: PPFN is a pointer to a function pointer and Function is structure holding an ordinal and a pointer to a function pointer.

typedef PFN* PPFN;

struct Function
{
        ULONG ulOrdinal;
	 PPFN  ppfnAddress;
};

Then a number of module variables. As you notice there is one function pointer for each function defined by the RMX communications API. The addresses of the function pointers are stored, together with the corresponding ordinal, in an array of function structures.

static HMODULE hmodRmxComms;
static ULONG RMXENTRY (*rmxClose)(HCONNECTION);
static ULONG RMXENTRY (*rmxConnect)(HCONNECTION);
static ULONG RMXENTRY (*rmxCreate)(PCSZ, HCONNECTION*);
static ULONG RMXENTRY (*rmxCreateUnique)(ULONG*, PSZ, HCONNECTION*);
static ULONG RMXENTRY (*rmxDisConnect)(HCONNECTION);
static ULONG RMXENTRY (*rmxGetServiceName)(PCSZ, ULONG*, PSZ);
static ULONG RMXENTRY (*rmxOpen)(PCSZ, PCSZ, HCONNECTION*);
static ULONG RMXENTRY (*rmxRead)(HCONNECTION, PBYTE, ULONG, ULONG*);
static ULONG RMXENTRY (*rmxWrite)(HCONNECTION, PCBYTE, ULONG);
static Function afFunctions[] =
{
  { ORD_RMXCLOSE,          (PPFN) &rmxClose },
  { ORD_RMXCONNECT,	   (PPFN) &rmxConnect },
  { ORD_RMXCREATE,	   (PPFN) &rmxCreate },
  { ORD_RMXCREATEUNIQUE,   (PPFN) &rmxCreateUnique },
  { ORD_RMXDISCONNECT,	   (PPFN) &rmxDisConnect },
  { ORD_RMXGETSERVICENAME, (PPFN) &rmxGetServiceName },
  { ORD_RMXOPEN,		   (PPFN) &rmxOpen },
  { ORD_RMXREAD,		   (PPFN) &rmxRead },
  { ORD_RMXWRITE,          (PPFN) &rmxWrite }
};

const ULONG cFunctions	   = sizeof(afFunctions)/sizeof(Function);

Let's then look at the DLL initialization routine of RMXCOMMS.DLL. The first thing is to query the value of the environment variable RMXCOMMS.

PCSZ  pcszRmxComms;
if (DosScanEnv("RMXCOMMS", &pcszRmxComms) != NO_ERROR)
return 0;

The value of RMXCOMMS is assumed to be the name of the actual communications DLL to use.

if (DosLoadModule(0, 0, pcszRmxComms, &hmodRmxComms) != NO_ERROR)
	 return 0;

Finally the functions can be resolved.

	for (int i = 0; i < cFunctions; i++)
	  {
	    ULONG
		ulOrdinal	= afFunctions[i].ulOrdinal;
	    PPFN
		ppfnAddress = afFunctions[i].ppfnAddress;

	    ULONG
		rc = DosQueryProcAddr(hmodRmxComms, ulOrdinal, 0, ppfnAddress);

	    if (rc != NO_ERROR)
		return 0;
	  }

This means in practice that once the DLL has successfully been loaded, the function pointers will point at valid functions. Now each exported function can simply be implemented in the following fashion.

ULONG RMXENTRY RmxCreate(PCSZ pszName, HCONNECTION* phConn)
{
 return rmxCreate(pszName, phConn);
}

Example

Suppose we have an application CLIENT.EXE that has been linked with RMXCOMMS.LIB which was generated from RMXCOMMS.DLL. And suppose that there exists the proper communications DLLs RMXPIPE.DLL, RMXTCPIP.DLL and RMXIPX.DLL. Then the following three client processes will all use different DLLS.

[C:\]set RMXCOMMS=RMXPIPE
[C:\]start client.exe
[C:\]set RMXCOMMS=RMXTCPIP
[C:\]start client.exe
[C:\]set RMXCOMMS=RMXIPX
[C:\]start client.exe

Sample Programs

Along with this issue of EDM/2 you should get RMXCOMMS.ZIP that contains RMXCOMMS.DLL, RMXPIPE.DLL and RMXTCPIP.DLL, and the test/demo programs SERVER.EXE, ENGINE.EXE and CLIENT.EXE, along with all the source. In order to try them out, you should place the DLLs in some directory along the LIBPATH. Remember to place the file RMXPIPE.DAT in the same directory as RMXPIPE.DLL, and remember to update the \tcpip\etc\services file if you intend to use RMXTCPIP.DLL. SERVER.EXE and ENGINE.EXE should be in the same directory.

If you want to run the server/engine pair and the client in separate computers you'll need a network. The DLLs must of course be present in both computers.

Starting the server

Launch a windowed command prompt and go to the directory where SERVER.EXE resides.

[C:\RMX\DEMO]set RMXCOMMS=RMXPIPE
[C:\RMX\DEMO]server

The server prints a notification message when it starts.

Starting the client

The client requires a few command line parameters in order to start. If it is started with the wrong parameters it prints:

 usage: client [-t #] ([-h host]|-e engine) [-v]
 
 	   -t #	 : Number of threads
 	   -h host	 : The name of the host
 	   -e engine : Use this as engine connection
 	   -v verbose: More chit chat
 
 Note the blank between the flag and the value
  
 example: client -t 5 -h \\ODIN
          client -v

The -t flags specifies how many threads the client should use. As default it uses 5. The flag -h specifies the host and -v turns on verbose mode (recommended). If you are running locally don't specify a host,

[C:\RMX\DEMO]set RMXCOMMS=RMXPIPE
[C:\RMX\DEMO]client -v

otherwise the argument of the flag should be the name of the computer where the server is running. The host name varies of course depending on which communications DLL you are using. In my environment I use:

[C:\RMX\DEMO]set RMXCOMMS=RMXPIPE
[C:\RMX\DEMO]client -h \\loke -v

with named pipes and

[C:\RMX\DEMO]set RMXCOMMS=RMXTCPIP
[C:\RMX\DMEO]client -h loke -v

or

[C:\RMX\DEMO]set RMXCOMMS=RMXTCPIP
[C:\RMX\DMEO]client -h 192.26.110.21 -v

with TCP/IP. When started the client opens a connection to the server and asks it to start an engine, the server starts the engine that creates a connection whose name is transferred back to the client. The client opens a new connection to the engine and starts sending messages of varying size to the engine that subsequently sends them unaltered back. The client then verifies that there has been no changes in the message.

Compiling the source

I've written the code using Borland C++ 1.5 so if you have that compiler you should be able - perhaps with a little tinkering - to build the DLLs and the applications. If you have another compiler you'll probably have to make more modifications. If you do make changes I'd appreciate if you would mail them to me so I can include them in the "official" version.

A word of caution

I've been using the RMXPIPE.DLL implementation for quite some time while I've been developing RMX and I have not experienced any problems. However, now when I tested the programs for this article, the client occasionally received an error code (ERROR_REQ_NOT_ACCEP to be exact) when it tried to send data to the server. I havn't experienced any problems at all with RMXTCPIP.DLL so I blame the named pipe implementation of OS/2.

Implementing RPCs

Even if this API hides quite a few of the complex details of client-server programming, it is still quite tedious to work with. Fortunately it can be made easier. This article is long as it is so I won't go into any details but I'll show you how WinInitialize is implemented in RMX. I might return to this subject in a later article. C++ templates can make the world a better place to program in...

Client

HAB APIENTRY WinInitialize(ULONG flOptions)
{
  ClientConnection
    &c = ClientConnection::connection();
  Packet
    &p = c.freshPacket();

  p << RMX_OS2PMWIN;
  p << (ORDINAL) ORD_WIN32INITIALIZE;
  p << flOptions;

  c.sendReceive();

  HAB
    hab;

  p >> hab;

  return hab;
}

Server

void PMWin::winInitialize(Packet& p)
{
  ULONG
    flOptions;

  p >> flOptions;

  HAB
    hab = WinInitialize(flOptions);

  p.reply() << hab;
}

Conclusion

So, that was the low level transfer mechanism of RMX. Feel free to use it in any way you see fit and don't hesitate to mail me if there is something you wonder about. The next article will be about how to mark an application for use with RMX.