The Shell Game - Alternative Shells for OS/2

From EDM2
Jump to: navigation, search

by Monte Copeland

During OS/2's six-year history, IBM has shipped three different shells for OS/2. OS/2 Version 1.0 had a character shell called the Program Selector. The shell for OS/2 versions 1.1, 1.2, and 1.3 was actually a suite of Presentation Manager (PM) applications for program launching, spooler management, file management, and system control. OS/2 2.0 premiered the Workplace Shell, the first implementation of Common User Access (CUA) '91, and the shell for current and future versions of OS/2.

Why Use a Different Shell?

With an easy-to-use GUI such as the Workplace Shell, why would you choose to use a different shell? Admittedly, using a different OS/2 shell is not for everybody. But some applications might benefit from a different shell: bulletin board systems, specialized servers, embedded systems, UNIX alternatives. The designer of such systems will forego an easy-to-use interface to regain system resources such as memory and thread slots. On the other hand, secure systems might require more shell, not less.

This article provides some shell alternatives for OS/2. One alternative is to subclass Workplace Shell objects to provide access control. Another removes Workplace Shell, but leaves PM. The last two shells completely remove PM, providing a low-resource, text-only shell.

Shell Access Control

"I want to stop users from opening a command prompt." This quote is typical of MIS managers who wish to restrict users' access. Because the Workplace Shell is based on object-oriented programming, you can invoke existing methods or subclass Workplace Shell objects and change their behavior.

A logical first step is to password-protect folders. Subclass the WPFileSystem class, override the _wpOpen method, and present a password dialog box before invoking _wpOpen in the parent. The International Technical Support Center (ITSC) Red Books OS/2 Workplace Shell Exploitation (GG24-4201) and OS/2 Version 2.0 Volume 4: Application Development (GG24-3774) contain some sample code on how to do this.

DeskMan/2 from Development Technologies, Inc. (803-790-9230) already does this and more. The PSP Application Assistance Center in Basingstoke, UK (+44-256 -344530) has a restricted-shell product available in most European countries.


All OS/2 shells have one thing in common: The name of the shell program appears on the PROTSHELL line in the config.sys file. This name must be an OS/2 protected-mode executable program (.exe). The .exe named on PROTSHELL is the first process spawned after system initialization (SYSINIT).

If the PROTSHELL executable is not a PM application and does not call WinCreateMsgQueue, then it will operate in a single-session, single-console, protect-mode-only, text-only environment. It can spawn other protected-mode, non-PM processes in the same session, but the operating system will not manage access to the console. A process can detach any process that does not require a console.

If the PROTSHELL executable is a PM application (it calls WinCreateMsgQueue), the session manager is initialized. That process has the distinction of being the session manager process. You can replace the PM shell this way, but the RUNWORKPLACE architecture is preferred.

The RUNWORKPLACE Architecture

In config.sys, the environment variable RUNWORKPLACE names an executable . When pmshell.exe is the PROTSHELL (as it usually is), it will spawn the program named by RUNWORKPLACE. Usually, the RUNWORKPLACE program is also pmshell.exe, and this is how Workplace Shell starts. By naming a different executable on the RUNWORKPLACE setting, you can start a different shell program.

This architecture is perfect for turnkey applications. You write a program, name it on the RUNWORKPLACE setting, and you have turn-it-on-and-use-it dedicated machine.

Some things work as expected: Presentation Manager is initialized, Ctrl-Esc presents the Window List, Alt-Esc switches sessions, REXX works, and startup.cmd starts. But the RUNWORKPLACE process is responsible for the rest: Starting and managing the spooler, installing PM printer drivers, presenting and starting programs, and providing a safe shutdown.

In this environment, OS/2 text-only applications, OS/2 Presentation Manager applications, DOS applications, and Windows applications run fine. Because there is no Workplace Shell process, you must test Workplace Shell-aware applications to see if they work; chances are they will. However, DLL-only Workplace Shell applications will not run because they expect pmshell.exe, running as the RUNWORKPLACE process, to load them.

MShell: A Mini PM Shell

Volume 3 of The Developer Connection for OS/2 CD-ROM contains a sample program called MShell that performs the minimum requirements for a RUNWORKPLACE shell. MShell is a mini PM shell that basically is a program launcher. See Figure 1.


Figure 1. MShell: A Mini PM Shell for OS/2

MShell calls an Original Equipment Manufacturer Interface (OEMI) API SplQmInitialize to start the spooler. Because the function prototype is missing from The Developer's Toolkit for OS/2 2.1, Code 1 shows you how to provide your own.


Code 1. Prototype for function to start the spooler.

To resolve the reference to SplQmInitialize at link time, import it from pmspl.dll in your .def file. See Figure 3. At runtime, call SplQmInitialize with a pointer to a ULONG. On return from a successful spooler start, expect the ULONG to contain zero. MShell uses the normal spooler APIs to manage the spooler including SplEnumQueue, SplHoldQueue, SplReleaseQueue, SplHoldJob, SplReleaseJob, and SplDeleteJob.


Code 2. Module definition file statements to resolve the external SplQmInitialize at link time.

Listing and starting programs is easy to do. MShell accepts program start information from a plain-text .ini file and presents the programs in a listbox. When the user selects a program, MShell spawns a background copy of cmd.exe to do the start. Starting DOS sessions with settings is tougher; the utility startdos.exe does that. To shut down, call WinShutdownSystem.

Many MShell users have 4MB machines with slow hard files. They perceive increased performance, but at the cost of usability. The reasons that a RUNWORKPLACE shell like MShell gets better performance involve the hard file because 1) MShell allocates 1.5MB less swappable memory than the Workplace Shell, so there is less paging to disk and 2) MShell does not use file extended attributes. Interestingly, MShell on unattended servers has little effect on performance; the system pages unused user-interface code to disk no matter what shell is running.


One interesting OS/2 configuration involves naming cmd.exe as the PROTSHELL process. Because cmd.exe does not initialize the session manager, the machine will have a single, protected-mode, full-screen session running cmd.exe. Attempts to start more sessions will fail.

The best reason to name cmd.exe on the PROTSHELL is to save time booting OS/2 from floppies. And, the best reason to boot from floppies is to run

(To make bootable OS/2 2.1 floppies, make copies of the OS/2 Installation Diskette and Diskette 1. From Diskette 1, delete the following files: bundle, sysinst2.exe and From Diskette 2, copy uhpfs.dll and to Diskette 1. Finally, edit config.sys on Diskette 1 and name cmd.exe on the PROTSHELL. For a single-floppy boot diskette, see bootos2.exe on your accompanying Developer Connection for OS/2 CD-ROM. On CompuServe, type Go OS2USER at the ! prompt, and see in Library 17.)

For embedded OS/2, name your .exe on PROTSHELL. Inexpensive Intel 80386 planar boards running OS/2 can make a multiprocess, multithreaded control system. Granted, OS/2 is not ROM-able in the traditional sense; it requires a solid-state, ROM-based floppy device from which to boot. Such a system would operate without paging (MEMMAN=NOSWAP) and without the latency of paging. For example, a 4MB system has roughly 2MB free in this configuration.

Using cmd.exe on the PROTSHELL gives you a text-only OS/2, but without multiple sessions.

TShell: A Text Shell For OS/2

Some OS/2 users have expressed interest in a text-only, non-GUI PROTSHELL for OS/2. TShell (T for Text) is on the Developer Connection for OS/2 CD-ROM, Volume 3. TShell provides multiple OS/2 sessions; it can start DOS and WIN-OS/2 sessions if the machine is so configured.

TShell is similar to the OS/2 1.0 shell, Program Selector. On the left, there is a list of programs to start. On the right, there is a list of programs that are currently running. See Figure 2. Press Tab to alternate between the two lists. TShell implements traditional OS/2 hot keys: Alt-Esc visits each session in a round-robin manner, Ctrl-Esc goes directly to TShell.


Figure 4. TShell: A Text-Only OS/2 Shell

OS/2 with TShell will not run OS/2 PM applications because the PMWIN window manager is not initialized. See the TShell readme file for a list of restrictions and known bugs.

With its restrictions, TShell might serve a purpose in low-resource machines, embedded systems, or test beds.

Notes on TShell:

  1. Memory Use: TShell and the 2.1 debug kernel uses 2.9MB of RAM. The retail kernel will use slightly less memory. TShell can boot in a 4MB machine with or without swapping, although swapping generally is preferred.
  2. Doesn't Spool: There is no spooler with TShell.
  3. CID Server: TShell works on CID servers using SRVIFS. By using a low-resource shell, CID installers devote more memory to file system cache.
  4. Boots Fast: TShell appeals to ring 0 device driver developers who frequently trash test machines and want the fastest possible reboot.
  5. Programmable Start List: See the TShell readme for notes about using pgmshell.exe to customize TShell.

Future Considerations

All of these alternatives work well on OS/2 2.1. However, the suitability of these shells on future versions of OS/2 must be understood. In the order of least risk to most risk:

  • Subclassing Workplace Shell objects is the most conventional and well-documented approach to alternative shells for OS/2.
  • MShell, a RUNWORKPLACE shell, uses one OEMI API to start the spooler. This API may change, or perhaps the responsibility for starting the spooler will move to the PROTSHELL process. Either way, spooling is a must, and there will always be a way to start the spooler. MShell lacks a PM print driver installation process. MShell is on The Developer Connection for OS/2 CD-ROM with source code.
  • Using cmd.exe as PROTSHELL for a single-session boot will work as long as cmd.exe does not require PM. To date, cmd.exe has no dependency on PM.
  • TShell uses undocumented session manager APIs for initializing, starting, switching, and enumerating sessions. These have never been documented - not even as OEMI APIs. TShell should run on future Intel versions of OS/2. TShell is on The Developer Connection for OS/2 CD-ROM without source code.

Finally, these shells are written and tested by the author. They are not subject to official testing by IBM. IBM makes no warranties about correctness or suitability of these shells for any purpose on any version of OS/2.

Reprint Courtesy of International Business Machines Corporation, © International Business Machines Corporation