OS/2 Installable File Systems Part 1

From EDM2
Revision as of 23:59, 19 March 2018 by Ak120 (Talk | contribs)

Jump to: navigation, search

Written by Andre Asselin

Part 1 Part 2 Part 3

Introduction

One of the most useful but obscure features of OS/2 is it's support for installable file systems (IFSs). This API allows you to extend OS/2 to do two things: access files stored on disk in formats other than FAT and HPFS, and access files that are stored on a network file server. For example, under many versions of UNIX, files are stored in the Berkeley fast file system format, which isn't compatible with FAT or HPFS. If you have both UNIX and OS/2 installed on the same computer with partitions setup for each, you could write an IFS to read and write the UNIX files while OS/2 is running and give all the programs running under OS/2 (including DOS and Windows programs) access to those files.

The other variety of IFS is called a remote file system or redirector, and allows file sharing over LANs. In this case, when a program calls for a file, the IFS acts as a client and sends the request over a network to a file server. The file server handles the request and sends the data back to the client. Examples of this type of IFS include Novell's Netware requester and the requester part of LAN Server. Sometimes, in addition to just sending the requests to the server, this type of IFS also has to translate between OS/2 file conventions and file conventions that the server understands. For example, the NFS (Network File System) protocol is completely UNIX oriented, so the NFS client (which is an IFS) has to handle UNIX conventions like case sensitive filenames and UNIX file attributes.

Up until now, there hasn't been much written on how to develop an IFS. It also doesn't help that the documentation on the IFS API has been available only by special request from IBM. This series is going to explore the IFS API in depth with working examples in order to take the mystery out of writing them. This first article is a brief overview of the IFS concept to give a feel for what's involved in writing one. In the coming months, I'm going to explore the IFS architecture in depth and build up a toolkit that can be used as a skeleton for writing an IFS.

IFS Structure and How It Fits In

An IFS (which is also known as an FSD for File System Driver) can be thought of as a specialized kind of physical device driver (PDD), although it is structured very differently. Instead of having only two entry points and being based on request packets, it is structured as an ordinary 16-bit DLL with a series of predefined entry points, broken up into four groups, that are exported to the system (see Table 1). The first group consists of the primary entry points that all IFSs are required to export. This group does basic file manipulation, such as opening and closing, and reading and writing files. Most of these entry points correspond pretty closely in name and function to related DosXxx calls. The other three groups of calls provide support for the swapper, file locking, and UNC (Universal Naming Convention). For each of these groups, if your IFS doesn't support that particular feature, you don't have to export any of the functions in that group of entry points. Note that exports should be made on a group-by-group basis; don't export only some of the entry points from a group.

Figure 1 shows where an IFS fits into the OS/2 system.The two shaded components are what make up an IFS. The piece that runs in ring 0 (called MY.IFS in the figure) is the main component. It is called by the OS/2 kernel through one of its FS_XXX entry points whenever something needs to be done on a drive that the IFS is managing. The IFS performs the function and returns the requested data or an error code. Typically, it will interact with a PDD to do its I/O. Like any ordinary DLL, an IFS doesn't have a process or threads of its own. Instead, it operates in the context of the caller's process and on the caller's thread (in the case of an IFS, the caller's thread and process are the thread and process of the application making the Doscall). In this sense, an IFS can be thought of as a passive library that's not activated until a program calls it.

Since an IFS has to be available to every application in the system, it is loaded into global system memory (using GDT selectors). Any memory it allocates (such as for buffers) should also come from global system memory so that no matter which process its entered from, it'll still have access to the memory.

The piece that runs in ring 3 (UMY.DLL in the figure) is a utility library. There is one utility DLL for each IFS; the naming convention is a 'U' followed by the IFS name with an extension of .DLL. It has four entry points; one each for FORMAT, RECOVER, SYS, and CHKDSK. When a user runs any of those programs and it detects that it is being run against a drive being managed by an IFS, it will call the appropriate entry point in the IFS's utility DLL. The DLL is then completely responsible for carrying out the command.

How a Doscall Gets Processed

A typical Doscall (such as DosOpen, DosRead, or DosWrite) is processed mainly by the IFS with some help from the OS/2 kernel (see [Figure 2]). When an application issues a Doscall, such as DosOpen (1), a component of the OS/2 kernel called the file system router does some preliminary validation of the parameters. If everything's OK, it decides which IFS the call should be sent to and calls the IFS at the appropriate entry point, which in the case of DosOpen is FS_OPENCREATE (2). The IFS then looks at the parameters that were passed and decides on a course of action to service the call. In some cases, it may not support the function (for example, trying to open a remote drive for direct, sector-by-sector access), in which case it immediately returns an error code. If it does support the function, it might first check its cache to see if it already has the requested data, in which case it can just return it. If it doesn't have the data in its cache, it will probably have to do some I/O to get it.

In this case, it calls one of OS/2's file system helpers, FSH_DOVOLIO (3), which acts as an intermediary between the IFS and the device driver. FSH_DOVOLIO generates a request packet and calls the strategy entry point of the appropriate device driver (4). The device driver performs the I/O (5), and eventually returns to FSH_DOVOLIO, which returns back to the IFS. The IFS can then examine the return code for errors and take appropriate action. If it needs to, the IFS can the device driver multiple times until it has all the requested data. It then returns back to the OS/2 kernel, which in turn returns back to the application. The application can then examine the return code to see if the operation was successful.

Ring 0 Issues

The type of IFS described above runs completely in ring 0 and can be thought of as a 'traditional' IFS. Unfortunately, it can be difficult to develop because of the restrictions placed on code that runs at ring 0 (PDDs and VDDs suffer from these restrictions as well). There are three main restrictions:

No preemptive multitasking - When an IFS or any other ring 0 code is entered, preemptive multitasking is temporarily disabled. The IFS can still yield to other threads (i.e. cooperative multitasking), but it will never be suspended unless it specifically asks for it. This has the benefit of eliminating some synchronization issues, but it also means that periodic yields have to be sprinkled throughout lengthy operations.

No Doscalls; only file system helpers and DevHlps - None of the ring 3 API's (such as the Doscalls) are available to ring 0 code (except at initialization, which is special). If you use any functions from the C library, you have to make sure that they don't in turn use any of the Doscalls (i.e. malloc and free don't work at ring 0 because they call DosAllocMem). As a partial replacement, there are a series of file system helper (FSH) calls available for IFS's (see Table 2) as well as the standard DevHlp calls available to all device drivers. These APIs perform common tasks such as allocating memory and handling semaphores, but they are more primitive and thus more difficult to use than the Doscalls.

Debugging is a pain - If you thought debugging PM applications was tricky, wait until you try an IFS! First, you won't be able to use your favourite debugger - you have to use a ring 0 debugger, such as the OS/2 kernel debugger or ASDT32. Neither of these can do source level debugging, so you have to know assembler, and they also have limited support for labels. There is also no such thing as a printf in ring 0, so you have to use a hex dump to look at variables. As if that weren't bad enough, OS/2's crash protection doesn't apply to ring 0 code; if your code has a bug and traps, it will bring down the system with an IPE (Internal Processing Error). This is where you get the blue screen, lots of register information, a suggestion to call IBM, etc. Frequent reboots tend to be the way of life.

The Split IFS/Control Program Approach

Because of these restrictions, a popular approach to developing an IFS is to split it between ring 0 and ring 3. The ring 0 piece (the IFS proper) contains only stub routines and passes all the calls up to a ring 3 control program. The control program handles the calls and then passes the results back to the ring 0 piece. The advantages of doing it this way are that developing the bulk of the IFS is easier (because it runs at ring 3, you don't have to worry about those restrictions mentioned above), and you have access to all the normal ring 3 APIs. This approach also allows you to layer an IFS on top of an already existing IFS; for example, DCF/2 and Stacker for OS/2 probably take this approach to use FAT or HPFS to store their compressed data files. This way they don't have to worry about totally rewriting the FAT or HPFS code - all they have to be concerned with is compressing the data on the fly and then using the already existing IFSs to actually store it on the disk.

This type of IFS is usually implemented using captive threads. [Figure 3] traces a call as it winds its way between an application, the IFS, and its control program. When the control program initializes, it creates one or more threads that call into the IFS (1 and 2). The IFS makes these threads block until some application calls the IFS with work to do (3 and 4). The IFS then sets up a parameter area, blocks the application's thread and unblocks the control program's thread. The IFS returns back to the control program (5 and 6), where it executes the request. The control program then puts the results back in the parameter block and calls the IFS again (7 and 8). The IFS then unblocks the application's thread and has it return with the results (9 and 10) while the control program's thread is again blocked waiting for a new request.

What's Ahead

Well, that's a high level overview of what's involved in writing an installable file system. In the next issue, I'm going to start developing the framework to implement a split ring 0 / ring 3 IFS. To illustrate how to use it, I'm going to write an IFS that supports file versions (like VMS).

While you're waiting for the next issue, you may want to start gathering the tools necessary to write an IFS. The first thing you'll need, of course, is the documentation on the IFS API and the libraries and header files that go along with it. To get it, you need to send a note on CompuServe requesting the IFS documentation to Mike Abrams (72420,216) in the OS2DF1 forum section 11 (Device Driver Development) with your name and address. If you don't have an account on CompuServe, you can send him a note using the Internet; address it to 72420.216@compuserve.com (note the period instead of the comma in the number).

You'll also need two C compilers. For the ring 0 piece, you'll need a 16-bit C compiler. Most IFS developers use Microsoft C 6.0, but I'm going to investigate the possibility of using Borland C++ instead. If I can make it work, that's what I'm going to use, but the code should be compatible with both compilers. For the ring 3 piece, you'll need a 32-bit compiler, such as IBM's C Set++ or Borland C++ for OS/2. Finally, you probably won't need it right away, but you should also get a hold of either the kernel debugger or ASDT32, which is available through IBM's EWS program.

If this series is interesting to you, please drop me a note telling me a little bit about yourself and your background so that I have a better idea who my audience is (it'll also help my motivation if I know that more than Steve and I are reading this series). While you're at it, if you have any feedback or ideas on what you'd like to see covered in future articles, I'd love to hear it. See you next month.

Table 1 - FSD Entry Points
Primary Entry Points
File Management

FS_OPENCREATE - Open or create a file

FS_CLOSE - Close a file

FS_READ - Read data from a file

FS_WRITE - Write data to a file

FS_CHGFILEPTR - Change the current position in a file

FS_COMMIT - Write all cached data for a file to disk

FS_DELETE - Delete a file

FS_NEWSIZE - Change the size of a file

FS_MOVE - Move or rename a file

FS_COPY - Copy a file

FS_FILEATTRIBUTE - Set or return the standard (DOS) attributes

FS_FILEIO - Perform an atomic read/write/lock operation

FS_FILEINFO - Set or return information on an open file

FS_PATHINFO - Set or return information on a file

Volume Management

FS_ATTACH - Associate or disassociate a drive letter with an FSD

FS_MOUNT - Examine a volume to determine if the FSD knows its format

FS_FSINFO - Get/Set file system information

FS_FLUSHBUF - Write all cached data for a volume to disk

Directory Management

FS_CHDIR - Change directory

FS_MKDIR - Make a new directory

FS_RMDIR - Remove a directory

Directory Search

FS_FINDCLOSE - End a search in progress

FS_FINDFIRST - Begin a new search

FS_FINDFROMNAME - Restart a search from a particular point

FS_FINDNEXT - Return next file name

FS_FINDNOTIFYCLOSE - Stop monitoring a directory for changes

FS_FINDNOTIFYFIRST - Start monitoring a directory file changes

FS_FINDNOTIFYNEXT - Return the next set of changes

FSD Extended Interface

FS_FSCTL - Extended interface to an FSD

FS_IOCTL - I/O control for devices

Miscellaneous

FS_NMPIPE - Support named pipe operations

FS_PROCESSNAME - Allow an FSD to canonicalize a file name to its own particular conventions

FS_SETSWAP - Designate a partition as the swap partition

FS_INIT - Initialize the FSD

FS_SHUTDOWN - Notify the FSD that the system is shutting down

FS_EXIT - Notify the FSD that a process is ending

Swapper Entry Points

FS_ALLOCATEPAGESPACE - Allocate space in the swapper file

FS_DOPAGEIO - Read or write page(s) to the swapper file

FS_OPENPAGEFILE - Open the swapper file

File Locking Entry Points

FS_CANCELLOCKREQUEST - Cancels a lock request

FS_FILELOCKS - Lock or unlock a region of a file

UNC Entry Point FS_VERIFYUNCNAME - Determine if FSD provides access to a specific UNC server
Table 2 - File System Helpers
FSH_ADDSHARE Add a name to the sharing set
FSH_CALLDRIVER Call a PDD's extended strategy entry point
FSH_CANONICALIZE Convert a filename to canonical form
FSH_CHECKEANAME Verify correctness of an EA name
FSH_CRITERROR Signal a hard error
FSH_DEVIOCTL Send a PDD an IOCTL request
FSH_DOVOLIO Call a PDD to transfer sectors
FSH_FINDCHAR Find first occurrence of a character in a string
FSH_FINDDUPHVPB Locates equivalent hVPBs
FSH_FORCENOSWAP Forces a segment permanently into memory
FSH_GETPRIORITY Get current thread's priority
FSH_GETVOLPARM Get data for current volume
FSH_INTERR Signal an Internal Processing Error
FSH_IOBOOST Give the current thread an I/O boost
FSH_IOSEMCLEAR Clear an I/O event semaphore
FSH_ISCURDIRPREFIX Check if current directory is a prefix of a file name
FSH_LOADCHAR Load character from a string (NLS)
FSH_NAMEFROMSFN Retrieve a file's name from its file handle number
FSH_PREVCHAR Move one character backwards in a string (NLS)
FSH_PROBEBUF Verify a region of memory for accessibility
FSH_QSYSINFO Retrieve system information
FSH_REGISTERPERFCTRS Register the IFS with PERFVIEW
FSH_REMOVESHARE Remove a name from the sharing set
FSH_SEGALLOC Allocate a segment
FSH_SEGFREE Deallocate a segment
FSH_SEGREALLOC Reallocate a segment
FSH_SEMCLEAR Clear a semaphore
FSH_SEMREQUEST Request a semaphore
FSH_SEMSET Set a semaphore
FSH_SEMSETWAIT Set a semaphore and wait on it
FSH_SEMWAIT Wait on a semaphore
FSH_SETVOLUME Force a volume to be mounted on a drive
FSH_STORECHAR Store a character in a string (NLS)
FSH_UPPERCASE Uppercase a string (NLS)
FSH_WILDMATCH Match a filename using wildcards
FSH_YIELD Yield the CPU to higher priority threads

Terminology

EMX/GCC
Can be retrieved from the official OS/2 anonymous FTP site, ftp-os2.nmsu.edu, in the directory /pub/os2/2.x/unix/gnu/emx-0.8f. An installation guide for the system can be found in the first issue of EDM/2.
DevHlps
A set of calls available to IFSs and device drivers to perform common functions, such as allocating memory.
Doscalls
Any of the DosXxx calls available in OS/2, such as DosOpen, DosRead, etc.
FSD (File System Driver)
A system extension that allows OS/2 to access files stored in different disk formats or across a network. Also called an IFS.
FSH (File System Helpers)
A set of calls available to IFSs that perform common functions, such as allocating memory and managing semaphores.
GDT (Global Descriptor Table)
The table of selectors that are valid across all processes.
IFS (Installable File System)
A system extension that allows OS/2 to access files stored in different disk formats or across a network. Also called an FSD.
IPE (Internal Processing Error)
An unrecoverable system halt caused by buggy ring 0 code.
LDT (Local Descriptor Table)
The table of selectors that is allocated on a per process basis. Every process has its own LDT. A particular LDT selector is valid only in its own process.
NFS (Network File System)
A protocol developed by Sun Microsystems to allow UNIX machines to share files across a network.
UNC (Universal Naming Convention)
The '\\server\path\file' naming convention, where 'server' refers to the server the file is on.
VMS
DEC's proprietary operating system for its VAX series of computers. One of the nice features it supports is version levels for files (sort of an automatic backup).