OS/2 Installable File Systems Part 1
Written by Andre Asselin
|Part 1||Part 2||Part 3|
One of the most useful but obscure features of OS/2 is it's support for installable file systems (IFSs). This API allows you to extend OS/2 to do two things: access files stored on disk in formats other than FAT and HPFS, and access files that are stored on a network file server. For example, under many versions of UNIX, files are stored in the Berkeley fast file system format, which isn't compatible with FAT or HPFS. If you have both UNIX and OS/2 installed on the same computer with partitions setup for each, you could write an IFS to read and write the UNIX files while OS/2 is running and give all the programs running under OS/2 (including DOS and Windows programs) access to those files.
The other variety of IFS is called a remote file system or redirector, and allows file sharing over LANs. In this case, when a program calls for a file, the IFS acts as a client and sends the request over a network to a file server. The file server handles the request and sends the data back to the client. Examples of this type of IFS include Novell's Netware requester and the requester part of LAN Server. Sometimes, in addition to just sending the requests to the server, this type of IFS also has to translate between OS/2 file conventions and file conventions that the server understands. For example, the NFS (Network File System) protocol is completely UNIX oriented, so the NFS client (which is an IFS) has to handle UNIX conventions like case sensitive filenames and UNIX file attributes.
Up until now, there hasn't been much written on how to develop an IFS. It also doesn't help that the documentation on the IFS API has been available only by special request from IBM. This series is going to explore the IFS API in depth with working examples in order to take the mystery out of writing them. This first article is a brief overview of the IFS concept to give a feel for what's involved in writing one. In the coming months, I'm going to explore the IFS architecture in depth and build up a toolkit that can be used as a skeleton for writing an IFS.
IFS Structure and How It Fits In
An IFS (which is also known as an FSD for File System Driver) can be thought of as a specialized kind of physical device driver (PDD), although it is structured very differently. Instead of having only two entry points and being based on request packets, it is structured as an ordinary 16-bit DLL with a series of predefined entry points, broken up into four groups, that are exported to the system (see Table 1). The first group consists of the primary entry points that all IFSs are required to export. This group does basic file manipulation, such as opening and closing, and reading and writing files. Most of these entry points correspond pretty closely in name and function to related DosXxx calls. The other three groups of calls provide support for the swapper, file locking, and UNC (Universal Naming Convention). For each of these groups, if your IFS doesn't support that particular feature, you don't have to export any of the functions in that group of entry points. Note that exports should be made on a group-by-group basis; don't export only some of the entry points from a group.
Figure 1 shows where an IFS fits into the OS/2 system.The two shaded components are what make up an IFS. The piece that runs in ring 0 (called MY.IFS in the figure) is the main component. It is called by the OS/2 kernel through one of its FS_XXX entry points whenever something needs to be done on a drive that the IFS is managing. The IFS performs the function and returns the requested data or an error code. Typically, it will interact with a PDD to do its I/O. Like any ordinary DLL, an IFS doesn't have a process or threads of its own. Instead, it operates in the context of the caller's process and on the caller's thread (in the case of an IFS, the caller's thread and process are the thread and process of the application making the Doscall). In this sense, an IFS can be thought of as a passive library that's not activated until a program calls it.
Since an IFS has to be available to every application in the system, it is loaded into global system memory (using GDT selectors). Any memory it allocates (such as for buffers) should also come from global system memory so that no matter which process its entered from, it'll still have access to the memory.
The piece that runs in ring 3 (UMY.DLL in the figure) is a utility library. There is one utility DLL for each IFS; the naming convention is a 'U' followed by the IFS name with an extension of .DLL. It has four entry points; one each for FORMAT, RECOVER, SYS, and CHKDSK. When a user runs any of those programs and it detects that it is being run against a drive being managed by an IFS, it will call the appropriate entry point in the IFS's utility DLL. The DLL is then completely responsible for carrying out the command.
How a Doscall Gets Processed
A typical Doscall (such as DosOpen, DosRead, or DosWrite) is processed mainly by the IFS with some help from the OS/2 kernel (see [Figure 2]). When an application issues a Doscall, such as DosOpen (1), a component of the OS/2 kernel called the file system router does some preliminary validation of the parameters. If everything's OK, it decides which IFS the call should be sent to and calls the IFS at the appropriate entry point, which in the case of DosOpen is FS_OPENCREATE (2). The IFS then looks at the parameters that were passed and decides on a course of action to service the call. In some cases, it may not support the function (for example, trying to open a remote drive for direct, sector-by-sector access), in which case it immediately returns an error code. If it does support the function, it might first check its cache to see if it already has the requested data, in which case it can just return it. If it doesn't have the data in its cache, it will probably have to do some I/O to get it.
In this case, it calls one of OS/2's file system helpers, FSH_DOVOLIO (3), which acts as an intermediary between the IFS and the device driver. FSH_DOVOLIO generates a request packet and calls the strategy entry point of the appropriate device driver (4). The device driver performs the I/O (5), and eventually returns to FSH_DOVOLIO, which returns back to the IFS. The IFS can then examine the return code for errors and take appropriate action. If it needs to, the IFS can the device driver multiple times until it has all the requested data. It then returns back to the OS/2 kernel, which in turn returns back to the application. The application can then examine the return code to see if the operation was successful.
Ring 0 Issues
The type of IFS described above runs completely in ring 0 and can be thought of as a 'traditional' IFS. Unfortunately, it can be difficult to develop because of the restrictions placed on code that runs at ring 0 (PDDs and VDDs suffer from these restrictions as well). There are three main restrictions:
No preemptive multitasking - When an IFS or any other ring 0 code is entered, preemptive multitasking is temporarily disabled. The IFS can still yield to other threads (i.e. cooperative multitasking), but it will never be suspended unless it specifically asks for it. This has the benefit of eliminating some synchronization issues, but it also means that periodic yields have to be sprinkled throughout lengthy operations.
No Doscalls; only file system helpers and DevHlps - None of the ring 3 API's (such as the Doscalls) are available to ring 0 code (except at initialization, which is special). If you use any functions from the C library, you have to make sure that they don't in turn use any of the Doscalls (i.e. malloc and free don't work at ring 0 because they call DosAllocMem). As a partial replacement, there are a series of file system helper (FSH) calls available for IFS's (see Table 2) as well as the standard DevHlp calls available to all device drivers. These APIs perform common tasks such as allocating memory and handling semaphores, but they are more primitive and thus more difficult to use than the Doscalls.
Debugging is a pain - If you thought debugging PM applications was tricky, wait until you try an IFS! First, you won't be able to use your favourite debugger - you have to use a ring 0 debugger, such as the OS/2 kernel debugger or ASDT32. Neither of these can do source level debugging, so you have to know assembler, and they also have limited support for labels. There is also no such thing as a printf in ring 0, so you have to use a hex dump to look at variables. As if that weren't bad enough, OS/2's crash protection doesn't apply to ring 0 code; if your code has a bug and traps, it will bring down the system with an IPE (Internal Processing Error). This is where you get the blue screen, lots of register information, a suggestion to call IBM, etc. Frequent reboots tend to be the way of life.
The Split IFS/Control Program Approach
Because of these restrictions, a popular approach to developing an IFS is to split it between ring 0 and ring 3. The ring 0 piece (the IFS proper) contains only stub routines and passes all the calls up to a ring 3 control program. The control program handles the calls and then passes the results back to the ring 0 piece. The advantages of doing it this way are that developing the bulk of the IFS is easier (because it runs at ring 3, you don't have to worry about those restrictions mentioned above), and you have access to all the normal ring 3 APIs. This approach also allows you to layer an IFS on top of an already existing IFS; for example, DCF/2 and Stacker for OS/2 probably take this approach to use FAT or HPFS to store their compressed data files. This way they don't have to worry about totally rewriting the FAT or HPFS code - all they have to be concerned with is compressing the data on the fly and then using the already existing IFSs to actually store it on the disk.
This type of IFS is usually implemented using captive threads. [Figure 3] traces a call as it winds its way between an application, the IFS, and its control program. When the control program initializes, it creates one or more threads that call into the IFS (1 and 2). The IFS makes these threads block until some application calls the IFS with work to do (3 and 4). The IFS then sets up a parameter area, blocks the application's thread and unblocks the control program's thread. The IFS returns back to the control program (5 and 6), where it executes the request. The control program then puts the results back in the parameter block and calls the IFS again (7 and 8). The IFS then unblocks the application's thread and has it return with the results (9 and 10) while the control program's thread is again blocked waiting for a new request.
Well, that's a high level overview of what's involved in writing an installable file system. In the next issue, I'm going to start developing the framework to implement a split ring 0 / ring 3 IFS. To illustrate how to use it, I'm going to write an IFS that supports file versions (like VMS).
While you're waiting for the next issue, you may want to start gathering the tools necessary to write an IFS. The first thing you'll need, of course, is the documentation on the IFS API and the libraries and header files that go along with it. To get it, you need to send a note on CompuServe requesting the IFS documentation to Mike Abrams (72420,216) in the OS2DF1 forum section 11 (Device Driver Development) with your name and address. If you don't have an account on CompuServe, you can send him a note using the Internet; address it to firstname.lastname@example.org (note the period instead of the comma in the number).
You'll also need two C compilers. For the ring 0 piece, you'll need a 16-bit C compiler. Most IFS developers use Microsoft C 6.0, but I'm going to investigate the possibility of using Borland C++ instead. If I can make it work, that's what I'm going to use, but the code should be compatible with both compilers. For the ring 3 piece, you'll need a 32-bit compiler, such as IBM's C Set++ or Borland C++ for OS/2. Finally, you probably won't need it right away, but you should also get a hold of either the kernel debugger or ASDT32, which is available through IBM's EWS program.
If this series is interesting to you, please drop me a note telling me a little bit about yourself and your background so that I have a better idea who my audience is (it'll also help my motivation if I know that more than Steve and I are reading this series). While you're at it, if you have any feedback or ideas on what you'd like to see covered in future articles, I'd love to hear it. See you next month.
|Primary Entry Points|
FS_OPENCREATE - Open or create a file
FS_CLOSE - Close a file
FS_READ - Read data from a file
FS_WRITE - Write data to a file
FS_CHGFILEPTR - Change the current position in a file
FS_COMMIT - Write all cached data for a file to disk
FS_DELETE - Delete a file
FS_NEWSIZE - Change the size of a file
FS_MOVE - Move or rename a file
FS_COPY - Copy a file
FS_FILEATTRIBUTE - Set or return the standard (DOS) attributes
FS_FILEIO - Perform an atomic read/write/lock operation
FS_FILEINFO - Set or return information on an open file
FS_PATHINFO - Set or return information on a file
FS_ATTACH - Associate or disassociate a drive letter with an FSD
FS_MOUNT - Examine a volume to determine if the FSD knows its format
FS_FSINFO - Get/Set file system information
FS_FLUSHBUF - Write all cached data for a volume to disk
FS_CHDIR - Change directory
FS_MKDIR - Make a new directory
FS_RMDIR - Remove a directory
FS_FINDCLOSE - End a search in progress
FS_FINDFIRST - Begin a new search
FS_FINDFROMNAME - Restart a search from a particular point
FS_FINDNEXT - Return next file name
FS_FINDNOTIFYCLOSE - Stop monitoring a directory for changes
FS_FINDNOTIFYFIRST - Start monitoring a directory file changes
FS_FINDNOTIFYNEXT - Return the next set of changes
|FSD Extended Interface||
FS_FSCTL - Extended interface to an FSD
FS_IOCTL - I/O control for devices
FS_NMPIPE - Support named pipe operations
FS_PROCESSNAME - Allow an FSD to canonicalize a file name to its own particular conventions
FS_SETSWAP - Designate a partition as the swap partition
FS_INIT - Initialize the FSD
FS_SHUTDOWN - Notify the FSD that the system is shutting down
FS_EXIT - Notify the FSD that a process is ending
|Swapper Entry Points||
FS_ALLOCATEPAGESPACE - Allocate space in the swapper file
FS_DOPAGEIO - Read or write page(s) to the swapper file
FS_OPENPAGEFILE - Open the swapper file
|File Locking Entry Points||
FS_CANCELLOCKREQUEST - Cancels a lock request
FS_FILELOCKS - Lock or unlock a region of a file
|UNC Entry Point||FS_VERIFYUNCNAME - Determine if FSD provides access to a specific UNC server|
|FSH_ADDSHARE||Add a name to the sharing set|
|FSH_CALLDRIVER||Call a PDD's extended strategy entry point|
|FSH_CANONICALIZE||Convert a filename to canonical form|
|FSH_CHECKEANAME||Verify correctness of an EA name|
|FSH_CRITERROR||Signal a hard error|
|FSH_DEVIOCTL||Send a PDD an IOCTL request|
|FSH_DOVOLIO||Call a PDD to transfer sectors|
|FSH_FINDCHAR||Find first occurrence of a character in a string|
|FSH_FINDDUPHVPB||Locates equivalent hVPBs|
|FSH_FORCENOSWAP||Forces a segment permanently into memory|
|FSH_GETPRIORITY||Get current thread's priority|
|FSH_GETVOLPARM||Get data for current volume|
|FSH_INTERR||Signal an Internal Processing Error|
|FSH_IOBOOST||Give the current thread an I/O boost|
|FSH_IOSEMCLEAR||Clear an I/O event semaphore|
|FSH_ISCURDIRPREFIX||Check if current directory is a prefix of a file name|
|FSH_LOADCHAR||Load character from a string (NLS)|
|FSH_NAMEFROMSFN||Retrieve a file's name from its file handle number|
|FSH_PREVCHAR||Move one character backwards in a string (NLS)|
|FSH_PROBEBUF||Verify a region of memory for accessibility|
|FSH_QSYSINFO||Retrieve system information|
|FSH_REGISTERPERFCTRS||Register the IFS with PERFVIEW|
|FSH_REMOVESHARE||Remove a name from the sharing set|
|FSH_SEGALLOC||Allocate a segment|
|FSH_SEGFREE||Deallocate a segment|
|FSH_SEGREALLOC||Reallocate a segment|
|FSH_SEMCLEAR||Clear a semaphore|
|FSH_SEMREQUEST||Request a semaphore|
|FSH_SEMSET||Set a semaphore|
|FSH_SEMSETWAIT||Set a semaphore and wait on it|
|FSH_SEMWAIT||Wait on a semaphore|
|FSH_SETVOLUME||Force a volume to be mounted on a drive|
|FSH_STORECHAR||Store a character in a string (NLS)|
|FSH_UPPERCASE||Uppercase a string (NLS)|
|FSH_WILDMATCH||Match a filename using wildcards|
|FSH_YIELD||Yield the CPU to higher priority threads|
- Can be retrieved from the official OS/2 anonymous FTP site, ftp-os2.nmsu.edu, in the directory /pub/os2/2.x/unix/gnu/emx-0.8f. An installation guide for the system can be found in the first issue of EDM/2.
- A set of calls available to IFSs and device drivers to perform common functions, such as allocating memory.
- Any of the DosXxx calls available in OS/2, such as DosOpen, DosRead, etc.
- FSD (File System Driver)
- A system extension that allows OS/2 to access files stored in different disk formats or across a network. Also called an IFS.
- FSH (File System Helpers)
- A set of calls available to IFSs that perform common functions, such as allocating memory and managing semaphores.
- GDT (Global Descriptor Table)
- The table of selectors that are valid across all processes.
- IFS (Installable File System)
- A system extension that allows OS/2 to access files stored in different disk formats or across a network. Also called an FSD.
- IPE (Internal Processing Error)
- An unrecoverable system halt caused by buggy ring 0 code.
- LDT (Local Descriptor Table)
- The table of selectors that is allocated on a per process basis. Every process has its own LDT. A particular LDT selector is valid only in its own process.
- NFS (Network File System)
- A protocol developed by Sun Microsystems to allow UNIX machines to share files across a network.
- UNC (Universal Naming Convention)
- The '\\server\path\file' naming convention, where 'server' refers to the server the file is on.
- DEC's proprietary operating system for its VAX series of computers. One of the nice features it supports is version levels for files (sort of an automatic backup).