A Review of C++ Compilers
Written by Gordon Zeglinski
- 1 Introduction
- 2 About the Tests
- 3 Compiler Overview
- 4 Summary of Features
- 5 On with the Benchmarks
- 6 Closing Remarks
- 7 Editor's Notes
- 8 Precompiled Headers
- 9 Options used
This article, examines various real-world aspects of the primary C++ compilers available for OS/2. The compilers included in this review are:
- Borland C++ for OS/2 version 1.0
- Watcom C/C++ version 9.5
- IBM C Set++ for OS/2 version 2.0
- EMX/gcc version 0.8g
I have omitted Zortech C++ from this list because I do not have access to this compiler. The last time I looked at this compiler, it did not include an OS/2 debugger.
These compilers will be compared on the bases of:
- Time to compile the test code
- Execution time of the test code
- Size of the resultant .EXE file
- Quality of bundled tools
- Bugs found
About the Tests
The test code consists of a series of matrix objects developed by the author. These objects rely on both floating point operations and list manipulations (integer math, pointer dereferencing, etc.). The object hierarchy makes extensive use of virtual, pure virtual, and inline "functions". "Functions" in this case refer to both operators and functions.
In this section, I will examine the non-quantitative aspects of the various compilers, i.e. how easy they are to install, ease of use, and quality of the support tools. Given this, one should keep in mind that the following discussion is subjective and, as such, is the opinion of the author.
Borland C++ for OS/2 version 1.0
This compiler is very similar to its windows counter part. Its IDE uses the same basic design and the windows version. The package comes with both a command line and PM-based compiler, a debugger, and documents on OS/2 programming in .INF format. Also, various on-line documents are provided to help use the the tools.
I have found several bugs in the compiler. The following list is by no means extensive.
- The editor in the IDE has a tendency to drop line feed characters every so often this leads to the occasional hard to find syntax error.
- None of the PM functions are accessible using the <CTRL><F1> help hotkeys from within the IDE.
- The compiler has problems filling in virtual function tables under certain instances. It leaves them as NULL. The results of calling a function in this state is a protection violation error.
- The debugger is next to unusable in my opinion. When debugging C++ programs (especially PM programs), one can expect to reboot their systems very often. This is due to OS/2's single threaded message queue and the inability to kill the debugger via repeated 's.
- TLINK is unable to properly handle segments with IOPL permission, resulting is a protection violation. There was a work around for this problem, but I have not tried it because bug #3 prevents me from using this compiler in the application that needs IOPL priviledges.
The Resource Workshop that ships with this version is similar to its Windows counterpart but lack much of its nice features. For instance, when editing menus one is simply typing a resource script into a large entry field control. The only resources that are edited visually are the dialog boxes, bitmaps and icons; however, the icons no longer work under OS/2 2.1 and the bitmap editor has some bugs in it that cause stray pixels to appear. The dialog editor is great, though, and is far better than the one IBM supplies with C Set++.
This version lacks OWL and the profiler which ships with its DOS/Windows counterpart. On the positive side, this is a fast compiler. It compiles source files faster than any other compiler tested. Also, it seems that there has been some bugs fixed in its optimizer because code that would bomb under the DOS compiler when optimized now works under the OS/2 compiler.
Tech support from Borland is available through Compuserve. I have heard that they no longer provide tech support to Internet mail addresses but cannot say if this is true. Past experience with them was a little disappointing. If you report a bug that has been fixed, they will tell you about a patch if it exists. Bug fixes integrated into the product, though, are usually done in a future version, which you have to purchase to get these fixes.
Watcom C/C++ version 9.5
The Watcom compiler ships with a modified version of the OS/2 Toolkit version 2.0. It generates code for 32 bit DOS, Windows (with a DOS extender), NT and OS/2. If multi-platform code generation is a must for you, then this is the compiler you want. It includes a text mode debugger and profiler. Another plus for this package is that it is capable of optimizing for the Pentium processor. Also included in the package is quite the hefty stack of documentation. However, in order to program in any of the above environments, one still has to buy the appropriate programming guides for the target operating systems. Shipped documentation includes:
- WATCOM C/C++32 Optimizing User's Guide
- WATCOM C/C++ Tools User's Guide
- WATCOM VIDEO User's Guide
- WATCOM VIDEO User's Guide Addendum
- WATCOM Linker User's Guide
- Supplement to WATCOM Linker User's Guide
- WATCOM C Library Reference
- WATCOM C++ Class Library Reference
- WATCOM C++ Container Class Library Reference
- WATCOM C Graphics Library Reference
- The C++ Programming Language
- WATCOM C Language Reference
- WATCOM C/C++32 Commonly Asked Questions & Answers
Although being a VIO mode app, the debugger is quite powerful. In addition to the typical features associated with a source-level debugger, it also allows debugger instructions to be executed at break points. However, there does not seem to be any C++ specific functionality, e.g. class browsers.
The profiler included is also a VIO mode application and is divided into two separate parts: the sampler and sample displayer. The sampler uses the system clock to periodically interrupt the executing program and see where it is. It also allows the programmer to set "profile points" or "marks" within the source code, allowing it to be used as either an intrusive or non-intrusive profiler. Because the profiler does not use any device drivers, it is likely that under OS/2 the sample rate is only approximately 32 Hz, due to the fact that the profiler can only use the standard API calls to gain access to the timer interrupts.
At any rate, the non-intrusiveness of the profiler is a negative point when it comes to profiling existing C++ code since in any OOP language, it is common to have many small functions that are frequently called. The only way to determine the impact of these functions is to place marks around them. For large amounts of code, this is undesirable because it has to be done manually. The sample displayer is pretty basic. It only gathers and displays time and frequency related data.
One of the things which bothers me the most about the product is that the Watcom linker does not use standard .DEF files.
In the few days that I have tested this package, the following bugs were found:
- The compiler has problems dealing with the construct virtual operator=. I was able to work around this bug so that I could perform the benchmarks.
- I could not get it to work properly with the Workframe version 1.1. This is probably due to the fact that I'm using the beta version of the Workframe from the last PDK until my copy of C Set++ arrives.
Having found what I consider to be a serious bug (#1 above), I decided to give their tech support a try, provided through email. My first bug report was answered within 2 hours and subsequent email were all answered the same day. One day after sending them the code, I got a reply. Unfortunately for me, there is a grey area in the C++ "standards" that revolve around the use of virtual operator=(). Watcom follows the approach taken in MSC 7.0 which is different than just about every other C++ compiler. Watcom will be bringing this area to the attention of the ANSI C++ standards committee.
IBM C Set++ version 2.0
Now that my copy of the GA version of C Set++ has arrived, I can compare it to the release version of the others and to the beta version some of you may have. The package includes the OS/2 toolkit, IBM's Workframe, and the CSet++ compiler. I'll concentrate on the compiler and it's support tools.
The comparison of the GA to the BETA version of this compiler can be summarized in one sentence: the GA version compiles code about 2-3 times faster and it's tools are far more stable.
One of the most important things (for some of my applications) about this compiler and the accompanying linker is that it is capable of interfacing with IOPL code segments. I have not tested Watcom's ability in this area. I also believe that the EMX package has input/output functions. Like Watcom, this compiler is compatible with version 3.0 of the C++ specifications, meaning that they support exceptions among other things. The Borland compiler does not and I'm not sure about the EMX package.
The debugger is PM-based and has object browsing abilities built into it. It also has features which are specific to debugging PM-based programs (ie. monitoring message queues and such). My biggest problem with any PM-based debugger is that, if it is buggy, it can hang both the system queue and itself such that a reset is required. It does not appear to have the ability to program actions at breakpoints like the Watcom compiler does.
After much use, the debugger performed almost flawlessly. I have found that it sometimes kills itself when the program monitor (a variable browser) is packed with variables. The best feature it has is its ability to save all of the settings and restore them when you resume debugging. These settings include breakpoints, user defined messages, etc. I have found a minor bug in its user defined messages - it did not properly display the message parameters after restarting the debugger although it did remember the messages I defined.
The profiler is excellent, feature wise, and is PM-based. It is an intrusive analyzer that requires one to compile the code with both debugging info and profiling hooks turned on. The programmer can also insert profiling points within the code. The profiler is a full featured execution analyzer capable of measuring time, the number of times a profiler hook was called, and displaying a call tree. The time and frequency data can be displayed in a number of formats. The call tree displays which functions called who and the number of times these calls were made. It has problems, however; I have found that one of its display options will not function with a trace file I generated. Even worse is that some of its display modules do not support long filenames, which is unacceptable.
The object browser is also quite useful; it allows a graphical and textual exploration of the classes used in the program. It displays the relationship between these objects by showing inheritance, who calls who, and class members.
I have not used the PM object library that comes with this compiler because I am creating my own library. Others have complained that there is a lot of overhead in using this library, though, and that it takes along time to compile code that uses it.
My biggest complaint about this product is its so-called documentation. I bought the 3.5" format package hoping to find tons of hard copy manuals. To my surprise, the hard copy documentation is very similar in size to the documentation that came with OS/2 2.0 GA. In several places, the hard copy documentation refers the reader to the on-line help. The only hard copy manuals with some thickness to them are:
- Class Libraries Reference Summary
- User Interface Class Library User's Guide (an IBM Red Book)
- Programming Guide
The rest of the manuals seem to have trivial content; if you were really stuck, you would typically be referred to the on-line help for the product or some other on-line file. There are also a few typo's in the manuals.
Latest Bug Fixes
My complaints about the lack of long filename support in EXTRA (the profiler) have been solved by applying CSD level 0001 to the package and then running the following little REXX program, which sets a flag in the EXTRA executables that allow them to see files with long names.
Note: This REXX file assumes that the program EXEHDR.EXE is on the path (it is included with the toolkit).
/* This exec addes the "newfiles" flag to the header of each of the Extra executables. This allows the user to use long names for trace files */ "EXEHDR /NEWFILES IXTRA.EXE" "EXEHDR /NEWFILES ITIME.EXE" "EXEHDR /NEWFILES IEXCDENS.EXE" "EXEHDR /NEWFILES ICALNEST.EXE" "EXEHDR /NEWFILES ISTATS.EXE" "EXEHDR /NEWFILES IDCGRAPH.EXE"
EMX/gcc version 0.8g
The EMX/GNU compiler package is a very impressive freeware compiler. The package includes the GNU gcc version 2.4.5 compiler and a debugger. A bunch of other Unix-like tools are also included. The debugger is a VIO program which is command line driven. I haven't used it for PM programming or interfacing with OS/2's API. The docs for the package are available via ftp from the primary OS/2 FTP sites. The reader can get these documents for themselves if they so desire.
Summary of Features
|Borland C++ for OS/2||Watcom C/C++||IBM CSet++||EMX/gcc|
|Multi-Platform||No||32 bit OS/2,DOS, Windows, NT||No||OS/2, DOS|
|Tech Support||compuserve||Internet email||Internet email||No official support|
On with the Benchmarks
The benchmarking was performed on a 486 DX/33-based machine with 16M of RAM. The test consist of two programs both are related to my work on object oriented matrix classes. One test (the full matrix) is mostly floating point while the other (the sparse matrix) is a combination of list manipulation and floating point. Listed are the times it took to compile the two first test both with and without optimizations enabled is measured, the times it took to execute the each test program, and the size of the resulting executables.
Note: in the following charts, (opt) means that optimization was turned on
These times are measured with a stopwatch and represent the time it took NMAKE or MAKE (in the case of EMX) to create the executable. The source code consists of approximately 11 files for each test.
Note: For each table below, all times are in seconds and all sizes are in bytes.
|Borland||Watcom||IBM||EMX/gcc||Borland (opt)||Watcom (opt)||IBM (opt)||EMX/gcc (opt)|
|Borland||Watcom||IBM||EMX/gcc||Borland (opt)||Watcom (opt)||IBM (opt)||EMX/gcc (opt)|
The full test measures the time to LU decompose a 200 x 200 matrix. The sparse test measures the time to LU decompose and solve a 800 x 800 sparse matrix.
|Borland||Watcom||IBM||EMX/gcc||Borland (opt)||Watcom (opt)||IBM (opt)||EMX/gcc (opt)|
I've noticed that while running the sparse test, the Watcom compiler required about 8M of RAM. All the other compilers produced executables that only required about 2M of RAM when executed, which requires further investigation. All of the compilers except Borland performed equally well in the sparse test.
When I began this series of tests, I was hoping that there would be a clear "winner." However, I do not think that one exists; each of the products have their own unique qualities which will appeal to different users. I strongly feel that for people looking simply for an OS/2 C++ compiler with excellent tools, CSet++ is the right choice. For others, I hope the guidelines below will help.
As a side note, I am very impressed with the EMX/GCC package. It is as good as any of the others and costs nothing. If I didn't need a profiler and want precompiled header files, I'd definetly save some cash and use it.
|Interfacing with IOPL code||CSet++, Watcom C/C++ or EMX/GCC(?)|
|Floating Point intensive code||EMX/GCC, Watcom C/C++, or CSet++|
|PM programming||CSet++ or Borland (because of precompiled header files). Any of the 4 are quite able in this area.|
|C++ code with working virtual functions||EMX/GCC or CSet++ (Watcom follows the MSC way of handling the operator=() )|
|Profiler||CSet++ or Watcom C/C++|
|Environment like BC 3.1||Borland C++ for OS/2|
|Really fast compilation||Borland C++ or CSet++ (CSet++ is slower but it's debugger is better)|
After mentioning in comp.os.os2.programmer.misc that this issue would contain a comparison of C++ compilers, I received a request from Tim Francis in the C-Set++ compiler group to review the article pre-press. I gladly sent him a copy with the stipulation that the article would not be modified based on any comments he made, to which he agreed. However, I feel it necessary to put these comments here, as a side-note, with the intent of demonstrating the apparent customer-driven attitude of this group. As a disclaimer, while I do know some people on the compiler development team, I have never had any business dealings with them and am not doing this as a favor to them or because of any bias I have.
Tim Francis writes:
I didn't find anything really wrong in the compiler review. I thought C-Set++ came out looking quite good, actually. I would be interested in seeing the actual code used in the execution benchmarks - we were last in the opt/full matrix test, and 2nd last in the opt/sparse matrix test. I'm certainly not trying to dispute these numbers, but our benchmarks tend to place us a little higher in the competition than that. Our performance evaluation guy would really like a look at what's happening, to see if we can improve anything.
The only other comment I have is in the summary of features, Tech support column. As documented in the C-Set++ package, we offer the following support:
- Non-defect support, and informal (best effort) defect support
- Formal defect support
- 1-800 number, available 24hrs/day, 7 days/week.
Obviously we feel that the support we offer is a key component of the product, so if you mention the above somewhere in EDM/2 I'd appreciate it.
For the purposes of completeness, I am also including the following from Ian Ameline:
If possible, could you have those floating point tests run with /Gf+ turned on - it will result in *much* faster FP code by relaxing our strict interpretation of the IEEE standard. The other compilers all use the more relaxed interpretation - and this places us at a bit of a performance disadvantage compared to them, but we do produce results that are the same as any other IEEE 64 bit FP processor (Of course the Intel one is uses 80 bits of precision naturally, and if we try to conform to the 64 bit standards, we have to truncate the numbers each time they're stored to a variable. This truncation is expensive)
Also, I'll make sure the long filename bug in Extra is fixed for CSD 2.
Ian's comment about the compiler's strict interpretation of IEEE standards prompted me to request that the author use the /Gf+ option for the optimized part of the benchmarks (he was using /Gf); he did so and reported that /Gf and /Gf+ resulted in no difference in time (it appears that /Gf invokes the default which is /Gf+), while /Gf- resulted in an time increase of 1.5 seconds.
Precompiled headers are very useful when including OS2.H. Precompiled header files can greatly decrease the amount of time necessary to compile files that include many and/or large header files. However, not all compilers are equal in this respect. Borland compiles all headers into one large file. This probably make it a bit faster to access than multiple files. But, when one header changes, all headers must be recompiled and stored in the large file. IBM's CSet++ on the other hand uses one precompiled file for each header. Thus, when you change one of your header files, only that file has to be recompiled. Also, the method employed by CSet++ allows the same precompiled headers to be used by multiple projects, unlike the method used by Borland.
The following command line options were used with Borland's compiler:
-I.. -If:\bcos2\include -ff -G -4 -O2it -vi -c -D_USE_POST_FIX_
The following command line options were used with IBM's compiler:
/I.. /Tdp /J- /Si- /O+ /Oi+ /Os+ /W1 /Gf /Gi /G4 /Gx+ /C /d_USE_POST_FIX_
The following command line options were used with Watcom's compiler:
/i=.. /i=. /i=f:\watcom\H /i=f:\TOOLKIT\C\OS2H /mf /4r /bt=os2 /sg /d_USE_POST_FIX_ /oneatx /zp4
The following command line options were used with GNU's EMX compiler:
-c -O2 -m486 -I..