View source for Adding syntax highlighting to EPM

''Written by [[Paul Floyd]]''

==Introduction==
[Note: a zip file is available with [http://www.edm2.com/0605/epm.zip the source] for this article. Ed]

In this article I'll cover adding automatic syntax generation and/or syntax colouring to EPM 6, IBM's programmable 32bit PM text editor. The language I'll use is Oberon, a descendant of Pascal and Modula.

==Requirements==
Firstly, a word about requirements. For all of this functionality, you need EPM 6.03b. I also recommend [[CustEPM]], but the version adapted for EPM 6 I have only ever seen on DevCon - the EPM 5.5(1) seems to be freely available on IBM FTP sites. You will need the EPM macro source and ETPM the e macro language compiler.

==Syntax Assist==
EPM 6.03b comes with syntax assist for C (but not C++), Pascal, Rexx, and E (the macro language EPM uses itself). Some people find syntax assist annoying, but I quite like the consistency that it tends to give to code I write.

Syntax assist can be turned on and off manually by issuing the "expand on/off" command in the EPM command dialog box (ESC or ctrl-l), or more permanently using either profile.erx or the mycnf.e (by preference) e macro file. Ctrl-x forces an expansion from the keyboard.

The functionality is implemented in the compiled "e" files. The language is quite easy, in many ways similar to Rexx. The source code that IBM ships is a bit hairy, having a great number of conditional compilation statements (compile if EVERSION > 4 sort of thing). There are a few conditional compilation flags that control whether the functionality is present. By default they go in your mycnf.e file, and take the form:
 C_SYNTAX_ASSIST = 1.

The syntax assist is based on "triggers", the first trigger is the space bar. So when the space bar is pressed, a "first expand" function gets called. If the preceding word was "for", for instance in a C file, then the first expand routine will add "( ; ; )" on the same line, and braces on following lines. The second trigger key is the enter key. Using the same example, the enter key in a C "for" expansion will take you from the first field of the "for" expression to the second, then the third, and then into the body of the statement.

==Warning - Limitation==
One problem with the syntax assist is that it isn't at all smart, and does no parsing of the context of the lines it adds. Sometimes this is not a problem - as far as I'm aware, there's nothing wrong with the C syntax generated. However, there is a little catch with the Oberon (and existing Pascal) assist. For all the statements that are generated, you can't be sure if it is the last within that compound statement or not. So the syntax assist always terminates statements with a semicolon. For C that's fine because the semicolon is the statement terminator. But for Pascal/Oberon, it is the statement separator. For instance,
<code>
 BEGIN
    IF (x == 1) THEN
       y := 2;
       z := x/y
    END; (* end if *) <= semicolon here is OK, it separates this
                         compound IF statement from the following FOR
 
    FOR a := 1 TO 10 DO
       (* do something with a *)
    END; (* end for *) <= semicolon here is unnecessary and could mask
                          problems
 END
</code>
In the above example, I haven't declared the variables used, and the BEGIN and END are just to show the scope of a compound statement. Also in this case, the syntax assist for the IF would have generated an "ELSE" (which I delete if I'm not going to use it).

==Syntax Colouring==
As well as syntax assist (which also existed for versions of the editor back into the mists of time - 10 years at least), EPM 6 has syntax colouring. This is only line by line analysis, it does not perform full parsing like editors such as [[LPEX]] and Emacs. EPM 6 comes with syntax colouring support for a greater range of languages: C (including C++), Rexx, FORTRAN 90, HTML, IPF, Java (though linked to the C/C++ support), Makefiles, RC files, Perl, TeX, and IBM Bookmaster.

Each language has its own syntax colouring file. The C/C++ file includes keywords for much of the OS/2 API (for the 2.1 Toolkit, as far as I know not a great deal has been added since, but making it completely up to date would be a considerable task) and the runtime C library. As with syntax assist, colouring can be controlled by conditional compile flags, like:
 C_KEYWORD_HIGHLIGHTING = 1

==Adding a New Language==
First you have to decide whether you want to have syntax assist or not. If you do I suggest trying to base your syntax assist macros on ones that exist already. For the following example, I used the Pascal assist as the basis, copying pkeys.e to okeys.e, and modifying where necessary.

The operations you need to perform are pretty straightforward. For each keyword you want to define syntax assist for, you have to put an entry in your "first expand" function. You may also want "second expand" functionality for that keyword, but it is not obligatory. Note however you can't have second expansion if you didn't add first expansion. I'll give as an example the "FOR" expansion in Oberon. The syntax looks like this
<code>
      FOR x := 1 TO 100 DO
           statements...
      END
 
         w=line ; the current line
         wrd=upcase(w) ; convert current line to upper case
         if wrd='FOR' then ; check to see if we are in a FOR expansion
            replaceline w' :=  TO  DO' ; yes : expand current line
   compile if WANT_END_COMMENTED
            ; if commented end required, add it
            insertline substr(wrd,1,length(wrd)-3)'END; (* endfor *)',.line+1
   compile else
            ; otherwise add uncommented end
            insertline substr(wrd,1,length(wrd)-3)'END;',.line+1
   compile endif
            ; I think this resets the cursor state and position
            if not insert_state() then insert_toggle
   compile if EVERSION > '5.50'
                call fixup_cursor()
   compile endif
            endif
            ; inserts the space
            keyin ' '
</code>
A few important e functions and variables used in the above example:
<code>
  .line = line number of cursor
  insertline inserts a new line at the given position
  replaceline replaces the line at the given position (current line if no
               line number specified)
</code>
Adding syntax colouring is relatively straightforward. It uses keyword files that define the mapping between keywords and colours. It can also cope with comments on a single line (but not spanning multiple lines). Again, I would start with copying one of the epmkwds.* files. You will need to decide if you want OS/2 keywords (functions and constants) to be included. They are all in the epmkwds.c file. The drawback of having all these keywords is that it greatly slows down the processing of files.

The format of the keywords file is documented in the "C" keywords file. The very first character acts as a comment delimiter, so choose one that is not used in your language. The following structures are supported:
<code>
 DELIM - delimiters (comments, strings)
 DELIMI - case insensitive delimiters
 KEYWORDS
 INSENSITIVE - case insensitive keywords
 CHARSET - characters in KEYWORDS and INSENSITIVE that are affected
 SPECIAL - as KEYWORDS/INSENSITIVE, but for all characters
 SPECIALI - case insensitive SPECIAL
 BREAKCHAR - allows breaks within KEYWORDS/INSENSITIVE
 ENDCHAR - as BREAKCHAR, but uncoloured character is the one after this
</code>
Here is an example:
<code>
 @KEYWORDS
 @
 @ -------------------- Oberon types ---------------------------
 BOOLEAN         -1      01
 CHAR            -1      01
 @ -------------------- Oberon statements ---------------------------
 IMPORT          -1      01
 BEGIN           -1      01
 END             -1      01
</code>
The first column is the foreground, the second the background (blue in this case). For DELIM, it is a bit more complicated:
<code>
 @DELIM
 @
 @ Start   Color Color  End     Escape
 @ string  bg    fg     string  character
   (*       -1     03   *)
   "        -1     05   "       \
</code>
(for cyan comments (* *) and magenta strings in double quotes).

Colours are as follows in EPM:
<code>
 BLACK          = 00  DARK_GREY      = 08
 BLUE           = 01  LIGHT_BLUE     = 09
 GREEN          = 02  LIGHT_GREEN    = 10
 CYAN           = 03  LIGHT_CYAN     = 11
 RED            = 04  LIGHT_RED      = 12
 MAGENTA        = 05  LIGHT_MAGENTA  = 13
 BROWN          = 06  YELLOW         = 14
 LIGHT_GREY     = 07  WHITE          = 15
</code>
OK, so we've covered adding syntax assist and syntax colouring. So far this has been quite simple, largely based on existing code and keywords files. Now we have to get EPM to recognize our types of files when it loads so that it knows when to apply the functions.

==Modifying EPM load routines==
As a rule, most simple changes to EPM macros can be achieved simply by altering the mycnf.e file. However, the changes we need to make require some of the base EPM macro files to be altered. The two files that to be added to are load.e and e.e.

In load.e, there are first of all a number of statements to ensure the conditional compilation flags are set, like:
<code>
 compile if not defined(C_KEYWORD_HIGHLIGHTING)
    C_KEYWORD_HIGHLIGHTING = 0
 compile endif
</code>
I added one for OBERON_KEYWORD_HIGHLIGHTING (which I set to 1 in mycnf.e).

You then have to add the line by line parsing of your syntax colouring based on the file extension of your language. Oberon files end normally with ".mod", but I also added ".ob2". Or together all the possible file extensions in the if test, and your keywords file to the toggle_parse call. Here is the snippet I added in load.e:
<code>
 compile if OBERON_KEYWORD_HIGHLIGHTING and EPM32 and
           not (OBERON_SYNTAX_ASSIST and ALTERNATE_KEYSETS)
    if wordpos(load_ext, 'MOD OB2') & .visible then
       'toggle_parse 1 epmkwds.MOD'
    endif
 compile endif
</code>
Note that the "and not (OBERON_SYNTAX_ASSIST and ALTERNATE_KEYSETS)" clause can be omitted if you do not add syntax assist. You also need to have the same block of code in the syntax assist file (okeys.e in this case) but without the "and not (OBERON_SYNTAX_ASSIST and ALTERNATE_KEYSETS)" condition.

The change required in e.e is simpler: you just have to add make it include your syntax assist macro file, e.g.,
<code>
 compile if OBERON_SYNTAX_ASSIST
   tryinclude 'okeys.e' -- Syntax-assist for Oberon programmers.
 compile endif
</code>
There are a couple of issues I still haven't resolved that are not within syntax assist and colouring. Firstly, there is the tags feature which basically allows you to create a text file summarizing the procedures in a project, and enables you to jump quickly to them. EPM 6 has support for tags with Modula, almost the same as Oberon, but unfortunately, procedure declarations is one of those differences. Oberon-2 has type bound procedures, where the module to which the procedure is bound precedes the function name (similar to scope resolution operators in C++, which also breaks the tag generation of C functions). Succinctly, it doesn't work on Oberon source.

Lastly, there is the folding feature (only in CustEPM as far as I know). I haven't even tried getting this to work on Oberon yet, so it certainly needs looking into.

That's about it. Recompile your e macros (I use the makemacs command that that comes with the source), make sure that the .ex files are in your [[EPMPATH]], and you're ready to go. Sample macros are provided in the file source.zip, but the mycnf.e file is only for example, if you simply copy it, you will also get all of my personal preferences.

In the end you get something that looks like this:

[[Image:epmoberon.gif]]

Of course, my taste in colours isn't likely to be universal!

(include e.e load.e okeys.e mycnf.e and epmkwds.mod as source, okeysel.e is included for backwards compatibility with ancient versions of the editor, and tags.e has some but not all modifications required for tags support).

[[Category:Tools Articles]][[Category:EPM]]