Re: Why separate the lexer and parser?

"Mark Hopkins" <mark@omnifest.uwm.edu> writes:
            Generally you'll keep the lexical scanner separate in order to
      modularise error-handling. Lexical errors are trapped and handled before
      the rest of the processor has a chance to see it, so the syntax analyser
      only needs to deal with a clean and consistent interface. In most
      imperative programming languages you can pretty much do the same thing
      with the syntax for expressions, for declarations and for statements.
      You'll run into a few cases like in C where labeled statements have to be
      distinguished from assignment statements (you have to look ahead), but
      that's no major deal.


            Otherwise there's no real reason to keep the separate. A syntax is
      just a syntax.
      ...


Back when I was young and foolish, and slightly less experienced, I wanted
to write a compiler. I had been forced to use separate lexer and parser
phases in a compilers course, so decided that I wasn't going to for this
new project. It was a sort-of Algol-like language, emitting code for the
IBM 370. I was able to make it work, and it wasn't all that bad. However,
it would be quite difficult to change the language without fiddling little
things all over the place, and it would be quite difficult to go into the
sources (in AlgolW) and figure them out. The real result of that
"experiment" was that I convinced myself that separating lexical analysis
from parsing is a good idea. I've written several compilers and
interpreters since then, and every one of them has had a separate lexical
analysis phase. Of couse, I have yet to learn the "lesson" of not doing
everything myself - I have never used either Lex or Yacc, and don't plan to.
--
Chris Gray cg@myrias.ab.ca [C macros - a powerful way to make bugs!]
Myrias Computer Technologies Inc. (403) 463-1337
--

Return to the comp.compilers page.
Search the comp.compilers archives again.