Re: C scanners, was Hand written or tool generated lexical analyzers for FORTRAN

Russ Cox <rsc@swtch.com>
30 Sep 2005 02:01:31 -0400

          From comp.compilers
Related articles
Hand written or tool generated lexical analyzers for FORTRAN pankaj.jangid@gmail.com (Pankaj) (2005-09-14)
Re: Hand written or tool generated lexical analyzers for FORTRAN gah@ugcs.caltech.edu (glen herrmannsfeldt) (2005-09-15)
Re: Hand written or tool generated lexical analyzers for FORTRAN fjscipio@rochester.rr.com (Fred J. Scipione) (2005-09-17)
Re: Hand written or tool generated lexical analyzers for FORTRAN pankaj.jangid@gmail.com (Pankaj) (2005-09-27)
Re: C scanners, was Hand written or tool generated lexical analyzers f rsc@swtch.com (Russ Cox) (2005-09-30)
Re: C scanners, was Hand written or tool generated lexical analyzers f nmm1@cus.cam.ac.uk (2005-10-02)
Re: C scanners, was Hand written or tool generated lexical analyzers f vesa.karvonen@cs.helsinki.fi (Vesa Karvonen) (2005-10-13)
| List of all articles for this month |
From: Russ Cox <rsc@swtch.com>
Newsgroups: comp.compilers
Date: 30 Sep 2005 02:01:31 -0400
Organization: Compilers Central
References: 05-09-054 05-09-061 05-09-069 05-09-127
Keywords: lex
Posted-Date: 30 Sep 2005 02:01:31 EDT

> How about an idea where lexer is divided into two layers. First is a
> scanner which takes character input and returns semi-classified tokens
> but the second stage, the lexer, will completely resolve the remaining
> tokens. i.e. second stage will classify keywords and identifiers.


This is exactly what C compilers with built-in preprocessors are
already forced to do. Since the preprocessor treats all words the
same, regardless of whether they are reserved words, type names, or
identifiers, it does what you call first stage lexing. Once the
preprocessor is finished, the compiler proper handles your second
stage lexing (with some help from the parser).


Russ


Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.