Re: Pondering the future of lexical analysis

> [I know there are Unicode versions of lex, such as the one from plan
> 9.


Not according to the docs: http://www.cs.bell-labs.com/magic/man2html/1/lex
(look under bugs)


> And yes, you only need to store valid transitions. One technique
> is to store the highest and lowest valid tokens in each state and a
> vector of transitions [lowest,highest]. -John]


Another technique, similar to the above, would be to use a 2-level table -
high-order 8 bits followed by low-order 8. If all the transitions (valid or
not) within high-8 are the same, code the transition. If there are
differences, refer to another table to deal with low-8. This has the
advantage of quick lookup (at most 2 array dereferences), but the
disadvantage of not being compatible beyond 16 bits (which is where Unicode
is heading).


--
Scott Nicol
snicol@apk.net

Return to the comp.compilers page.
Search the comp.compilers archives again.