Re: Can Coco/R do multiple tokenizations

Gene Wirchenko <>
16 Aug 2005 11:18:10 -0400

          From comp.compilers

Related articles
Can Coco/R do multiple tokenizations (2005-08-13)
Re: Can Coco/R do multiple tokenizations (George Neuner) (2005-08-16)
Re: Can Coco/R do multiple tokenizations (Hans-Peter Diettrich) (2005-08-16)
Re: Can Coco/R do multiple tokenizations (Gene Wirchenko) (2005-08-16)
Re: Can Coco/R do multiple tokenizations (Chris F Clark) (2005-08-21)
Re: Can Coco/R do multiple tokenizations (Darius Blasband) (2005-08-21)
| List of all articles for this month |

From: Gene Wirchenko <>
Newsgroups: comp.compilers
Date: 16 Aug 2005 11:18:10 -0400
Organization: Compilers Central
References: 05-08-053
Keywords: lex
Posted-Date: 16 Aug 2005 11:18:10 EDT wrote:

> Consider a langauage, which allows ! and = in its identifiers.
> Of course usual C operators like !,= etc are also allowed.
> Consder this string (note no whitespaces ):
> 'a!=b'
> Valid tokenization/parsing can yield several posibbilityes
> 1. 'a!=b' .. a single token.
> 2. 'a!' '=' 'b' .. an assignment
> 3. 'a' '!=' 'b' .. an comparision.
> The accpetenace of a particular choice is influenced by
> 1. has a token already been defined (if 'a' and 'b' have been
> defined, than (3) gets priority )

          What if a!b or a! have also been defined? In the latter case,
how does one assign the value of b to a!? (This is assuming that the
= operator in your language assigns right to left. (If you are going
to mess with other quasi-conventions, you might well mess with this.)

> 2. what are some tokens following this or preceding this string.
> .. if preceded by 'z =', than (3) can be omitted,
> .. if followed by '= z' then (1) is more probable.

          I assume the spaces are significant. If not, there is even more

> In case of ambiguity I'ld idealy like to generate error and abort.

          I know I would, at the language selection stage. It would
require force to get me to use a language like this. There is simply
too much room for error. C has
                    if (a=b)
This is notably worse.

> Now can Coco/R, (can any other parser/lexer generator ) can do
> multiple tokenizations & parser-tree-generations , so that I can
>give a priotity to each of these three, and it can call my function
>to accept one over others.


Gene Wirchenko

Computerese Irregular Verb Conjugation:
          I have preferences.
          You have biases.
          He/She has prejudices.

Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.