Re: String tokenizer place in Chomsky hierarchy?

Rock Brentwood <markwh04@yahoo.com>
Fri, 25 Apr 2008 12:24:46 -0700 (PDT)

          From comp.compilers

Related articles
String tokenizer place in Chomsky hierarchy? TegiriNenashi@gmail.com (Tegiri Nenashi) (2008-04-08)
Re: String tokenizer place in Chomsky hierarchy? maharri@gmail.com (Mitch) (2008-04-11)
Re: String tokenizer place in Chomsky hierarchy? DrDiettrich1@aol.com (Hans-Peter Diettrich) (2008-04-11)
Re: String tokenizer place in Chomsky hierarchy? sipayi@gmail.com (2008-04-15)
Re: String tokenizer place in Chomsky hierarchy? markwh04@yahoo.com (Rock Brentwood) (2008-04-25)
| List of all articles for this month |

From: Rock Brentwood <markwh04@yahoo.com>
Newsgroups: comp.compilers,comp.theory
Date: Fri, 25 Apr 2008 12:24:46 -0700 (PDT)
Organization: Compilers Central
References: 08-04-030
Keywords: lex, theory
Posted-Date: 25 Apr 2008 23:55:03 EDT

On Apr 8, 12:44 pm, Tegiri Nenashi <TegiriNena...@gmail.com> wrote:
> Formally, a set of terminals is partitioned into a separator or a set
> of separators, and the rest of terminals. Then, string tokenizer
> translates a given word into a set (or list) of words. Here we have
> the first technical difficulty, what exactly this translation is?


The inverse of a monoid homomorphism. The homomorphism it is the
inverse of is the one that maps the monoid of token sequences to the
monoid of character sequences.



Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.