Re: Expected Token Density in Random Stream

Andrew Tomazos <andrew@tomazos.com>
Tue, 13 Dec 2011 06:00:40 -0800 (PST)

          From comp.compilers
Related articles
Expected Token Density in Random Stream andrew@tomazos.com (Andrew Tomazos) (2011-12-07)
Re: Expected Token Density in Random Stream kaz@kylheku.com (Kaz Kylheku) (2011-12-11)
Re: Expected Token Density in Random Stream andrew@tomazos.com (Andrew Tomazos) (2011-12-13)
Re: Expected Token Density in Random Stream gene.ressler@gmail.com (Gene) (2011-12-19)
| List of all articles for this month |
From: Andrew Tomazos <andrew@tomazos.com>
Newsgroups: comp.compilers
Date: Tue, 13 Dec 2011 06:00:40 -0800 (PST)
Organization: Compilers Central
References: 11-12-015 11-12-016
Keywords: parse, theory
Posted-Date: 14 Dec 2011 13:28:26 EST

On Dec 11, 6:56 pm, Kaz Kylheku <k...@kylheku.com> wrote:
> On 2011-12-07, Andrew Tomazos <and...@tomazos.com> wrote:
>
> > Summary: We want to find out how often a given token appears in a
> > random stream formed by concatenating randomly chosen strings from a
> > given set of strings.
> > (Note hits can overlap each other)
>
> But tokens do not overlap, so you're not actually extracting tokens. Using
> C tokens as an example, the C token >>= is one hit, not four. The longest
> match calls for extracting three characters and moving on.


Substitute occurrences of the word "token" in my post for "key
string" (or just "string") and reinterpret.
    -Andrew.
[I suppose, but finding tokens would be more interesting. -John]


Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.