Re: flex scanner too huge, suggestions?

Troy Cauble <troy@bell-labs.com>
1 Mar 2001 02:33:52 -0500

          From comp.compilers

Related articles
flex scanner too huge, suggestions? troy@bell-labs.com (Troy Cauble) (2001-02-17)
Re: flex scanner too huge, suggestions? tej@melbpc.org.au (Tim Josling) (2001-02-23)
Re: flex scanner too huge, suggestions? snicol@apk.net (Scott Nicol) (2001-02-23)
Re: flex scanner too huge, suggestions? rkrayhawk@aol.com (2001-02-23)
Re: flex scanner too huge, suggestions? olsenc@ichips.intel.com (2001-02-25)
Re: flex scanner too huge, suggestions? Ron@Profit-Master.com (Ron Pinkas) (2001-02-25)
Re: flex scanner too huge, suggestions? troy@bell-labs.com (Troy Cauble) (2001-03-01)
| List of all articles for this month |

From: Troy Cauble <troy@bell-labs.com>
Newsgroups: comp.compilers
Date: 1 Mar 2001 02:33:52 -0500
Organization: Bell Labs, Holmdel, NJ, USA
References: 01-02-097
Keywords: lex
Posted-Date: 01 Mar 2001 02:33:52 EST

Troy Cauble <troy@bell-labs.com> wrote:
: I have a working flex/bison implementation to parse a text
: protocol. The problem is that the scanner is huge.


: Made with LFLAGS = -Cfe, size gives
: 42068(.text) + 24(.data) + 60(.bss) + 1687886(.rodata) = 1730038


: Made with LFLAGS = -Cem, size gives
: 42496(.text) + 24(.data) + 60(.bss) + 508326(.rodata) = 550906


: This is still pretty large for some of our target environments.


: I haven't tried to disable features to determine what makes
: it so huge, but the protocol is case-insensitive and context
: (position) sensitive. I addressed the context sensitivity
: with flex start states.




Thanks to all who replied here and via e-mail.


I thought I'd post what worked for future archive readers.


The big win for my flex scanner was rewriting INCLUSIVE start
states as EXCLUSIVE. I had about 50 start states with
1 to 5 rules each, to handle portions of the protocol.


Moving to EXTERNAL start states meant I had to add additional
rules for each start state (but a small subset of the INITIAL
rules). This resulted in more rules in the source file, but
much smaller tables. .rodata is less than half of what it was
at -Cem.


Given the table reduction, I'm *guessing* that 50 inclusive
start states generates 50 times the INITIAL (non start state)
rules. (Standard rules plus those in start state 1; standard
rules plus those in start state 2; ...)


Oddly enough, the oft-suggested "single identifier rule with
keyword lookup" actually increased my table size slightly
before I did rewrote the start states. I can't guess why.
(It wasn't the lookup function, I didn't even bother to
write that.)


-troy


Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.