Re: Parser validation/test suites ?

Kenn Heinrich wrote:
> A question perhaps more software engineering than compiler theory, but
> here goes:
>
> How do people construct a good validation and regression test system
> for a compiler, in particular a parser module? I'm thinking along the
> lines of a set of test source programs, and expected results. Some
> kind of script automagically runs all the sources, and produces a
> simple log file (Web page, ...?) which could be checked into your
> version control for tracking the compiler right along with the source.
>
> How this set might be organized? By directories, by a suite that must
> pass, a suite that must fail, a suite that tests optional extensions,
> etc...


First you will need a tool for running the regression testing. You can
write your own tool, but be careful writing that tool does not end up
being a major software engineering project of its own. I have seen that
happen once, but I have also twice seen people write such a tool in a
short time. Personally I would search the Internet to see if there was
some free code, that met my needs. Alternatively I would buy a tool.


Assuming that I did not need to write a tool, I would organize the test
suite as recommended in the documentation for the tool. Further, I
would net make a special test suite for testing the parser. The later
parts of the compiler must also be tested, and there is no reason why
these parts should not be subject to the test examples used for testing
the parser, e.g., the semantic checker should be capable of handling
programs with syntax errors, and why writing two test suites, one for
testing the correct handling of syntax errors in the parser and one of
testing the correct handling of syntax errors in the semantic checker?


That brings me to my last advice: Try getting as many examples as you
can handle. Computers are fast, and they can easily run large
regression test suites. However, writing large test suites require many
resources, and I would suggest that you go hunting for examples of VHDL
code on the Internet. This may give you many examples of code with no
syntax errors, and thus it will not free you from writing test cases.
However, the code from the Internet will be invaluable when you are
going to test later parts of your tool. Further, users tend to write
code in a different style than writers of test cases, and if you do not
include examples of code written by users then there might be bugs in
parts of the tool, that is frequently used by users but but rarely
tested by your home written test examples. That could result in the
users thinking that your tool is buggy, while the tool really is of
acceptable quality--except for a few often used parts.


At last, I would avoid parser generators that use the error recovery of
yacc. That error recovery was fine in the early seventies where parser
generators and parsers had to be squeezed down in 16 bit computers, but
it has been outdated for at least 20 years. Today you should expect an
error recovery to be better at guessing the intentions of the programmer
and/or be easier to work with for the compiler writer. I would suggest
that you go hunting for at parser generator with a better error
recovery. Modern parser generators may also have a more powerful
language for describing the language accepted by the generated parser.
These advantages could end up saving you money even if you have to buy
the parser generator.


Karsten Nyblad

Return to the comp.compilers page.
Search the comp.compilers archives again.