How to create AST parser which allows syntax errors?

First, what to read about parsing and building AST?

How to create parser for a language (like SQL) that will build an AST and allow syntax errors?

For example, for "3+4*5":

  +
 / \
3   *
   / \
  4   5

And for "3+4*+" with syntax error, parser would guess that the user meant:

  +
 / \
3   *
   / \
  4   +
     / \
    ?   ?

Where to start?

SQL:

    SELECT_________________
   /           \           \
  .           FROM        JOIN
 / \           |         /    \
a city_name  people   address  ON
                                |
                                =______________
                               /               \
                              .____             .
                             /     \           / \
                            p  address_id     a  id

Answers


The standard answer to the question of how to build parsers (that build ASTs), is to read the standard texts on compiling. Aho and Ullman's "Dragon" Compiler book is pretty classic. If you haven't got the patience to get the best reference materials, you're going to have more trouble, because they provide theory and investigate subtleties. But here is my answer for people in a hurry, building recursive descent parsers.

One can build parsers with built-in error recovery. There are many papers on this sort of thing, a hot topic in the 1980s. Check out Google Scholar, hunt for "syntax error repair". The basic idea is that the parser, on encountering a parsing error, skips to some well-known beacon (";" a statement delimiter is pretty popular for C-like languages, which is why you got asked in a comment if your language has statement terminators), or proposes various input stream deletions or insertions to climb over the point of the syntax error. The sheer variety of such schemes is surprising. The key idea is generally to take into account as much information around the point of error as possible. One of the most intriguing ideas I ever saw had two parsers, one running N tokens ahead of the other, looking for syntax-error land-mines, and the second parser being feed error repairs based on the N tokens available before it encounters the syntax error. This lets the second parser choose to act differently before arriving at the syntax error. If you don't have this, most parser throw away left context and thus lose the ability to repair. (I never implemented such a scheme.)

The choice of things to insert can often be derived from information used to build the parser (often First and Follow sets) in the first place. This is relatively easy to do with L(AL)R parsers, because the parse tables contain the necessary information and are available to the parser at the point where it encounters an error. If you want to understand how to do this, you need to understand the theory (oops, there's that compiler book again) of how the parsers are constructed. (I have implemented this scheme successfully several times).

Regardless of what you do, syntax error repair doesn't help much, because it is almost impossible to guess what the writer of the parsed document actually intended. This suggests fancy schemes won't be really helpful. I stick to simple ones; people are happy to get an error report and some semi-graceful continuation of parsing.

A real problem with rolling your own parser for a real language, is that real languages are nasty messy things; people building real implementations get it wrong and frozen in stone because of existing code bases, or insist on bending/improving the language (standards are for wimps, goodies are for marketing) because its cool. Expect to spend a lot of time re-calibrating what you think the grammar is, against the ground truth of real code. As a general rule, if you want a working parser, better to get one that has a track record rather than roll it yourself.

A lesson most people that are hell-bent to build a parser don't get, is that if they want to do anything useful with the parse result or tree, they'll need a lot more basic machinery than just the parser. Check my bio for "Life After Parsing".


Need Your Help

How can I take advantage of circleci parallelism in my Java / Maven / Surefire project?

java junit junit4 maven-surefire-plugin circleci

I have a Java project that uses Maven and the [maven-surefire-plugin][1] to run Junit4 tests. I'm building with CircleCi. How can I enable parallelism so that my test suite runs faster?

Rails BLOB/TEXT column used in key specification without a key length

mysql ruby-on-rails rails-migrations

I am trying to run a rails migration and I am seeing the error "BLOB/TEXT column used in key specification without a key length"... However, I'm specifying the length in the migration class. Rails

About UNIX Resources Network

Original, collect and organize Developers related documents, information and materials, contains jQuery, Html, CSS, MySQL, .NET, ASP.NET, SQL, objective-c, iPhone, Ruby on Rails, C, SQL Server, Ruby, Arrays, Regex, ASP.NET MVC, WPF, XML, Ajax, DataBase, and so on.