Lecture 29 | Compiler I

From Tokens to Structure

Grammars, parse trees, and recursive descent.

Parsing

A Token Stream Is Not Yet a Program

The tokenizer can recognize every token in both sequences. Only one follows the language's structural rules.

let x = x + 1;

let = x 1 + ;

Parsing determines whether the tokens fit the grammar and identifies the construct formed by each group.

Grammars

Rules for Building Valid Constructs

Terminal

A token that appears in the source, such as while, (, or an identifier.

Non-terminal

A higher-level construct formed from terminals and other non-terminals, such as an expression or statement.

Context-free grammar

A set of rules describing how each non-terminal can be formed from simpler elements.

Grammars

Read a Grammar in Both Directions

A partial C grammar beside a nested while-loop accepted by that grammar

Construct: use the rules to generate a valid program.

Analyze: use the same rules to explain the structure of a given program.

Grammar Notation

Alternatives, Optional Parts, and Repetition

'while'Terminalappears verbatim

expressionNon-terminalanother grammar rule

x | yChoicex or y

x?Optionalzero or one x

x*Repetitionzero or more x

whileStatement: 'while' '(' expression ')' '{' statements '}'

Exercise 1

Use the Grammar as a Checklist

returnStatement: 'return' expression? ';'

Which sequences match this rule?

return;valid

return x;valid

return xinvalid: missing ;

return x y;invalid unless x y forms one expression

Parsing

The Result Is Hierarchical

Matching the token stream against nested grammar rules uncovers a hierarchy called a parse tree or derivation tree.

statement

↓

whileStatement

↓

while(expression)statement

Non-terminals are internal nodes. Tokens are the terminal leaves.

Next: See the complete tree for a nested statement.

Parse Trees

One Program, Many Nested Constructs

Exercise 2

Sketch a Small Parse Tree

letStatement: 'let' identifier '=' expression ';'
expression: term (op term)*
term: identifier | integerConstant
op: '+' | '-'

Sketch the tree for let x = y + 2;. Which nodes are non-terminals?

Top-down: letStatement contains an expression; that expression contains two term nodes separated by the terminal +.

Recursive Descent

Turn Every Grammar Rule into a Routine

statement→compileStatement()

whileStatement→compileWhile()

expression→compileExpression()

term→compileTerm()

When a rule contains a non-terminal, its routine calls the routine for that non-terminal. The call structure mirrors the parse tree.

Recursive Descent

Tracing a While Statement

while (count < 100) { let count = count + 1; }

compileWhile() consumes while and (.
It calls compileExpression() for count < 100.
It consumes ) and {.
It calls compileStatements(), which calls compileLet().
Finally, it consumes the closing }.

Choosing a Rule

The Current Token Usually Decides

let→compileLet()

if→compileIf()

while→compileWhile()

do→compileDo()

return→compileReturn()

Jack was designed so that the first token usually identifies the alternative. Parsing a term that begins with an identifier requires one-token lookahead.

Exercise 3

When Is Lookahead Necessary?

An identifier can begin several kinds of term. What does the next token reveal?

scorevariable

items [array entry

draw (subroutine call

Screen .qualified subroutine call

Syntax Errors

A Failed Expectation Reveals the Error

while (x < 10 { let x = x + 1; }

After compiling the expression, the grammar requires ). The parser sees { instead.

A useful diagnostic records:what was expected;what was found;where it was found.

The same grammar that recognizes valid programs helps identify invalid ones.

Summary of This Lecture

Parsing Explains How the Tokens Fit Together

Grammar

Defines valid constructions from terminals and non-terminals.

Parse tree

Records the hierarchical derivation of one program.

Recursive descent

Implements each grammar rule as a mutually recursive routine.

Lecture 30 | Compiler I

Building the Jack Syntax Analyzer

From the Jack grammar to structured XML.

Go to Lecture 30