Lecture 28 | Compiler I

From Source Code to Tokens

The first stage of understanding a Jack program.

Compilation

What Must a Compiler Do?

Syntax analysis

Recognize the structure of the source program: classes, declarations, statements, expressions, and terms.

Code generation

Use that structure to produce an equivalent program in the target language.

This week isolates the first task: understanding the syntax of Jack.

Next: Locate syntax analysis inside the Jack compiler.

Syntax Analysis

The Front End of the Jack Compiler

A Jack program enters a tokenizer and parser; XML is produced in Project 10 and VM code follows code generation in Project 11

Project 10: generate XML that exposes the recognized structure. Later: retain the parsing logic and replace XML output with VM code generation.

Syntax Analysis

Recognizing Nested Structure

Humans see the structure of a program almost immediately. A syntax analyzer must discover it explicitly.

classcontains subroutines

→

subroutinecontains statements

→

statementcontains expressions

→

expressioncontains terms

while (count < 100) {
  let count = count + 1;
}

Next: Replace the character stream with meaningful atoms.

Lexical Analysis

Characters Become Tokens

A source-code fragment is converted into a sequence of tokens while comments and layout disappear

Lexical analysis, also called scanning or tokenizing, groups characters into the language's smallest meaningful units.

Jack Tokens

Five Token Types

Keywordclasswhilereturn

Symbol{+;

IdentifiercountMainsum2

Integer constant015332767

String constant"Hello world"

An identifier may contain letters, digits, and underscores, but it cannot begin with a digit.

Exercise 1

Tokenize One Jack Statement

Write the tokens in order and classify each one.

let total = total + 1;

letkeywordtotalidentifier =symboltotalidentifier +symbol1integer constant ;symbol

Lexical Analysis

What the Tokenizer Ignores—and Preserves

Ignored between tokens

spaces, tabs, and newlines;
// comments;
/* ... */ and /** ... */ comments.

Preserved as one token

Everything between the quotes of a string constant—including spaces and punctuation—is its value.

"not // a comment"

Comment removal must therefore respect whether the scanner is currently inside a string.

Exercise 2

What Should Survive Tokenization?

let message = "a // b"; // explain the value
let x = 7/* old value */+2;

Is // inside the string a comment?
Does removing the block comment merge 7 and +?

Answer: The string remains one token with value a // b; the second statement tokenizes as let x = 7 + 2 ;.

Tokenizer Output

Expose Every Token as XML

Jack source

if (x < 153) {
  let city = "Paris";
}

Selected XML tokens

<keyword> if </keyword>
<symbol> ( </symbol>
<identifier> x </identifier>
<symbol> &lt; </symbol>
<integerConstant> 153 </integerConstant>

XML reserves <, >, ", and &; output them as <, >, ", and &.

JackTokenizer

A Small, Testable Interface

hasMoreTokens()Is another token available?

advance()Make the next token current.

tokenType()Classify the current token.

keyWord() / symbol()Return a keyword or symbol value.

identifier()Return the identifier text.

intVal() / stringVal()Return the constant's value.

The value-specific routine is called only when tokenType() reports the matching type.

Exercise 3

Classify—or Reject—Each Candidate

classkeyword

Classidentifier

_count2identifier

2countinvalid identifier

32767integer constant

32768not a valid Jack integer constant

Summary of This Lecture

A Tokenizer Gives the Parser Better Atoms

Input

A stream of source-code characters.

Work

Ignore layout and comments; recognize five token types.

Output

An ordered token stream that preserves program text, not yet its structure.

Lecture 29 | Compiler I

From Tokens to Structure

Grammars, parse trees, and recursive descent.

Go to Lecture 29