Compilaon - TAUmaon/teaching/2014-2015/compilation/compilatio… · code exe Source text txt...

transcript

Compila(on 0368-‐3133

Lecture 2: Lexical Analysis

Syntax Analysis (1): CFLs, CFGs, PDAs

Noam Rinetzky 1

What is a Compiler? source language target language

Compiler

Executable

exe Source

Compiler vs. Interpreter

Toy compiler

The Real Anatomy of a Compiler

Executable

Source

txt Lexical Analysis

Sem. Analysis

Process text input

characters Syntax Analysis tokens AST

Intermediate code

generation

Annotated AST

Intermediate code

optimization

IR Code generation IR

Target code optimization

Symbolic Instructions

SI Machine code generation

Write executable

output

Lexical Analysis

Syntax Analysis

Lexical Analysis

9 Lexical analyzers are also known as “scanners’’

Lexical Analysis: from Text to Tokens

Lexical Analysis

Syntax Analysis

Sem. Analysis

Inter. Rep.

Code Gen.

x = b*b – 4*a*c

<ID,”x”> <EQ> <ID,”b”> <MULT> <ID,”b”> <MINUS> <INT,4> <MULT> <ID,”a”> <MULT> <ID,”c”>

Token Stream

•  Scan the input •  Par((ons the text into stream of tokens

– Numbers –  Iden(fiers –  Keywords –  Punctua(on

•  Tokens usually represented as (kind, value) •  Defined using regular expressions*

•  “word” in the source language •  “meaningful” to the syntac(cal analysis

What does Lexical Analysis do?

•  Language: fully parenthesized expressions Context free language

Regular languages

( ( 23 + 7 ) * 19 )

Expr → Num | LP Expr Op Expr RP Num → Dig | Dig Num Dig → ‘0’|‘1’|‘2’|‘3’|‘4’|‘5’|‘6’|‘7’|‘8’|‘9’ LP → ‘(’ RP → ‘)’ Op → ‘+’ | ‘*’

What does Lexical Analysis do?

•  Language: fully parenthesized expressions Context free language

( ( 23 + 7 ) * 19 )

Regular languages

Expr → Num | LP Expr Op Expr RP Num → Dig | Dig Num Dig → ‘0’|‘1’|‘2’|‘3’|‘4’|‘5’|‘6’|‘7’|‘8’|‘9’ LP → ‘(’ RP → ‘)’ Op → ‘+’ | ‘*’

LP LP Num Op Num RP Op Num RP

From scanning to parsing((23 + 7) * x)

) ? * ) 7 + 23 ( (

RP Id OP RP Num OP Num LP LP

Lexical Analyzer

program text

token stream

Parser Grammar: Expr → ... | Id Id → ‘a’ | ... | ‘z’

Num(23) Num(7)

Abstract Syntax Tree

valid syntax error

Some basic terminology

•  Lexeme (aka symbol) -‐ a series of lecers separated from the rest of the program according to a conven(on (space, semi-‐column, comma, etc.)

•  Pacern -‐ a rule specifying a set of strings. Example: “an iden(fier is a string that starts with a lecer and con(nues with lecers and digits” –  (Usually) a regular expression

•  Token -‐ a pair of (pacern, acributes)

Example void match0(char *s) /* find a zero */

if (!strncmp(s, “0.0”, 3))

return 0. ;

VOID ID(match0) LPAREN CHAR DEREF ID(s) RPAREN

LBRACE

IF LPAREN NOT ID(strncmp) LPAREN ID(s) COMMA STRING(0.0) COMMA NUM(3) RPAREN RPAREN

RETURN REAL(0.0) SEMI

RBRACE

EOF 16

Regular languages

•  Formal languages –  Σ = finite set of lecers – Word = sequence of lecer –  Language = set of words

•  Regular languages defined equivalently by –  Regular expressions –  Finite-‐state automata

Example: Integers w/o Leading Zeros

Digit = 1|2|…|9

Digit0 = 0|Digit Pos = Digit Digit0* Integer = 0 | Pos| -‐Pos

Challenge: Ambiguity

•  If = if•  Id = Lecer (Lecer | Digit)*

•  “if” is a valid iden(fiers… what should it be? •  ‘’iffy” is also a valid iden(fier

•  Solu(on –  Longest matching token –  Break (es using order of defini(ons…

•  Keywords should appear before iden(fiers 19

Building a Scanner – Take I

•  Input: String

•  Output: Sequence of tokens

Building a Scanner – Take I Token nextToken() { char c ; loop: c = getchar(); switch (c){ case ` `: goto loop ; case `;`: return SemiColumn; case `+`: c = getchar() ; switch (c) { case `+': return PlusPlus ; case '=’ return PlusEqual; default: ungetc(c); return Plus; }; case `<`: … case `w`: … } 21

There must be a becer way!

A becer way

•  AutomaGcally generate a scanner

•  Define tokens using regular expressions

•  Use finite-‐state automata for detec(on

Reg-‐exp vs. automata

•  Regular expressions are declara(ve – Good for humans – Not “executable”

•  Automata are opera(ve – Define an algorithm for deciding whether a given word is in a regular language

– Not a natural nota(on for humans 24

Overview

•  Construct a nondeterminis(c finite-‐state automaton (NFA) from regular expression

•  Determinize the NFA into a determinis(c finite-‐state automaton (DFA)

•  DFA can be directly used to iden(fy tokens 25

Automata theory: a bird’s-‐eye view

Determinis(c Automata (DFA)

•  M = (Σ, Q, δ, q0, F) –  Σ -‐ alphabet –  Q – finite set of state –  q0 ∈ Q – ini(al state –  F ⊆ Q – final states –  δ : Q × Σ à Q -‐ transi(on func(on

•  For a word w, M reach some state x – M accepts w if x ∈ F

DFA in pictures

accepting state

start state

transition

•  An automaton is defined by states and transi(ons

a,b,c a,b,c

Accep(ng Words

• Words are read leq-‐to-‐rightcba

• Missing transi(on = non-‐acceptance –  “Stuck state”

• Words are read leq-‐to-‐right

Accep(ng Words

Rejec(ng Words

• Missing transi(on means non-‐acceptancecbb

Non-‐determinis(c Automata (NFA)

•  M = (Σ, Q, δ, q0, F) –  Σ -‐ alphabet –  Q – finite set of state –  q0 ∈ Q – ini(al state

–  F ⊆ Q – final states –  δ : Q × (Σ ∪ {ε}) → 2Q -‐ transi(on func(on

•  DFA: δ : Q × Σ à Q

•  For a word w, M can reach a number of states X –  M accepts w if X ∩ M ≠ {}

•  Possible: X = {}

•  Possible ε-‐transi(ons

•  Allow mul(ple transi(ons from given state labeled by same lecer

Accep(ng words

• Maintain set of states

Accep(ng words

Accep(ng words•  Accept word if reached an accep(ng state

NFA+Є automata

•  Є transi(ons can “fire” without reading the input

start a

NFA+Є run example

start a

NFA+Є run example•  Now Є transi(on can non-‐determinis(cally take place

start a

NFA+Є run example

start a

NFA+Є run example

start a

NFA+Є run example

start a

•  Є transi(ons can “fire” without reading the input

NFA+Є run example

• Word accepted

start a

From regular expressions to NFA

•  Step 1: assign expression names and obtain pure regular expressions R1…Rm

•  Step 2: construct an NFA Mi for each regular expression Ri

•  Step 3: combine all Mi into a single NFA

•  Ambiguity resolu8on: prefer longest accep8ng word 48

From reg. exp. to automata•  Theorem: there is an algorithm to build an NFA+Є automaton for any regular expression

•  Proof: by induc8on on the structure of the regular expression

R = ε

R = φ

R = a a

Basic constructs

Composi(on R = R1 | R2 ε M1

R = R1R2

ε M1 M2

Repe((on

R = R1*

Naïve approach

•  Try each automaton separately

•  Given a word w: –  Try M1(w) –  Try M2(w) – … –  Try Mn(w)

•  Requires rese{ng aqer every acempt 54

Actually, we combine automata

1 2 a a

4 b 5 b 6

7 8 b a*b+ b a

10 b 11 a 12 b 13

a abb a*b+ abab

combines

Corresponding DFA

0 1 3 7 9

a 2 4 7 10

5 8 11 b

12 13 a b

abb a*b+ a*b+

Combine automata: an example.

Combine a, abb, a*b+, abab.

1# 2#a#

7# 8#b#

a*b+#b#a#

Scanning with DFA

•  Run un(l stuck – Remember last accepGng state

•  Go back to accep(ng state •  Return token

Ambiguity resolu(on

•  Longest word •  Tie-‐breaker based on order of rules when words have same length

1# 2#a#

7# 8#b#

a*b+#b#a#

Examples

0 1 3 7 9

a 2 4 7 10

5 8 11 b

12 13 a b

abb a*b+ a*b+

1# 2#a#

7# 8#b#

a*b+#b#a#

abaa: gets stuck aqer aba in state 12, backs up to state (5 8 11) pacern is a*b+, token is ab Tokens: <a*b+, ab> <a,a><a,a> 59

Examples

0 1 3 7 9

a 2 4 7 10

5 8 11 b

12 13 a b

abb a*b+ a*b+

abba: stops aqer second b in (6 8), token is abb because it comes first in spec 60 Tokens: <abb, abb> <a,a>

1# 2#a#

7# 8#b#

a*b+#b#a#

Summary of Construc(on

•  Describe tokens as regular expressions –  Decide acributes (values) to save for each token

•  Regular expressions turned into a DFA –  Also, records which acributes (values) to keep

•  Lexical analyzer simulates the run of an automata with the given transi(on table on any input string

A Few Remarks

•  Turning an NFA to a DFA is expensive, but –  Exponen(al in the worst case –  In prac(ce, works fine

•  The construc(on is done once per-‐language – At Compiler construc(on (me – Not at compila(on (me

Implementa(on

Implementa(on by Example if { return IF; } [a-‐z][a-‐z0-‐9]* { return ID; } [0-‐9]+ { return NUM; } [0-‐9]”.”[0-‐9]+|[0-‐9]*”.”[0-‐9]+ { return REAL; } (\-‐\-‐[a-‐z]*\n)|(“ “|\n|\t) { ; } . { error(); }

xy, i, zs98

3,32, 032

0.55, 33.1

-‐-‐comm\n \n, \t, “ “ ID

ID error REAL

NUM REAL

error w.s. error w.s.

9 10 11 12

int edges[][256]= { /* …, 0, 1, 2, 3, ..., -‐, e, f, g, h, i, j, ... */

/* state 0 */ {0, …, 0, 0, …, 0, 0, 0, 0, 0, ..., 0, 0, 0, 0, 0, 0}, /* state 1 */ {13, … , 7, 7, 7, 7, …, 9, 4, 4, 4, 4, 2, 4, …, 13, 13}, /* state 2 */ {0, …, 4, 4, 4, 4, ..., 0, 4, 3, 4, 4, 4, 4, …, 0, 0}, /* state 3 */ {0, …, 4, 4, 4, 4, …, 0, 4, 4, 4, 4, 4, 4, , 0, 0}, /* state 4 */ {0, …, 4, 4, 4, 4, …, 0, 4, 4, 4, 4, 4, 4, …, 0, 0}, /* state 5 */ {0, …, 6, 6, 6, 6, …, 0, 0, 0, 0, 0, 0, 0, …, 0, 0}, /* state 6 */ {0, …, 6, 6, 6, 6, …, 0, 0, 0, 0, 0, 0, 0, ..., 0, 0}, /* state 7 */ /* state … */ ... /* state 13 */ {0, …, 0, 0, 0, 0, …, 0, 0, 0, 0, 0, 0, 0, …, 0, 0} }; 65

ID error REAL

NUM REAL

9 10 11 12

Pseudo Code for Scanner char* input = … ; Token nextToken() { lastFinal = 0; currentState = 1 ; inputPositionAtLastFinal = input; currentPosition = input; while (not(isDead(currentState))) {

nextState = edges[currentState][*currentPosition]; if (isFinal(nextState)) { lastFinal = nextState ; inputPositionAtLastFinal = currentPosition; } currentState = nextState; advance currentPosition; } input = inputPositionAtLastFinal + 1; return action[lastFinal]; }

Example

Input: “if -‐-‐not-‐a-‐com”

2 blanks

ID error REAL

NUM REAL

9 10 11 12

final state input

0 1 if -‐-‐not-‐a-‐com

return IF

ID error REAL

NUM REAL

9 10 11 12

found whitespace

final state input

0 1 --not-a-com

12 12 --not-a-com

12 0 --not-a-com

final state input

0 1 -‐-‐not-‐a-‐com

9 9 -‐-‐not-‐a-‐com

9 10 -‐-‐not-‐a-‐com

9 0 -‐-‐not-‐a-‐com error

ID error REAL

NUM REAL

9 10 11 12

final state input

0 1 -‐not-‐a-‐com

ID error REAL

NUM REAL

9 10 11 12

Concluding remarks

•  Efficient scanner • Minimiza(on •  Error handling •  Automa(c crea(on of lexical analyzers

Efficient Scanners

•  Efficient state representa(on •  Input buffering •  Using switch and gotos instead of tables

Minimiza(on

•  Create a non-‐determinis(c automaton (NDFA) from every regular expression

•  Merge all the automata using epsilon moves (like the | construc(on)

•  Construct a determinis(c finite automaton (DFA) –  State priority

•  Minimize the automaton –  separate accep(ng states by token kinds

Example if { return IF; } [a-‐z][a-‐z0-‐9]* { return ID; } [0-‐9]+ { return NUM; }

error NUM

Example if { return IF; } [a-‐z][a-‐z0-‐9]* { return ID; } [0-‐9]+ { return NUM; }

error NUM

Example

error NUM

if { return IF; } [a-‐z][a-‐z0-‐9]* { return ID; } [0-‐9]+ { return NUM; }

error NUM

NUM NUM

Example

if { return IF; } [a-‐z][a-‐z0-‐9]* { return ID; } [0-‐9]+ { return NUM; }

error NUM

NUM NUM

Error Handling •  Many errors cannot be iden(fied at this stage •  Example: “fi (a==f(x))”. Should “fi” be “if”? Or is it a rou(ne name?

–  We will discover this later in the analysis –  At this point, we just create an iden(fier token

•  Some(mes the lexeme does not match any pacern –  Easiest: eliminate lecers un(l the beginning of a legi(mate lexeme –  Alterna(ves: eliminate/add/replace one lecer, replace order of two adjacent

lecers, etc.

•  Goal: allow the compila(on to con(nue •  Problem: errors that spread all over

Automa(cally generated scanners

•  Use of Program-‐Genera(ng Tools –  Specifica(on è Part of compiler –  Compiler-‐Compiler

Stream of tokens

JFlex regular expressions

input program scanner 80

Use of Program-‐Genera(ng Tools

•  Input: regular expressions and ac(ons •  Ac(on = Java code

•  Output: a scanner program that •  Produces a stream of tokens •  Invoke ac(ons when pacern is matched

Stream of tokens

JFlex regular expressions

input program scanner 81

Line Coun(ng Example

•  Create a program that counts the number of lines in a given input text file

Crea(ng a Scanner using Flex

int num_lines = 0; %% \n ++num_lines; . ; %% main() { yylex(); printf( "# of lines = %d\n", num_lines); }

Crea(ng a Scanner using Flex

ini(al

newline \n

int num_lines = 0; %% \n ++num_lines; . ; %% main() { yylex(); printf( "# of lines = %d\n", num_lines); }

JFLex Spec File User code: Copied directly to Java file %% JFlex direc(ves: macros, state names %% Lexical analysis rules:

–  Op(onal state, regular expression, ac(on –  How to break input to tokens –  Ac(on when token matched

Possible source of javac errors down

the road

DIGIT= [0-‐9] LETTER= [a-‐zA-‐Z]

YYINITIAL

{LETTER} ({LETTER}|{DIGIT})*

Crea(ng a Scanner using JFlex import java_cup.runtime.*; %% %cup %{ private int lineCounter = 0; %} %eofval{ System.out.println("line number=" + lineCounter); return new Symbol(sym.EOF); %eofval} NEWLINE=\n %% {NEWLINE} { lineCounter++; } [^{NEWLINE}] { }

Catching errors

• What if input doesn’t match any token defini(on?

•  Trick: Add a “catch-‐all” rule that matches any character and reports an error – Add aqer all other rules

A JFlex specifica(on of C Scanner import java_cup.runtime.*; %% %cup %{ private int lineCounter = 0; %} Letter= [a-‐zA-‐Z_] Digit= [0-‐9] %% ”\t” { } ”\n” { lineCounter++; } “;” { return new Symbol(sym.SemiColumn);} “++” { return new Symbol(sym.PlusPlus); } “+=” { return new Symbol(sym.PlusEq); } “+” { return new Symbol(sym.Plus); } “while” { return new Symbol(sym.While); } {Letter}({Letter}|{Digit})*

{ return new Symbol(sym.Id, yytext() ); } “<=” { return new Symbol(sym.LessOrEqual); } “<” { return new Symbol(sym.LessThan); }

Missing

•  Crea(ng a lexical analysis by hand •  Table compression •  Symbol Tables •  Nested Comments •  Handling Macros

Lexical Analysis: What

•  Input: program text (file) •  Output: sequence of tokens

Lexical Analysis: How

•  Construct a nondeterminis(c finite-‐state automaton (NFA) from regular expression

•  Determinize the NFA into a determinis(c finite-‐state automaton (DFA)

•  DFA can be directly used to iden(fy tokens 91

Lexical Analysis: Why

•  Read input file •  Iden(fy language keywords and standard iden(fiers •  Handle include files and macros •  Count line numbers •  Remove whitespaces •  Report illegal symbols

•  [Produce symbol table]

Syntax Analysis (1)

Context Free Languages Context Free Grammars Pushdown Automata

The Real Anatomy of a Compiler

Executable

Source

txt Lexical Analysis

Sem. Analysis

Process text input

characters Syntax Analysis tokens AST

Intermediate code

generation

Annotated AST

Intermediate code

optimization

IR Code generation IR

Target code optimization

Symbolic Instructions

SI Machine code generation

Write executable

output

Lexical Analysis

Syntax Analysis

Frontend: Scanning & Parsing((23 + 7) * x)

) x * ) 7 + 23 ( (

Lexical Analyzer

program text

token stream

Parser Grammar: E → ... | Id Id → ‘a’ | ... | ‘z’

Num(23) Num(7)

valid syntax error

From scanning to parsing((23 + 7) * x)

) x * ) 7 + 23 ( (

Lexical Analyzer

program text

token stream

Parser Grammar: E → ... | Id Id → ‘a’ | ... | ‘z’

Num(23) Num(7)

valid syntax error

Parsing •  Construct a structured representa(on of the input

•  Challenges –  How do you describe the programming language? –  How do you check validity of an input?

•  Is a sequence of tokens a valid program in the language?

–  How do you construct the structured representa(on? – Where do you report an error?

Some founda(ons

Context free languages (CFLs)

•  L01 = { 0n1n | n > 0 }

•  Lpolyndrom = {pp’ | p ∊ Σ* , p’=reverse(p)}

•  Lpolyndrom# = {p#p’ | p ∊ Σ* , p’=reverse(p), # ∉ Σ}

Context free grammars (CFG)

•  V – non terminals (syntac(c variables) •  T – terminals (tokens) •  P – deriva(on rules

–  Each rule of the form V t (T 4 V)*

•  S – start symbol

G = (V,T,P,S)

What can CFGs do?

•  Recognize CFLs

•  S t 0T1 •  T t 0T1 | ℇ

~ language-‐defining power

Recognizing CFLs

•  Context Free Grammars (CFG)

•  Nondeterminis(c push down automata (PDA)

Pushdown Automata (PDA)

•  Nondeterminis(c PDAs define all CFLs

•  Determinis(c PDAs model parsers. – Most programming languages have a determinis(c PDA

–  Efficient implementa(on

Intui(on: PDA

•  An ε-‐NFA with the addi(onal power to manipulate one stack

control (ε-‐NFA)

Intui(on: PDA

•  Think of an ε-‐NFA with the addi(onal power that it can manipulate a stack

•  PDA moves are determined by: –  The current state (of its “ε-‐NFA”) –  The current input symbol (or ε) –  The current symbol on top of its stack

Intui(on: PDA

if (oops) then stat:= blah else abort

Current

control (ε-‐NFA)

Intui(on: PDA

• Moves: –  Change state –  Replace the top symbol by 0…n symbols

• 0 symbols = “pop” (“reduce”) • 0 < symbols = sequence of “pushes” (“shiq”)

•  Nondeterminis(c choice of next move

PDA Formalism

•  PDA = (Q, Σ, Γ, δ, q0, $, F): – Q: finite set of states –  Σ: Input symbols alphabet –  Γ: stack symbols alphabet –  δ: transi(on func(on –  q0: start state –  $: start symbol –  F: set of final states

The Transi(on Func(on

•  δ(q, a, X) = { (p1, σ1), … ,(pn, σn)} –  Input: triplet

• A state q ∊ Q • An input symbol a ∊ Σ or ε • A stack symbol X ∊ Γ

– Output: set of 0 … k ac(ons of the form (p, σ) • A state p ∊ Q • σ a sequence X1⋯Xn ∊ Γ* of stack symbols

Ac(ons of the PDA

•  Say (p, σ) ∊ δ(q, a, X) –  If the PDA is in state q and X is the top symbol and a is at the front of the input

–  Then it can • Change the state to p. • Remove a from the front of the input

–  (but a may be ε).

• Replace X on the top of the stack by σ.

Example: Determinis(c PDA

•  Design a PDA to accept {0n1n | n > 1}. •  The states:

–  q = We have not seen 1 so far • start state

–  p = we have seen at least one 1 and no 0s since –  f = final state; accept.

Example: Stack Symbols

•  $ = start symbol. – Also marks the bocom of the stack, –  Indicates when we have counted the same number of 1’s as 0’s.

•  X = “counter” –  used to count the number of 0s we saw

Example: Transi(ons

•  δ(q, 0, $) = {(q, X$)}. •  δ(q, 0, X) = {(q, XX)}.

–  These two rules cause one X to be pushed onto the stack for each 0 read from the input.

•  δ(q, 1, X) = {(p, ε)}. –  When we see a 1, go to state p and pop one X.

•  δ(p, 1, X) = {(p, ε)}. –  Pop one X per 1.

•  δ(p, ε, $) = {(f, $)}. –  Accept at bocom. 113

Ac(ons of the Example PDA

0 0 0 1 1 1

X X X $

0 0 0 1 1 1

Example: Non Determinis(c PDA

•  A PDA that accepts palindromes –  L {pp’ ∊ Σ* | p’=reverse(p)}

Compilaon - TAUmaon/teaching/2014-2015/compilation/compilatio… · code exe Source text txt...

Documents