Compiler Tools
Lex/Yacc – Flex & Bison
Compiler Front End (from Engineering a Compiler)
Scanner (Lexical Analyzer)• Maps stream of characters into words
Basic unit of syntax x = x + y ; becomes <id,x> <eq,=> <id,x> <plus_op,+> <id,y> <sc,; >
• The actual words are its lexeme• Its part of speech (or syntactic category) is called its token
type• Scanner discards white space & (often) comments
Sourcecode Scanner
Intermediate RepresentationParser
Errors
tokens
Speed is an issue in scanning
use a specialized recognizer
The Front End (from Engineering a Compiler)
Parser• Checks stream of classified words (parts of speech) for
grammatical correctness• Determines if code is syntactically well-formed• Guides checking at deeper levels than syntax• Builds an IR representation of the code
Parsing is harder than scanning. Better to put more rules in scanner (whitespace etc).
Sourcecode Scanner
IRParser
Errors
tokens
Flex – Fast Lexical Analyzer
FLEXscanner
(program to recognize patterns
in text)
regular expressions& C-code rules
lex.yy.c contains yylex()
compile
executable – analyzesand executes input
Here’s where we’ll put the regular expressions to good use!
(Scanner generator)
Flex input file
3 sectionsdefinitions
%%
rules
%%
user code
Definition Section Examples
name definition DIGIT [0-9]
ID [a-z][a-z0-9]* A subsequent reference to {DIGIT}+"."{DIGIT}* is identical to:([0-9])+"."([0-9])*
C Code
Can include C-code in definitions%{
/* This is a comment inside the definition
*/
#include <math.h> // may need headers
#include <stdio.h> // for printf in BB
#include <stdlib.h> // for exit(0) in BB
%}
Rules
The rules section of the flex input contains a series of rules of the form: pattern action
In the definitions and rules sections, any indented text or text enclosed in %{ and %} is copied verbatim to the output (with the %{ %}'s removed). The %{ %}'s must appear unindented on lines by themselves.
Example: Simple Pascal-like recognizer
Definitions section:/* scanner for a toy Pascal-like language */
%{ /* need for the call to atof() below */
#include <math.h> %}DIGIT [0-9] ID [a-z][a-z0-9]*
Remember these are on a lineby themselves, unindented!
}Lines inserted as-is intoresulting code
} Definitions that can be used inrules section
Example continued
Rules section:%%
{DIGIT}+ { printf("An integer: %s (%d)\n", yytext, atoi(yytext ));}
{DIGIT}+"."{DIGIT}* {printf("A float: %s (%g)\n", yytext, atof(yytext));}
if|then|begin|end|procedure|function {printf("A keyword: %s\n", yytext);}
{ID} { printf( "An identifier: %s\n", yytext ); }
"+"|"-"|"*"|"/" { printf( "An operator: %s\n", yytext ); }
"{"[^}\n]*"}" /* eat up one-line comments */
[ \t\n]+ /* eat up whitespace */
. { printf( "Unrecognized character: %s\n", yytext ); }
pattern actiontext that matched the pattern(a char*)
Example continued
User code (required for flex, in library for lex)
%% yywrap() {} // needed to link, unless libfl.a is available// OR put %option noyywrap at the top of a flex file.
int main(int argc, char ** argv ) {
++argv, --argc; /* skip over program name */
if ( argc > 0 ) yyin = fopen( argv[0], "r" );
else yyin = stdin; yylex();
}
lexer function produced by lex
lex input file
Lex techniques
Hardcoding lists not very effective. Often use symbol table. Example in lec & yacc, not covered in class but see me if you’re interested.
Bison – like Yacc (yet another compiler compiler)
Context-free Grammarin BNF form, LALR(1)* Bison
Bison parser (c program)group tokens according togrammar rules
Bison parser provides yyparse
You must provide:• the lexical analyzer (e.g., flex)• an error-handling routine named yyerror• a main routine that calls yyparse
*LookAhead Left Recursive
Bison Parser
Same sections as flex (yacc came first): definitions, rules, C-Code
We’ll discuss rules first, then definitions and C-Code
Bison Parser – Rule Section
Consider CFG <statement> -> ID = <expression>
Would be written in bison “rules” section as:statement: NAME ‘=‘ expression
| expression { printf("= %d\n", $1); } ;
expression: NUMBER ‘+’ NUMBER { $$ = $1 + $3; } | NUMBER ‘-’ NUMBER { $$ = $1 + $3; } | NUMBER { $$ = $1; }
; Use : between lhs and rhs, place ; at end. What are $$? next slide…
white space
; at end
NOTE: The first rule in statement won’t be operational yet…
More on bison Rules and Actions
$1, $3 refer to RHS values. $$ sets value of LHS. In expression, $$ = $1 + $3 means it sets the value of
lhs (expression) to NUMBER ($1) + NUMBER ($3) A rule action is executed when the parser reduces that
rule (will have recognized both NUMBER symbols) lexer should have returned a value via yylval (next slide)
statement: NAME ‘=‘ expression
| expression { printf("= %d\n", $1); }
;
expression: NUMBER ‘+’ NUMBER { $$ = $1 + $3; }
| NUMBER ‘-’ NUMBER { $$ = $1 - $3; }
;
$$ $1 $2 $3
when is thisexecuted?
Coordinating flex and bison
Example to return int value:[0-9]+ { yylval = atoi(yytext); return NUMBER;}
returns recognized tokensets value for use in actions
This one just returns the numericvalue of the string stored in yytext
atoi is C function to convert string to integer
In prior flex examples we just returned tokens, not values
Also need to skip whitespace, return symbols
[ \t] ; /* ignore white space */
\nreturn 0; /* logical EOF */
. return yytext[0];
Bison Rule Details
Unlike flex, bison doesn’t care about line boundaries, so add white space for readability
Symbol on lhs of first rule is start symbol, can override with %start declaration in definition section
Symbols in bison have values, must be “declared” as some type YYSTYPE determines type Default for all values is int We’ll be using different types for YYSTYPE in the
SimpleCalc exercises
Bison Parser – Definition Section
Definition SectionTokens used in grammar should be defined.
Example rule: expression: NUMBER ‘+’ NUMBER { $$ = $1 + $3; } The token NUMBER should be defined. Later
we’ll see cases where expression should also be defined, and how to define tokens with other data types. %token must be lowercase, e.g.,:
%token NUMBERFrom the tokens that are defined, Bison will
create an appropriate header fileSingle quoted characters can be used as
tokens without declaring them, e.g., ‘+’, ‘=‘ etc.
Lex - Definition Section
Must include the header created by bison
Must declare yylval as extern%{
#include "simpleCalc.tab.h
extern int yylval;
#include <math.h>
%}
Bison Parser – C Section
At a minimum, provide yyerror and main routines
yyerror(char *errmsg){ fprintf(stderr, "%s\n", errmsg);}
main(){yyparse();
}
Bison Intro Exercise
Download SimpleCalc.y, SimpleCalc.l and mbison.bat
Create calculator executable mbison simpleCalc
FYI, mbison includes these steps: bison -d simpleCalc.y flex -L -osimpleCalc.c simpleCalc.l gcc -c simpleCalc.c gcc -c simpleCalc.tab.c gcc -Lc:\progra~1\gnuwin32\lib simpleCalc.o simpleCalc.tab.o -osimpleCalc.exe -lfl –ly
Test with valid sentences (e.g., 3+6-4) and invalid sentences.
Understanding simpleCalc%{#include "simpleCalc.tab.h"extern int yylval;%}
%%[0-9]+ { yylval = atoi(yytext); return NUMBER; }[ \t]; /* ignore white space */\n return 0; /* logical EOF */. return yytext[0];
%%/*---------------------------------------*//* 5. Other C code that we need. */ yyerror(char *errmsg){ fprintf(stderr, "%s\n", errmsg);}
main(){
yyparse();}
#ifndef YYTOKENTYPE# define YYTOKENTYPE /* Put the tokens into the symbol table, so that GDB and other debuggers know about them. */ enum yytokentype { NAME = 258, NUMBER = 259 };#endif/* Tokens. */#define NAME 258#define NUMBER 259
simpleCalc.tab.h
simpleCalc.l
Explanation:When the lexer recognizes a number[0-9]+ it returns the token NUMBERand sets yylval to the corresponding integer value.When the lexer sees a carriage return it returns 0. If it sees a space or tab it ignores it.When it sees any other character it returns that character (the first character in the yytext buffer). If the yyparse recognizes it – good! Otherwise the parser can generate an error.
Understanding simpleCalc, continued
%token NAME NUMBER%%statement: NAME '=' expression
| expression { printf("= %d\n", $1); };
expression: expression '+' NUMBER { $$ = $1 + $3; }| expression '-' NUMBER { $$ = $1 - $3; }| NUMBER { $$ = $1; };
ExplanationExecute simpleCalc and enter expression 1+2main program calls yyparse. This calls lex to recognize 1 as a NUMBER (puts 1 in yylval), sets $$ = $1 Calls lex which returns +, matches ‘+’ in first expression rhsCalls lex to recognize 2 as a NUMBER (puts 2 in yylval)Recognize expression + NUMBER and “reduce” this rule, does action {$$ = $1 + $3}. Recognizes expression as a statement, so it does the printf action.
Adding other variable types*
YYSTYPE determines the data type of the values returned by the lexer.
If lexer returns different types depending on what is read, include a union:
%union { // C feature, allows one memory area to char cval; // be interpreted in different ways.
char *sval; // For bison, will be used with yylvalint ival;
} The union will be placed at the top of your .y file
(in the definitions section) Tokens and non-terminals should be defined
using the union
* relates to SimpleCalc exercise 2
Adding other variable types - Example
Definitions in simpleCalc.y:%union {float fval;int ival;
}%token <ival>NUMBER%token <fval>FNUMBER%type <fval> expression
Use union in rules in simpleCalc.l:{DIGIT}+ { yylval.ival = atoi(yytext); return NUMBER;}
Processing lexemes in flex*
Sometimes you want to modify a lexeme before it is passed to bison. This can be done by putting a function call in the flex rules
Example: to convert input to lower case put a prototype for your function in the
definition section (above first %%) write the function definition in the C-code
section (bottom of file) call your function when the token is
recognized. Use strdup to pass the value to bison.
* relates to SimpleCalc exercise 3
Example continued
%{ #include “example.tab.h“
void make_lower(char *text_in);%}%%[a-zA-Z]+ {make_lower(yytext); yylval.sval = strdup(yytext); return KEYWORD; }%%void make_lower(char *text_in){int i; for (i=0; i<strlen(yytext); ++i)
yytext[i]=tolower(yytext[i]); }
need prototype here
function code in C section
function call to process textmake duplicate using strdupreturn token type
Adding actions to rules *
For more complex processing, functions can be added to bison.
Remember to add a prototype at the top, and the function at the bottom
* relates to SimpleCalc exercise 4
Processing more than one line *
To process more than one line, ensure the \n is simply ignored
Use a recursive rule to allow multiple inputs
* relates to SimpleCalc exercise 4
Summary of steps (from online manual)
The actual language-design process using Bison, from grammar specification to a working compiler or interpreter, has these parts:
1. Formally specify the grammar in a form recognized by Bison (i.e., machine-readable BNF). For each grammatical rule in the language, describe the action that is to be taken when an instance of that rule is recognized. The action is described by a sequence of C statements.
2. Write a lexical analyzer to process input and pass tokens to the parser.
3. Write a controlling function (main) that calls the Bison-produced parser.
4. Write error-reporting routines.