A PRIL 05, 2006 B IELEFELD, G ERMANY "A NALYZING A MBIGUITY OF C ONTEXT-FREE G RAMMARS " Analyzing...

transcript

APRIL 05, 2006BIELEFELD, GERMANY "ANALYZING AMBIGUITY OF CONTEXT-FREE GRAMMARS"

Analyzing Ambiguity of Context-Free Grammars

BRICS, Department of Computer Science

University of Aarhus, Denmark

Claus Brabrandbrabrand@brics.dk

Anders Mølleramoeller@brics.dk

[ 2 ]BIELEFELD, GERMANY "ANALYZING AMBIGUITY OF CONTEXT-FREE GRAMMARS" APRIL 05, 2006

“Analyzing Ambiguity of Context-Free Grammars”“Analyzing Ambiguity of Context-Free Grammars”

// Abstract

Context-free grammar ambiguity is undecidable.

However, just because the problem is undecidable does not mean that there are no useful approximations to the problem.

We exhibit a linguistic characterization of context-free ambiguity which transforms the problem of checking a grammar for ambiguity from a grammatical or structural property to a linguistic property, dealing solely with the set of strings in specific parts of the grammar.

We then show how to exploit this characterization to obtain a conservative approximation to the problem (based on the [Mohri-Nederhof, 2000] regular approximation of context-free grammars). The resulting approximation is capable of acquitting unambiguous grammars that the LR(k) cannot (for any k).

Finally, we give some pointers on how to boost the precision of the analysis even further.

However, just because the problem is undecidable does not mean that there are no useful approximations to the problem.

// Outline

Introduction Vertical / Horizontal Ambiguity Characterization of Ambiguity (Over-)Approximation Approximation (AMN) Examples and Assessment Related Work Conclusion

// Context-Free Grammar

N finite set of nonterminals finite set of terminals s N start nonterminal : N P(E*) production function, E = N

G = N, , s,

Assume: All nN reachable (from s) All nN derive some (finite) string

L : G P(*) language of G, L(G)

// CFG Decision Problems Decidable:

Membership: L(GCFG) !

Emptiness: L(GCFG) = !

Intersection (w/ REG): L(GCFG) L(RREG) = ! …

Undecidable: Intersection (w/ CFG): L(GCFG) L(G’CFG) = ? …

Ambiguity: *: multiple derivation trees ?

// Ambiguity: Undecidable!

Undecidable!

However…

unambiguous ambiguous

Ambiguity: *: multiple derivation trees ?

// “Side-Stepping Undecidability”

Unsafe approximation:

Safe approximation:

However, just because it’s undecidable, doesn’t mean there aren’t (good) approximations! Indeed, the whole area of static analysis works on “side-stepping undecidability”.

safe (over-)approximation

safe (under-)approximation

unsafe approximation

// Motivation

Use safe (over-)approximation:

“Yes!” “G guaranteed unambiguous”!!! Safely use any GLR parser on G

Because: we never get two parses at runtime!

Hence: dynamic parse ambiguity static parse ambiguity

// Motivation (cont’d)

Undecidability means: “there’ll always be a slack”:

However, still useful! Possible interpretations of “No?”:

Treat as error (reject grammar): “Please redesign your grammar” (as in LR(k))

Treat as warning: “Here are some potential problems”

// Vertical Unambiguity

“Vertical unambiguity”:

Example:

n N : , ’ (n) : ’ L() L(’) =

Z : x A y : x B y A : aB : a

Ambiguous string:

~ “reduce/reduce conflict” in [Yacc]

// Horizontal Unambiguity

“Horizontal unambiguity”:

where:

Example:

n N: (n): i [1..||-1]: L(0 .. i-1) L(i .. ||-1 ) =

: P(*) P(*) P(*)

X Y = { xay | x,y* a+ x,xaL(X) y,ayL(Y) }

Z : A B A : x a : xB : a y : y

Ambiguous string:

~ “shift/reduce conflict” in [Yacc]

// Characterization of Ambiguity

Theorem 1:

Lemma 1a: (“”)

Lemma 1b: (“”)

G G G unambiguous

// Proof (Lemma 1a): “”

Lemma 1a:

…contrapositively:

Proof: Assume G ambiguous (i.e. 2 der. trees for )

Show: by induction in max height of the 2 derivation trees

G G G unambiguous

G ambiguous G G

// Proof (Lemma 1a): “” (Base)

Base case (height 1): The ambiguity means that:

However, this means that = t0 t1 ... t||-1 = '(i.e. the two trees must be the same); and so the result holds vacuously

// Proof (Lemma 1a): “” (I.H.)

Induction step (height n): Assume induction hypothesis (for height n-1)

The ambiguity means:

i ’i’

… …i … …’i’

||-1= ’0 ’|’|-10.. .. .. ..

// Proof (Lemma 1a): “” (’)

Case (i.e. different production):

…but then i.e., we have a vertical ambiguity:

L() L(’) {}

i ’i’

… …i … …’i’

||-1= ’0 ’|’|-10.. .. .. ..

// Proof (Lemma 1a): “” (=’,1)

Case (i.e., same prod. ): i.e. “the top of the trees are the same”

Case : ambiguity in subtreei (i.e. & ambiguously derive same i): Induction hypothesis (on these subtrees)

i : i = ’i

… …i … …i’

||-1= 0 ||-10.. .. .. ..

Ti T'i

Case (i.e., same prod. ): Case (i.e., ):

Now let k = min{ i | i = 'i }

...then:

i : i ’i

i : i = ’i

L(0 .. k) L(k+1 .. || ) {xay'} G

// Proof (Lemma 1a): “” (=’,2)

i : i = ’i = ’

= xay'

// Proof (Lemma 1b): “”

Lemma 1b:

...contrapositively:

Proof: Assume “ ” (vertical conflict):

But then derive (using reachability + derivability of N):

s * x N x * x a * x a y

s * x N x ’ * x a * x a y

N * a, N ’ * a, L() L(’) {a}

G G G unambiguous

G ambiguous G G

for some N

// Proof (Lemma 1b): “” (cont’d)

Assume “ ” (horizontal conflict): Then for some NN:

But then derive (using reachability + derivability of N):

s * v N v * v x * v x a y * v x a y w

s * v N v * v x a * v x a y * v x a y w

N , L() L()

x,y * : a + : x,xa L() y,ay L()

// (Over-)Approximation (A)

(Over-)Approximation A : E* P(*)

A decidable “ ” and “ ” decidable on co-dom(A)

Approximated vertical unambiguity:

Approximated horizontal unambiguity:

E* : L() A()

n N : , ’ (n) : A() A(’) =

n N: (n): i [1..||-1]: A(0 .. i-1) A(i .. ||-1) =

// Ambiguity Approximation

Theorem 2:

Proof:

“Conflicts w/ smaller sets conflicts w/ larger sets”:

G unambiguous

A() A() = L() L() =

G G G G

// Compositionality (of A’s)

Proposition 3:

Proof: Follows from definition [proof omitted]

i.e. “Approximations are compositional”!:

A, A’ decidable (over-)approximations A A’ decidable (over-)approximation

A A’

// Choice(s) of A?

A*() = * (constant) Worst approximation... …but safe!

Useless: “Cannot determine that any grammars are unambiguous”

worst approximation

// Choice(s) of A? (cont’d)

AMN() = [Mohri-Nederhof]() CFG REGDFA (Over-)Approximation

Properties of this “ ”: Good (over-)approximation! Now we deal with regular languages:

Almost everything is decidable!

Works on language, L(G); i.e. not on the structure of the grammar, G

“Regular Approximation of Context-Free Grammars through Transformation”[Mohri-Nederhof, 2000]

Black-box

// Decidability (of AMN)

“” decidable (constructively, using DFAs)

O(|XNFA||YNFA|)

“ ” decidable (constructively, using DFAs)

O(|XNFA||YNFA|)

AMN decidable

With potential counterexamples (DFAs)

G unambiguous

AMNAMN

For X,Y regular languages (NFAs):

All overlappings, “xay”, as DFAs; variant of “” construction!

// Decision Algorithm for (X Y)

XNFA YNFA

[X;Y]NFA

a path :

XNFA YNFA

[X Y]NFA

X Y = Ø

~ ~~ ~

[Mohri-Nederhof]: O(n2vh) Vertical Amb: O(n3v4h4) Horizontal Amb: O(n3v3h5) Total: O(n3v3h4(v+h)) O(g5)

// Asymptotic (Time) Complexity

N1 : e1,1 … ea,1

: … : e1,p … ea,p

n = |N| v = max{|(N)|, NN} h = max{||, (N), NN} g = nvh = |G|

// Implementation

Java impl.: "grambiguity" (510 lines, using): "dk.brics.automaton" [ http://www.brics.dk/automaton/ ] "dk.brics.grammar" [ http://www.brics.dk/grammar/ ] Java String Analyzer [ http://www.brics.dk/JSA/ ]

%> java –jar Grambiguity.jar G1.cfg

*** (potential) vertical ambiguity detected: 'S[aS]' vs. 'S[Sa]' shortest ambiguous string: "."

*** (potential) vertical ambiguity detected: 'S[aa]' vs. 'S[SS]' shortest ambiguous string: "()"

*** (potential) vertical ambiguity detected: 'S[aS]' vs. 'S[SS]' shortest ambiguous string: "."

*** (potential) vertical ambiguity detected: 'S[Sa]' vs. 'S[SS]' shortest ambiguous string: "."

*** (potential) vertical ambiguity detected: 'S[SS]' vs. 'S[empty]' shortest ambiguous string: ""

*** (potential) horizontal ambiguity detected: 'S[SS:0..0]' vs. 'S[SS:1..1]' shortest ambiguous string: "."

*** (potentially) ambiguous grammar: 5 (potential) vertical ambiguities 1 (potential) horizontal ambiguities

/* ambiguous */

S[aa] : "(" S2 ")" ; [aS] : "." S ; [Sa] : S "." ; [SS] : S S ; [empty] : ;

/* unambiguous */

P[aPa] : "a" P "a" ; [a] : "a" ; [empty] : ;

unambiguous grammar!

// Examples: Palindromes Palindromic examples:

P : "a" P "a" ; : "a" ; : ;

P : "a" P "a" ; : "b" P "b" ; : "b" ; : "a" ; : ;

P : "a" P "a" ; : ;

R : "a" R "b" ; : "b" R "a" ; : "a" "b" ; : "b" "a" ;

R : "a" R "b" ; : "b" R "a" ; : ;

Note: these are allnon-LR(k)

grammars !!!

// Examples: Odd/Even Keeping track of parity (odd/even):

Start : Even ; : Odd ;

Even : "(" "(" Odd ")" ")" ; : ;

Odd : "(" "(" Odd ")" ")" ; : "(" ")" ;

A(Even) = A(Odd) =

// Examples: RNA Analysis (G1) RNA Analysis (G1):

%> java –jar Grambiguity.jar G1.cfg

*** (potential) vertical ambiguity detected: 'S[aa]' vs. 'S[SS]' shortest ambiguous string: "()"

/* ambiguous */

S[aa] : "(" S ")" ; [aS] : "." S ; [Sa] : S "." ; [SS] : S S ; [empty] : ;

// Examples: RNA Analysis (G2) RNA Analysis (G2):

*** (potential) vertical ambiguity detected: 'S[aPa]' vs. 'S[SS]' shortest ambiguous string: "()"

*** (potential) vertical ambiguity detected: 'P[aPa]' vs. 'P[S]' shortest ambiguous string: "()"

/* ambiguous */

S[aPa] : "(" P ")" ; [aS] : "." S ; [Sa] : S "." ; [SS] : S S ; [empty] : ;

P[aPa] : "(" P ")" ; [S] : S ;

// Examples: RNA Analysis... RNA Analysis (G3,G4,G5,G6):

S[aS] : "." S ; [T] : T ; [empty] : ;

T[Ta] : T "." ; [aSa] : "(" S ")" ; [TaSa] : T "(" S ")" ;

S[LS] : L S ; [L] : L ;

L[aFa] : "(" F ")" ; [a] : "." ;

F[aFa] : "(" F ")" ; [LS] : L S ;

S[aS] : "." S ; [aSaS] : "(" S ")" S ; [empty] : ;

S[aPa] : "(" P ")" ; [aL] : "." L ; [Ra] : R "." ; [LS] : L S ;

L[aPa] : "(" P ")" ; [aL] : "." L ;

R[Ra] : R "." ; [empty] : ;

P[aPa] : "(" P ")" ; [aNa] : "(" N ")" ;

N[aL] : "." L ; [Ra] : R "." ; [LS] : L S ;

// Examples: RNA Analysis (G7) RNA Analysis (G7,G8):

*** (potential) vertical ambiguity detected: 'P[aPa]' vs. 'P[aNa]' shortest ambiguous string: "(((.)"

*** (potentially) ambiguous grammar: 1 (potential) vertical ambiguity 0 (potential) horizontal ambiguities

S[aPa] : "(" P ")" ; [aL] : "." L ; [Ra] : R "." ; [LS] : L S ;

L[aPa] : "(" P ")" ; [aL] : "." L ;

R[Ra] : R "." ; [empty] : ;

P[aPa] : "(" P ")" ; [aNa] : "(" N ")" ;

N[aL] : "." L ; [Ra] : R "." ; [LS] : L S ;

*** (potential) vertical ambiguity detected: 'P[aPa]' vs. 'P[aNa]' shortest ambiguous string: "(((.)"

S[aS] : "." S ; [T] : T ; [empty] : ;

T[Ta] : T "." ; [aPa] : "(" P ")" ; [TaPa] : T "(" P ")" ;

P[aPa] : "(" P ")" ; [aNa] : "(" N ")" ;

N[aS] : "." S ; [Ta] : T "." ; [TaPa] : T "(" P ")" ;

G8Note: these are all

spurious errorsdue to imprecisions

in the analysis

// Examples: Expressions Expressions:E[term] : T ; [plus] : E "+" T ; T[x] : "x" ; [par] : "(" E ")" ;

*** (potential) vertical ambiguity detected: 'E[term]' vs. 'E[plus]' shortest ambiguous string: "x+x"

*** (potential) horizontal ambiguity detected: 'E[plus:0..0]' vs. 'E[plus:1..2]' shortest ambiguous string: "x+x+x"

*** (potential) horizontal ambiguity detected: 'E[plus:0..1]' vs. 'E[plus:2..2]' shortest ambiguous string: "x+x+x"

Note: General problem with non-linear recursive structures

However, there's a trick...

// Examples: Expressions (cont'd) Expressions:E[term] : T ; [plus] : E "+" T ; T[x] : "x" ; [par] : "(" E ")" ;

E[term] : T ; [plus] : E "+" T ; T[x] : "x" ; [par] : "(" E ")" ;

E[term] : T ; [plus] : E "+" T ; T[x] : "x" ; [par] : "(" E ")" ; unambiguous grammar!

unfold

w = x+(x+(x+x)+x)+x

E : T : E "+" T T : "x" : "(" E ")"

unfold trick:(inside/outside)parentheses

// Benchmark Grammars

LALR(1)LR(1)LR(2)LR(3)LR(4)LR(5)LR(6)LR(7)LR(8)

LR(k)..

AMBIGUOUS

UNAMBIGUOUS

// Related Work (Dynamic)

Dynamic disambiguation: “Disambiguation-by-convention”:

Longest match, most specific match, …

Customizable: [Bison v. 1.5+]: %dprec, %merge [ASF+SDF]: “disambiguation filters”

Dynamic ambiguity interception: GLR ([Tomita], [Early], [Bison], [ASF+SDF], …)

// Related Work (Static)

Static disambiguation: “Disambiguation-by-convention”:

First match, most specific match, …

Customizable: [Yacc]: %left, %right, %nonassoc, %prec

Static ambiguity interception: LL(k), LALR(1), LR(k), … Our work goes here

// Conclusion

“Approximating Context-Free Grammar Ambiguity”“Approximating Context-Free Grammar Ambiguity”Context-free grammar ambiguity is undecidable.

However, just because the problem is undecidable does not mean that there are no useful /approximations/ to the problem.

A PRIL 05, 2006 B IELEFELD, G ERMANY "A NALYZING A MBIGUITY OF C ONTEXT-FREE G RAMMARS " Analyzing...

Documents