Abstraction-Based Interaction Model for

SynthesisHila Peleg, Technion

Shachar Itzhaky, TechnionSharon Shoham, Tel Aviv University

The research leading to these results has received funding from the European Union's - Seventh Framework Programme (FP7) under grant agreement n° 615688 – ERC- COG-PRIME.

Programming by Example

Task: find the most frequent bigram in a string.

2Lieberman, H. (2000). Programming by example. Communications of the ACM, 43(3), 72-72.

Programming by Example

Task: find the most frequent bigram in a string."abdfibfcfdebdfdebdihgfkjfdebd"

⇓"bd"

2Lieberman, H. (2000). Programming by example. Communications of the ACM, 43(3), 72-72.

Programming by Example

Task: find the most frequent bigram in a string."abdfibfcfdebdfdebdihgfkjfdebd"

⇓"bd"

input.takeRight(2)

2Lieberman, H. (2000). Programming by example. Communications of the ACM, 43(3), 72-72.

Programming by Example

Task: find the most frequent bigram in a string."abdfibfcfdebdfdebdihgfkjfdebd"

⇓"bd"

input.takeRight(2)

2Lieberman, H. (2000). Programming by example. Communications of the ACM, 43(3), 72-72.

Programming by Example

Task: find the most frequent bigram in a string."abdfibfcfdebdfdebdihgfkjfdebd"

⇓"bd"

input.takeRight(2)

"abbba"

⇓"bb"

2Lieberman, H. (2000). Programming by example. Communications of the ACM, 43(3), 72-72.

Programming by Example

Task: find the most frequent bigram in a string."abdfibfcfdebdfdebdihgfkjfdebd"

⇓"bd"

input.takeRight(2)

"abbba"

⇓"bb"

2

input.substring(1,3)

Lieberman, H. (2000). Programming by example. Communications of the ACM, 43(3), 72-72.

Programming Not Only by Example

Input⇓

Output

3Peleg, H., Shoham, S., & Yahav, E. (2017). Programming Not Only by Example. ICSE 2018 (Upcoming).

Programming Not Only by Example

Input⇓

Output𝑚 ∈ 𝑀 𝑚 (𝑖𝑛𝑝𝑢𝑡) = 𝑜𝑢𝑡𝑝𝑢𝑡

3Peleg, H., Shoham, S., & Yahav, E. (2017). Programming Not Only by Example. ICSE 2018 (Upcoming).

Programming Not Only by Example

Input⇓

Output𝑚 ∈ 𝑀 𝑚 (𝑖𝑛𝑝𝑢𝑡) = 𝑜𝑢𝑡𝑝𝑢𝑡

input.takeRight(2)

3Peleg, H., Shoham, S., & Yahav, E. (2017). Programming Not Only by Example. ICSE 2018 (Upcoming).

Programming Not Only by Example

Input⇓

Output𝑚 ∈ 𝑀 𝑚 (𝑖𝑛𝑝𝑢𝑡) = 𝑜𝑢𝑡𝑝𝑢𝑡

input.takeRight(2)

Exclude programs with takeRight

3Peleg, H., Shoham, S., & Yahav, E. (2017). Programming Not Only by Example. ICSE 2018 (Upcoming).

Programming Not Only by Example

Input⇓

Output𝑚 ∈ 𝑀 𝑚 (𝑖𝑛𝑝𝑢𝑡) = 𝑜𝑢𝑡𝑝𝑢𝑡

input.takeRight(2)

Exclude programs with takeRight

𝑚 ∈ 𝑀 𝒑(𝑚)

3Peleg, H., Shoham, S., & Yahav, E. (2017). Programming Not Only by Example. ICSE 2018 (Upcoming).

Our Goal

• To model user-driven synthesis• Works in practice but we do not understand its

limitations

• Properties• Of the synthesizer

• Of the user

• Guarantees• Termination (in paper)

• Are “bad sessions” recoverable

4

Iterative, interactive synthesis

𝑼∗

𝑺𝒊

𝐴𝑖 ∈ 2𝒫

𝑞𝑖 ∈ 𝑀

Partial specificationAvailable predicatesIdeal target set

Candidate programSearch space

State

5

Select

𝑺𝒊

𝑞𝑖 ∈ 𝑀

?

6

• Candidate program is selected via someselection criterion: 𝑆𝑒𝑙𝑒𝑐𝑡

• 𝑆𝑒𝑙𝑒𝑐𝑡 usually designed to return a program from 𝑈∗ ASAP (in 1-2 iterations)

• There is little theoretical work about the long run

Predicates

Programs

2𝑀

2𝒫

𝛼 𝐶 = 𝑝 ∈ 𝒫 ∀𝑚 ∈ 𝐶.𝑚 ⊨ 𝑝

𝛾 𝐴 = 𝑚 ∈ 𝑀 ∀𝑝 ∈ 𝐴.𝑚 ⊨ 𝑝

An abstract domain

7

Predicates

Programs

2𝑀

2𝒫

𝛼 𝐶 = 𝑝 ∈ 𝒫 ∀𝑚 ∈ 𝐶.𝑚 ⊨ 𝑝

𝛾 𝐴 = 𝑚 ∈ 𝑀 ∀𝑝 ∈ 𝐴.𝑚 ⊨ 𝑝

An abstract domain

7

𝑆𝑒𝑙𝑒𝑐𝑡

Predicates

Programs

2𝑀

2𝒫

𝛼 𝐶 = 𝑝 ∈ 𝒫 ∀𝑚 ∈ 𝐶.𝑚 ⊨ 𝑝

𝛾 𝐴 = 𝑚 ∈ 𝑀 ∀𝑝 ∈ 𝐴.𝑚 ⊨ 𝑝

An abstract domain

7

𝑆𝑒𝑙𝑒𝑐𝑡

Synthesis Session

• A synthesis session:

𝒮 = 𝐴0, 𝑞1 𝐴1, 𝑞2 …

• Synthesizer state: 𝑆𝑖 = 𝑆𝑖−1 ⊓ 𝐴𝑖• 𝑞𝑖 = 𝑆𝑒𝑙𝑒𝑐𝑡 𝑆𝑖−1 , or 𝑞𝑖 ∈ 𝛾(𝑆𝑖−1) ∪ {⊥}

• If qi ∈ 𝑈∗ ∪ {⊥}, the session terminates

Initial specifications

Candidate program

User answer

8

Synthesis user

• The user is aiming for some ideal set of programs 𝑈∗ ⊆ 𝑈 (where 𝑈 is the universe of all programs)

• The realizable target set is 𝑀∗ = 𝑀 ∩ 𝑈∗

• Correctness: A user step is correct when 𝐴𝑖 ⊆ 𝑝 ∈ 𝒫 ∣ ∃𝑚 ∈ 𝑈

∗. 𝑚 ⊨ 𝑝

9

Synthesis user

• The user is aiming for some ideal set of programs 𝑈∗ ⊆ 𝑈 (where 𝑈 is the universe of all programs)

• The realizable target set is 𝑀∗ = 𝑀 ∩ 𝑈∗

• Correctness: A user step is correct when 𝐴𝑖 ⊆ 𝑝 ∈ 𝒫 ∣ ∃𝑚 ∈ 𝑈

∗. 𝑚 ⊨ 𝑝

9

(1 until n).fold((x,z)=>x+z)

sum(range(1,n+1))

var sum=0

for(int i = 1; i

Synthesis user

• The user is aiming for some ideal set of programs 𝑈∗ ⊆ 𝑈 (where 𝑈 is the universe of all programs)

• The realizable target set is 𝑀∗ = 𝑀 ∩ 𝑈∗

• Correctness: A user step is correct when 𝐴𝑖 ⊆ 𝑝 ∈ 𝒫 ∣ ∃𝑚 ∈ 𝑈

∗. 𝑚 ⊨ 𝑝

9

(1 until n).fold((x,z)=>x+z)

sum(range(1,n+1))

var sum=0

for(int i = 1; i

Synthesis user

• The user is aiming for some ideal set of programs 𝑈∗ ⊆ 𝑈 (where 𝑈 is the universe of all programs)

• The realizable target set is 𝑀∗ = 𝑀 ∩ 𝑈∗

• Correctness: A user step is correct when 𝐴𝑖 ⊆ 𝑝 ∈ 𝒫 ∣ ∃𝑚 ∈ 𝑈

∗. 𝑚 ⊨ 𝑝

9

(1 until n).fold((x,z)=>x+z)

sum(range(1,n+1))

var sum=0

for(int i = 1; i

Synthesis user

• The user is aiming for some ideal set of programs 𝑈∗ ⊆ 𝑈 (where 𝑈 is the universe of all programs)

• The realizable target set is 𝑀∗ = 𝑀 ∩ 𝑈∗

• Correctness: A user step is correct when 𝐴𝑖 ⊆ 𝑝 ∈ 𝒫 ∣ ∃𝑚 ∈ 𝑈

∗. 𝑚 ⊨ 𝑝

9

(1 until n).fold((x,z)=>x+z)

sum(range(1,n+1))

var sum=0

for(int i = 1; i

Synthesis user

• The user is aiming for some ideal set of programs 𝑈∗ ⊆ 𝑈 (where 𝑈 is the universe of all programs)

• The realizable target set is 𝑀∗ = 𝑀 ∩ 𝑈∗

• Correctness: A user step is correct when 𝐴𝑖 ⊆ 𝑝 ∈ 𝒫 ∣ ∃𝑚 ∈ 𝑈

∗. 𝑚 ⊨ 𝑝

9

(1 until n).fold((x,z)=>x+z)

sum(range(1,n+1))

var sum=0

for(int i = 1; i x+z) + 1

Synthesis user

• The user is aiming for some ideal set of programs 𝑈∗ ⊆ 𝑈 (where 𝑈 is the universe of all programs)

• The realizable target set is 𝑀∗ = 𝑀 ∩ 𝑈∗

• Correctness: A user step is correct when 𝐴𝑖 ⊆ 𝑝 ∈ 𝒫 ∣ ∃𝑚 ∈ 𝑈

∗. 𝑚 ⊨ 𝑝

9

(1 until n).fold((x,z)=>x+z)

sum(range(1,n+1))

var sum=0

for(int i = 1; i x+z) + 1

User guarantees

• In a synthesis session:1. The user is correct for as long as possible.

When not possible, 𝐴𝑖 =⊥

2. The user will always accept a program from 𝑀∗

10

Progress

• Adding new predicate doesn’t guarantee that the session is progressing

• 𝑆𝑛−1 ⊏ 𝑆𝑛 could still mean that 𝛾 𝑆𝑛 = 𝛾(𝑆𝑛−1)

• Synthesizers usually don’t check

11

𝑴∗

𝜸(𝑺𝟎)𝜸(𝑺𝟏)𝑆1

𝑆0

𝑆2

Easiest way to make progress

12

𝑚

𝐴𝑖 s.t. ∃𝑝 ∈ 𝐴𝑖 . 𝑚 ⊭ 𝑝

Easiest way to make progress

12

𝑚

𝐴𝑖 s.t. ∃𝑝 ∈ 𝐴𝑖 . 𝑚 ⊭ 𝑝

• In other words: something that’s wrong with the current program

• Rule out at least the current program

Easiest way to make progress

12

𝑚

𝐴𝑖 s.t. ∃𝑝 ∈ 𝐴𝑖 . 𝑚 ⊭ 𝑝

• In other words: something that’s wrong with the current program

• Rule out at least the current program• Important when 𝑞𝑖 ∈ 𝛾(𝑆𝑖+1) ⇒ 𝑆𝑒𝑙𝑒𝑐𝑡 𝑆𝑖+1 = 𝑞𝑖• Easy to check, but too strong

A Different Model of Progress

• 𝐴𝑖 makes weak progress if 𝛾 𝑆𝑛−1 ⊓ 𝐴𝑖 = 𝛾 𝑆𝑛 ⊊ 𝛾 𝑆𝑛−1

• We can provide positive reinforcement

13

A Different Model of Progress

• 𝐴𝑖 makes weak progress if 𝛾 𝑆𝑛−1 ⊓ 𝐴𝑖 = 𝛾 𝑆𝑛 ⊊ 𝛾 𝑆𝑛−1

• We can provide positive reinforcement

13

(1 to n).fold((x,z)=>x+z) + 1

A Different Model of Progress

• 𝐴𝑖 makes weak progress if 𝛾 𝑆𝑛−1 ⊓ 𝐴𝑖 = 𝛾 𝑆𝑛 ⊊ 𝛾 𝑆𝑛−1

• We can provide positive reinforcement

13

(1 to n).fold((x,z)=>x+z) + 1

A Different Model of Progress

• 𝐴𝑖 makes weak progress if 𝛾 𝑆𝑛−1 ⊓ 𝐴𝑖 = 𝛾 𝑆𝑛 ⊊ 𝛾 𝑆𝑛−1

• We can provide positive reinforcement

• Harder to check: 𝑆𝑖 ⇏ 𝐴𝑖• Once we know there is progress, we know

some things about termination (see paper)

13

(1 to n).fold((x,z)=>x+z) + 1

Convergence

• A session converges if 𝛾(𝑆𝑛) ⊆ 𝑀∗

• User correctness means the session ends at state 𝑛

• Converges successfully: ∅ ≠ 𝛾 𝑆𝑛 ⊆ 𝑀∗

14

𝑴∗

𝜸(𝑺𝒏)

Core set

• Core set: the set of finiteunderapproximations of 𝑀∗

ℬ = 𝐵 ⊆ 𝒫0 ≠ 𝛾 𝐵 ⊆ 𝑀∗

∧ 𝐵 ∈ ℕ

𝑼∗

𝑴∗

15

Core set

• Core set: the set of finiteunderapproximations of 𝑀∗

ℬ = 𝐵 ⊆ 𝒫0 ≠ 𝛾 𝐵 ⊆ 𝑀∗

∧ 𝐵 ∈ ℕ

𝑼∗

𝑴∗

𝑩𝟐𝑩𝟏

𝑩𝟑

15

Core set

𝑼∗

𝑴∗

𝑩𝟐𝑩𝟏

𝑩𝟑

15

• Core set: the set of finiteunderapproximations of 𝑀∗

ℬ = 𝐵 ⊆ 𝒫0 ≠ 𝛾 𝐵 ⊆ 𝑀∗

∧ 𝐵 ∈ ℕ

• User intention is realizable if 𝑀∗ ≠ ∅

• 𝓟-realizability: can converge under 𝒫

Core set

𝑼∗

𝑴∗

𝑩𝟐𝑩𝟏

𝑩𝟑𝑀∗ is 𝒫-realizable if ℬ ≠ ∅

15

• Core set: the set of finiteunderapproximations of 𝑀∗

ℬ = 𝐵 ⊆ 𝒫0 ≠ 𝛾 𝐵 ⊆ 𝑀∗

∧ 𝐵 ∈ ℕ

• User intention is realizable if 𝑀∗ ≠ ∅

• 𝓟-realizability: can converge under 𝒫

Infeasible point

• A state where 𝛾 𝑆𝑖 ∩𝑀

∗ = ∅

• 𝑆𝑒𝑙𝑒𝑐𝑡 can’t succeed, even in best case

• The First Infeasible Pointis the first point of failure

• Can we backtrack before an infeasible point?

16

𝑴∗

𝑩𝟐𝑩𝟏

𝑩𝟑

𝑺𝒊

Point of inevitable failure• State 𝑆𝑖 is an POIF if ∀𝐵 ∈ ℬ. 𝛾 𝑆𝑖 ∩ 𝛾 𝐵 = ∅

• Specifically, 𝑆𝑖 is an POIF if 𝛾 𝑆𝑖 ∩𝑀

∗ = ∅𝑴∗

𝑩𝟐𝑩𝟏

𝑩𝟑

𝑺𝟎𝑺𝟏

𝑺𝟐

17

Point of inevitable failure• State 𝑆𝑖 is an POIF if ∀𝐵 ∈ ℬ. 𝛾 𝑆𝑖 ∩ 𝛾 𝐵 = ∅

• Specifically, 𝑆𝑖 is an POIF if 𝛾 𝑆𝑖 ∩𝑀

∗ = ∅

• But not necessarily 𝑴∗

𝑩𝟐𝑩𝟏

𝑩𝟑

𝑺𝟎𝑺𝟏

𝑺𝟐

17

𝑺𝒊

Point of inevitable failure

𝑴∗

𝑩𝟐𝑩𝟏

𝑩𝟑

𝑺𝟎𝑺𝟏

𝑺𝟐

𝑺𝒊

17

• State 𝑆𝑖 is an POIF if ∀𝐵 ∈ ℬ. 𝛾 𝑆𝑖 ∩ 𝛾 𝐵 = ∅

• Specifically, 𝑆𝑖 is an POIF if 𝛾 𝑆𝑖 ∩𝑀

∗ = ∅

• But not necessarily

• As long as 𝑆𝑖 is not an POIF we can still converge

• Can we backtrack before an inevitable point of failure?

Backtracking from failure

Theorem: for any 𝑘 ∈ ℕ there exists a session 𝒮 where state 𝑆𝑖 is an point of inevitable failure and only state 𝑆𝑘+𝑖 is the first infeasible point.

18

Backtracking from failure

Essentially: there is no bound on the number of steps to backtrack once failure is apparent.

Proof: by construction

Theorem: for any 𝑘 ∈ ℕ there exists a session 𝒮 where state 𝑆𝑖 is an point of inevitable failure and only state 𝑆𝑘+𝑖 is the first infeasible point.

18

An unbounded session

19

Point of inevitable failure

1st infeasible point

Of length any 𝑘 ∈ ℕ

𝛾 𝑆𝑖 ∩𝑀∗ = ∅

∀𝐵 ∈ ℬ. 𝛾 𝐵 ∩ 𝛾(𝑆𝑖) = ∅

Construction

• Candidate program space 𝑀 is spanned by:• if-else• ==• lists of ints ([],[1,2], etc.)• recursive call f• the input variable i• cons• max• remove• sort• reverse

20

Construction

• Candidate program space 𝑀 is spanned by:• if-else• ==• lists of ints ([],[1,2], etc.)• recursive call f• the input variable i• cons• max• remove• sort• reverse

Example candidate:if (i==[]) []

else cons(max(i),

f(remove(i,max(i))

20

Construction

• Candidate program space 𝑀 is spanned by:• if-else• ==• lists of ints ([],[1,2], etc.)• recursive call f• the input variable i• cons• max• remove• sort• reverse

Example candidate:if (i==[]) []

else cons(max(i),

f(remove(i,max(i))

20

Example candidate:reverse(sort(i))

Construction

• Candidate program space 𝑀 is spanned by:• if-else• ==• lists of ints ([],[1,2], etc.)• recursive call f• the input variable i• cons• max• remove• sort• reverse

Example candidate:if (i==[]) []

else cons(max(i),

f(remove(i,max(i))

20

Example candidate:reverse(sort(i))

• Available predicates in 𝒫:• Input-output examples

• 𝑒𝑥𝑐𝑙𝑢𝑑𝑒(𝑒), for any program element e

Construction

Task: sort a list in descending order.

21

Construction

Input: [], Output: []Input: [1,2], Output: [2,1]

Task: sort a list in descending order.

21

Construction

Input: [], Output: []Input: [1,2], Output: [2,1]

reverse(i)

Task: sort a list in descending order.

21

Construction

Input: [], Output: []Input: [1,2], Output: [2,1]

reverse(i)

Task: sort a list in descending order.

exclude(reverse)

21

Construction

Input: [], Output: []Input: [1,2], Output: [2,1]

reverse(i)

Task: sort a list in descending order.

exclude(reverse) Point of Inevitable Failure

21

Construction

Input: [], Output: []Input: [1,2], Output: [2,1]

reverse(i)

Task: sort a list in descending order.

exclude(reverse)

if (i==[1,2])[2,1]

else i

Point of Inevitable Failure

21

Construction

if (i==[1,2])[2,1]

else i

22

Construction

Input: [1,3], Output: [3,1]

if (i==[1,2])[2,1]

else i

22

Construction

Input: [1,3], Output: [3,1]

if (i==[1,2]) [2,1]

else if (i==[1,3]) [3,1]

else i

if (i==[1,2])[2,1]

else i

22

Construction

Input: [1,3], Output: [3,1]

if (i==[1,2]) [2,1]

else if (i==[1,3]) [3,1]

else i

if (i==[1,2])[2,1]

else i

⋮

22

Construction

Input: [1,3], Output: [3,1]

if (i==[1,2]) [2,1]

else if (i==[1,3]) [3,1]

else i

if (i==[1,2])[2,1]

else i

⋮

22

Example candidate:if (i==[]) []

else cons(max(i),

f(remove(i,max(i))

Construction

Input: [1,3], Output: [3,1]

if (i==[1,2]) [2,1]

else if (i==[1,3]) [3,1]

else i

if (i==[1,2])[2,1]

else i

⋮

22

Construction

Input: [1,3], Output: [3,1]

if (i==[1,2]) [2,1]

else if (i==[1,3]) [3,1]

else i

exclude(==)

if (i==[1,2])[2,1]

else i

⋮

22

Construction

Input: [1,3], Output: [3,1]

if (i==[1,2]) [2,1]

else if (i==[1,3]) [3,1]

else i

exclude(==)

⊥

if (i==[1,2])[2,1]

else i

⋮

22

Conclusion

• An iterative, interactive model of synthesis

• An abstract domain of predicates

• Progress

• Convergence

• The unboundedness of backtracking

• We hope these results help future synthesizer designers

23