+ All Categories
Home > Education > Testing for Educational Gaming and Educational Gaming for Testing

Testing for Educational Gaming and Educational Gaming for Testing

Date post: 09-May-2015
Category:
Upload: tao-xie
View: 319 times
Download: 1 times
Share this document with a friend
Description:
A talk given at Google Test Automation Conference (GTAC 2013) https://developers.google.com/google-test-automation-conference/
54
Collaborators: Microsoft Research RiSE (Nikolai Tillmann, Jonathan de Halleux) Microsoft Research Connections (Judith Bishop) Testing for Educational Gaming and Educational Gaming for Testing Tao Xie North Carolina State University contact: [email protected]
Transcript
Page 1: Testing for Educational Gaming and Educational Gaming for Testing

Collaborators: Microsoft Research RiSE (Nikolai Tillmann, Jonathan de Halleux)

Microsoft Research Connections (Judith Bishop)

Testing for Educational Gaming andEducational Gaming for Testing

Tao XieNorth Carolina State University

contact: [email protected]

Page 2: Testing for Educational Gaming and Educational Gaming for Testing

Testing Educational Gaming

Dynamic Symbolic Execution (Pex)Parameterized Unit Testing

secret

Pex for Fun (Pex4Fun)

Training

Support

http://research.microsoft.com/pex/ http://pex4fun.com

Page 3: Testing for Educational Gaming and Educational Gaming for Testing

Agenda

BackgroundPex, Dynamic Symbolic Execution,Parameterized Unit Testing

Writing Code in a Browser

Coding Duels

Educational Platform

Conclusion

Page 4: Testing for Educational Gaming and Educational Gaming for Testing

PEX, DYNAMIC SYMBOLIC EXECUTION,PARAMETERIZED UNIT TESTING

Background

Page 5: Testing for Educational Gaming and Educational Gaming for Testing

http://research.microsoft.com/pex/

Page 6: Testing for Educational Gaming and Educational Gaming for Testing

void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}

 

         

   

 

 

 

 

 

 

 

                 

Dynamic Symbolic Execution in Pex

http://pex4fun.com/HowDoesPexWork

Page 7: Testing for Educational Gaming and Educational Gaming for Testing

void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}

 

         

   

 

Input

null 

 

 

 

                 

Dynamic Symbolic Execution in Pex

http://pex4fun.com/HowDoesPexWork

Page 8: Testing for Educational Gaming and Educational Gaming for Testing

void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}

Ta==null

 

         

   

 

Input

null 

 

 

Execute&Monitor

Observed constraints

a==null               

Dynamic Symbolic Execution in Pex

http://pex4fun.com/HowDoesPexWork

Page 9: Testing for Educational Gaming and Educational Gaming for Testing

void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}

Ta==null

Constraints to solve

a!=null    

   

 

Input

null 

 

 

Execute&Monitor

Choose next path

Observed constraints

a==null               

Dynamic Symbolic Execution in Pex

http://pex4fun.com/HowDoesPexWork

Page 10: Testing for Educational Gaming and Educational Gaming for Testing

void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}

Ta==null

Constraints to solve

a!=null    

   

 

Input

null{}

 

 

Execute&MonitorSolve

Choose next path

Observed constraints

a==null               

Dynamic Symbolic Execution in Pex

http://pex4fun.com/HowDoesPexWork

Page 11: Testing for Educational Gaming and Educational Gaming for Testing

void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}

a.Length>0F

TFa==null

Constraints to solve

a!=null    

   

 

Input

null{}

 

 

Execute&MonitorSolve

Choose next path

Observed constraints

a==nulla!=null &&!(a.Length>0)           

Dynamic Symbolic Execution in Pex

http://pex4fun.com/HowDoesPexWork

Page 12: Testing for Educational Gaming and Educational Gaming for Testing

void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}

a.Length>0F

TFa==null

Constraints to solve

a!=null a!=null &&a.Length>0 

   

 

Input

null{}

 

 

Execute&MonitorSolve

Choose next path

Observed constraints

a==nulla!=null &&!(a.Length>0)           

Dynamic Symbolic Execution in Pex

http://pex4fun.com/HowDoesPexWork

Page 13: Testing for Educational Gaming and Educational Gaming for Testing

void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}

a.Length>0

a[0]==123…

TF

T

F

Fa==null

Constraints to solve

a!=null a!=null &&a.Length>0 

   

 

Input

null{}

{0}

 

Execute&MonitorSolve

Choose next path

Observed constraints

a==nulla!=null &&!(a.Length>0)           

Dynamic Symbolic Execution in Pex

http://pex4fun.com/HowDoesPexWork

Page 14: Testing for Educational Gaming and Educational Gaming for Testing

void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}

a.Length>0

a[0]==123…

TF

T

F

Fa==null

Constraints to solve

a!=null a!=null &&a.Length>0 

   

 

Input

null{}

{0}

 

Execute&MonitorSolve

Choose next path

Observed constraints

a==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890     

Dynamic Symbolic Execution in Pex

http://pex4fun.com/HowDoesPexWork

Page 15: Testing for Educational Gaming and Educational Gaming for Testing

void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}

a.Length>0

a[0]==123…

TF

T

F

Fa==null

Constraints to solve

a!=null a!=null &&a.Length>0 a!=null &&a.Length>0 &&a[0]==123456890

Input

null{}

{0}

 

Execute&MonitorSolve

Choose next path

Observed constraints

a==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890     

Dynamic Symbolic Execution in Pex

http://pex4fun.com/HowDoesPexWork

Page 16: Testing for Educational Gaming and Educational Gaming for Testing

void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}

a.Length>0

a[0]==123…

TF

T

F

Fa==null

Constraints to solve

a!=null a!=null &&a.Length>0 a!=null &&a.Length>0 &&a[0]==123456890

Input

null{}

{0}

{123…}

Execute&MonitorSolve

Choose next path

Observed constraints

a==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890     

Dynamic Symbolic Execution in Pex

http://pex4fun.com/HowDoesPexWork

Page 17: Testing for Educational Gaming and Educational Gaming for Testing

void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}

a.Length>0

a[0]==123…

TF

T

F

Fa==null

T

Constraints to solve

a!=null a!=null &&a.Length>0 a!=null &&a.Length>0 &&a[0]==123456890

Input

null{}

{0}

{123…}

Execute&MonitorSolve

Choose next path

Observed constraints

a==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890a==null &&a.Length>0 &&a[0]==1234567890

Dynamic Symbolic Execution in Pex

http://pex4fun.com/HowDoesPexWork

Page 18: Testing for Educational Gaming and Educational Gaming for Testing

void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}

a.Length>0

a[0]==123…

TF

T

F

Fa==null

T

Constraints to solve

a!=null a!=null &&a.Length>0 a!=null &&a.Length>0 &&a[0]==123456890

Input

null{}

{0}

{123…}

Execute&MonitorSolve

Choose next path

Observed constraints

a==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890a==null &&a.Length>0 &&a[0]==1234567890

Done: There is no path left.

Dynamic Symbolic Execution in Pex

http://pex4fun.com/HowDoesPexWork

Page 19: Testing for Educational Gaming and Educational Gaming for Testing

   

     

Vision: Parameterized Unit Testing

void AddAndCountTest(List list, int item) {  var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count);}

Page 20: Testing for Educational Gaming and Educational Gaming for Testing

   

     

Vision: Parameterized Unit Testing

void AddAndCountTest(List list, int item) {  var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count);}

for any list,for any item,

Page 21: Testing for Educational Gaming and Educational Gaming for Testing

   

     

Vision: Parameterized Unit Testing

void AddAndCountTest(List list, int item) {  var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count);}

… adding 1 item increases Count by 1

Page 22: Testing for Educational Gaming and Educational Gaming for Testing

   

     

Vision: Parameterized Unit Testing

void AddAndCountTest(List list, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count);}

Page 23: Testing for Educational Gaming and Educational Gaming for Testing

Parameterized Unit Test = Unit Test with Parameters

     

Vision: Parameterized Unit Testing

void AddAndCountTest(List list, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count);}

Page 24: Testing for Educational Gaming and Educational Gaming for Testing

Parameterized Unit Test = Unit Test with Parameters

Separation of concernsSpecification of behaviorData to achieve coverage

Vision: Parameterized Unit Testing

void AddAndCountTest(List list, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count);}

Page 25: Testing for Educational Gaming and Educational Gaming for Testing

Parameterized Unit Testing in PracticeGetting Popular

Parameterized Unit Tests (PUTs) commonly supported by various test frameworks

• .NET: Supported by .NET test frameworks• Java: Supported by JUnit 4.X

Generating test inputs for PUTs supported by tools• .NET: Supported by Microsoft Research Pex• Java: Supported by Agitar AgitarOne

(http://www.agitar.com/)

Page 26: Testing for Educational Gaming and Educational Gaming for Testing

http://pex4fun.com/CoverMe

Page 27: Testing for Educational Gaming and Educational Gaming for Testing

Limitations

16K characters of code (single file / editor window)*

Single-threaded only

No environment interactions*

No non-determinism*

Try to avoid floating-point computations

Max. 20s total exploration time*

* Limitation only applies to pex4fun.com, but not standalone Pex tool

Page 28: Testing for Educational Gaming and Educational Gaming for Testing

Dynamic Symbolic ExecutionSummary

“Ask Pex” sends code to cloud

Code is compiled and analyzed in cloud

Dynamic Symbolic Execution automatically finds relevant interesting test inputs that achieve high code coverage

Results are shown in browser

Page 29: Testing for Educational Gaming and Educational Gaming for Testing

WRITING CODEIN A BROWSER

Code Auto-Completion

Page 30: Testing for Educational Gaming and Educational Gaming for Testing

Auto-Completion

Page 31: Testing for Educational Gaming and Educational Gaming for Testing

Writing Code in BrowserSummary

Code is compiled in cloud

Code is executed in cloud

Auto-completion via cloud/Javascript in browser

Page 32: Testing for Educational Gaming and Educational Gaming for Testing

CODING DUELSFun and Engaging Serious Game – Win by Writing Code

Page 33: Testing for Educational Gaming and Educational Gaming for Testing

Coding Duels

1,168,577 clicked 'Ask Pex!'

Page 34: Testing for Educational Gaming and Educational Gaming for Testing

Coding Duels

Pex computes “semantic diff” in cloudcode written in browser vs.secret reference implementationYou win when Pex finds no differences

secret

Page 35: Testing for Educational Gaming and Educational Gaming for Testing

Behind the Scene of Pex for Fun

Secret Implementation

class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); }}

Player Implementation

class Player { public static int Puzzle(int x) { return x; }}

class Test {public static void Driver(int x) { if (Secret.Puzzle(x) != Player.Puzzle(x)) throw new Exception(“Mismatch”); }}

behaviorSecret Impl == Player Impl

18

Page 36: Testing for Educational Gaming and Educational Gaming for Testing

Behind the Scene of Pex for Fun

Secret Implementation

class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); }}

Player Implementation

class Player { public static int Puzzle(int x) { return x; }}

class Test {public static void Driver(int x) { if (Secret.Puzzle(x) != Player.Puzzle(x)) throw new Exception(“Mismatch”); }}

behaviorSecret Impl == Player Impl

18

Page 37: Testing for Educational Gaming and Educational Gaming for Testing

Behind the Scene of Pex for Fun

Secret Implementation

class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); }}

Player Implementation

class Player { public static int Puzzle(int x) { return x; }}

class Test {public static void Driver(int x) { if (Secret.Puzzle(x) != Player.Puzzle(x)) throw new Exception(“Mismatch”); }}

behaviorSecret Impl == Player Impl

18

Page 38: Testing for Educational Gaming and Educational Gaming for Testing

Coding DuelsFun and Engaging

Iterative gameplayAdaptivePersonalizedNo cheatingClear winning criterion

Page 39: Testing for Educational Gaming and Educational Gaming for Testing

Example User Feedback

“It really got me *excited*. The part that got me most is about spreading interest in teaching CS: I do think that it’s REALLY great for teaching | learning!”

“I used to love the first person shooters and the satisfaction of blowing away a whole team of Noobies playing Rainbow Six, but this is far more fun.”

“I’m afraid I’ll have to constrain myself to spend just an hour or so a day on this really exciting stuff, as I’m really stuffed with work.”

Released since 2010

X

Page 40: Testing for Educational Gaming and Educational Gaming for Testing

Coding Duel Competition @ICSE 2011

http://pexforfun.com/icse2011

Page 41: Testing for Educational Gaming and Educational Gaming for Testing

Coding DuelsSummary

Coding Duel: write function that matches specification given by another implementation

Semantic equivalence checked in cloud

Student’s tasks:

Infer likely intended behavior

Express behavior as code

http://pex4fun.com/CreatingAndPublishingCodingDuels http://pex4fun.com/TipsAndTricksForGreatDuels

Page 42: Testing for Educational Gaming and Educational Gaming for Testing

SOCIAL EXPERIENCEConnecting Students – Competitive yet Self-Paced Environment

Page 43: Testing for Educational Gaming and Educational Gaming for Testing

Social Experience

Page 44: Testing for Educational Gaming and Educational Gaming for Testing

Social ExperienceSummary

CommunityHigh score lists, leaderboardLive feed

http://pex4fun.com/Community.aspxhttp://pex4fun.com/Livefeed.aspx

Page 45: Testing for Educational Gaming and Educational Gaming for Testing

PEX4FUN, AN EDUCATIONAL PLATFORMPages – Courses – Automated Assessments

Page 46: Testing for Educational Gaming and Educational Gaming for Testing

Teaching and Learning

Page 47: Testing for Educational Gaming and Educational Gaming for Testing

Coding Duels for Automatic Grading @Grad Software Engineering Course

http://pexforfun.com/gradsofteng

Page 48: Testing for Educational Gaming and Educational Gaming for Testing

Coding Duels for Training Testingpublic static string Puzzle(int[] elems, int capacity, int elem) { if ((maxsize <= 0) || (elems == null) || (elems.Length > (capacity + 1))) return "Assumption Violation!"; Stack s= new Stack(capacity); for (int i = 0; i < elems.Length; i++) s.Push(elems[i]); int origSize = s.GetNumOfElements(); //Please fill in below test scenario on the s stack

//The lines below include assertions to assert the program behavior PexAssert.IsTrue(s.GetNumOfElements() == origSize + 1); PexAssert.IsTrue(s.Top() == elem); PexAssert.IsTrue(!s.IsEmpty()); PexAssert.IsTrue(s.IsMember(elem)); return s.GetNumOfElements().ToString() + "; “ + s.Top().ToString() + "; “ + s.IsMember(elem).ToString() + "; " + s.IsEmpty(); }

Set up a stack with some elements

Cache values used in assertions

Page 49: Testing for Educational Gaming and Educational Gaming for Testing

Usage Scenarios of Pex4Fun• Massive Open Online Courses (MOOC): Challenges– Grading, addressed by Pex4Fun– Cheating [Open Challenge]

• Course assignments (students/professionals)– E.g., intro programming, software engineering

• Student/professional competitions– E.g., coding-duel competition at ICSE 2011

• Assessment of testing/programming/problem solving skills for job applicants– Not just final results of problem solving but also process!

Page 50: Testing for Educational Gaming and Educational Gaming for Testing

Teaching and LearningSummary

How to become a teacher:Sign in, choose nicknameSend us email with nickname

Teachers canCreate coursesReuse existing or author existing pagesGet automated assessments of studentsSee students code

http://pex4fun.com/TeachingWithPexForFun

Page 51: Testing for Educational Gaming and Educational Gaming for Testing

Outlook

Analyzing code submissionsHow to guide students?Detect plagiarismFurther ranking

Support for more languages

Page 52: Testing for Educational Gaming and Educational Gaming for Testing

More ReadingNikolai Tillmann, Jonathan De Halleux, Tao Xie, Sumit Gulwani and Judith BishopTeaching and Learning Programming and Software Engineering via Interactive GamingIn Proceedings of the 35th International Conference on Software Engineering (ICSE 2013), Software Engineering Education (SEE), San Francisco, CA, May 2013.http://people.engr.ncsu.edu/txie/publications/icse13see-pex4fun.pdf

Page 53: Testing for Educational Gaming and Educational Gaming for Testing

ConclusionPexTest generation for .NETbased on dynamic symbolic executionEnables Parameterized Unit Testing

Pex4Fun on the web.NET: C#, Visual Basic, F#Execution, code analysis with PexCoding DuelsGreat for teaching, learning

http://pex4fun.comhttp://research.microsoft.com/pex

Page 54: Testing for Educational Gaming and Educational Gaming for Testing

Testing Educational Gaming

secret

Training

Support

http://research.microsoft.com/pex/ http://pex4fun.com

Q & AThank you!

contact: [email protected]

Acknowledgments: NSF grants CCF-0845272, CCF-0915400, CNS-0958235, CNS-1160603, a Microsoft Research SEIF Award, and a Microsoft Research Award.


Recommended