Testing for Educational Gaming and Educational Gaming for Testing

transcript

Collaborators: Microsoft Research RiSE (Nikolai Tillmann, Jonathan de Halleux)

Microsoft Research Connections (Judith Bishop)

Testing for Educational Gaming andEducational Gaming for Testing

Tao XieNorth Carolina State University

contact: taoxie@gmail.com

Testing Educational Gaming

Dynamic Symbolic Execution (Pex)Parameterized Unit Testing

secret

Pex for Fun (Pex4Fun)

Training

Support

http://research.microsoft.com/pex/ http://pex4fun.com

Agenda

BackgroundPex, Dynamic Symbolic Execution,Parameterized Unit Testing

Writing Code in a Browser

Coding Duels

Educational Platform

Conclusion

PEX, DYNAMIC SYMBOLIC EXECUTION,PARAMETERIZED UNIT TESTING

Background

http://research.microsoft.com/pex/

void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}

Dynamic Symbolic Execution in Pex

http://pex4fun.com/HowDoesPexWork

Ta==null

Execute&Monitor

Observed constraints

a==null

Ta==null

Constraints to solve

a!=null

Execute&Monitor

Choose next path

a==null

Ta==null

a!=null

null{}

Execute&MonitorSolve

Choose next path

a==null

a.Length>0F

TFa==null

a!=null

null{}

Choose next path

a==nulla!=null &&!(a.Length>0)

a.Length>0F

TFa==null

a!=null a!=null &&a.Length>0

null{}

Choose next path

a.Length>0

a[0]==123…

Fa==null

null{}

Choose next path

a.Length>0

a[0]==123…

Fa==null

null{}

Choose next path

a==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890

a.Length>0

a[0]==123…

Fa==null

a!=null a!=null &&a.Length>0 a!=null &&a.Length>0 &&a[0]==123456890

null{}

Choose next path

a.Length>0

a[0]==123…

Fa==null

null{}

{123…}

Choose next path

a.Length>0

a[0]==123…

Fa==null

null{}

{123…}

Choose next path

a==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890a==null &&a.Length>0 &&a[0]==1234567890

a.Length>0

a[0]==123…

Fa==null

null{}

{123…}

Choose next path

a==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890a==null &&a.Length>0 &&a[0]==1234567890

Done: There is no path left.

Vision: Parameterized Unit Testing

void AddAndCountTest(List list, int item) { var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count);}

for any list,for any item,

… adding 1 item increases Count by 1

void AddAndCountTest(List list, int item) { Assume.IsTrue(list != null); var count = list.Count; list.Add(item); Assert.AreEqual(count + 1, list.Count);}

Parameterized Unit Test = Unit Test with Parameters

Separation of concernsSpecification of behaviorData to achieve coverage

Parameterized Unit Testing in PracticeGetting Popular

Parameterized Unit Tests (PUTs) commonly supported by various test frameworks

• .NET: Supported by .NET test frameworks• Java: Supported by JUnit 4.X

Generating test inputs for PUTs supported by tools• .NET: Supported by Microsoft Research Pex• Java: Supported by Agitar AgitarOne

(http://www.agitar.com/)

http://pex4fun.com/CoverMe

Limitations

16K characters of code (single file / editor window)*

Single-threaded only

No environment interactions*

No non-determinism*

Try to avoid floating-point computations

Max. 20s total exploration time*

* Limitation only applies to pex4fun.com, but not standalone Pex tool

Dynamic Symbolic ExecutionSummary

“Ask Pex” sends code to cloud

Code is compiled and analyzed in cloud

Dynamic Symbolic Execution automatically finds relevant interesting test inputs that achieve high code coverage

Results are shown in browser

WRITING CODEIN A BROWSER

Code Auto-Completion

Auto-Completion

Writing Code in BrowserSummary

Code is compiled in cloud

Code is executed in cloud

Auto-completion via cloud/Javascript in browser

CODING DUELSFun and Engaging Serious Game – Win by Writing Code

Coding Duels

1,168,577 clicked 'Ask Pex!'

Coding Duels

Pex computes “semantic diff” in cloudcode written in browser vs.secret reference implementationYou win when Pex finds no differences

secret

Behind the Scene of Pex for Fun

Secret Implementation

class Secret { public static int Puzzle(int x) { if (x <= 0) return 1; return x * Puzzle(x-1); }}

Player Implementation

class Player { public static int Puzzle(int x) { return x; }}

class Test {public static void Driver(int x) { if (Secret.Puzzle(x) != Player.Puzzle(x)) throw new Exception(“Mismatch”); }}

behaviorSecret Impl == Player Impl

Coding DuelsFun and Engaging

Iterative gameplayAdaptivePersonalizedNo cheatingClear winning criterion

Example User Feedback

“It really got me *excited*. The part that got me most is about spreading interest in teaching CS: I do think that it’s REALLY great for teaching | learning!”

“I used to love the first person shooters and the satisfaction of blowing away a whole team of Noobies playing Rainbow Six, but this is far more fun.”

“I’m afraid I’ll have to constrain myself to spend just an hour or so a day on this really exciting stuff, as I’m really stuffed with work.”

Released since 2010

Coding Duel Competition @ICSE 2011

http://pexforfun.com/icse2011

Coding DuelsSummary

Coding Duel: write function that matches specification given by another implementation

Semantic equivalence checked in cloud

Student’s tasks:

Infer likely intended behavior

Express behavior as code

http://pex4fun.com/CreatingAndPublishingCodingDuels http://pex4fun.com/TipsAndTricksForGreatDuels

SOCIAL EXPERIENCEConnecting Students – Competitive yet Self-Paced Environment

Social Experience

Social ExperienceSummary

CommunityHigh score lists, leaderboardLive feed

http://pex4fun.com/Community.aspxhttp://pex4fun.com/Livefeed.aspx

PEX4FUN, AN EDUCATIONAL PLATFORMPages – Courses – Automated Assessments

Teaching and Learning

Coding Duels for Automatic Grading @Grad Software Engineering Course

http://pexforfun.com/gradsofteng

Coding Duels for Training Testingpublic static string Puzzle(int[] elems, int capacity, int elem) { if ((maxsize <= 0) || (elems == null) || (elems.Length > (capacity + 1))) return "Assumption Violation!"; Stack s= new Stack(capacity); for (int i = 0; i < elems.Length; i++) s.Push(elems[i]); int origSize = s.GetNumOfElements(); //Please fill in below test scenario on the s stack

//The lines below include assertions to assert the program behavior PexAssert.IsTrue(s.GetNumOfElements() == origSize + 1); PexAssert.IsTrue(s.Top() == elem); PexAssert.IsTrue(!s.IsEmpty()); PexAssert.IsTrue(s.IsMember(elem)); return s.GetNumOfElements().ToString() + "; “ + s.Top().ToString() + "; “ + s.IsMember(elem).ToString() + "; " + s.IsEmpty(); }

Set up a stack with some elements

Cache values used in assertions

Usage Scenarios of Pex4Fun• Massive Open Online Courses (MOOC): Challenges– Grading, addressed by Pex4Fun– Cheating [Open Challenge]

• Course assignments (students/professionals)– E.g., intro programming, software engineering

• Student/professional competitions– E.g., coding-duel competition at ICSE 2011

• Assessment of testing/programming/problem solving skills for job applicants– Not just final results of problem solving but also process!

Teaching and LearningSummary

How to become a teacher:Sign in, choose nicknameSend us email with nickname

Teachers canCreate coursesReuse existing or author existing pagesGet automated assessments of studentsSee students code

http://pex4fun.com/TeachingWithPexForFun

Outlook

Analyzing code submissionsHow to guide students?Detect plagiarismFurther ranking

Support for more languages

More ReadingNikolai Tillmann, Jonathan De Halleux, Tao Xie, Sumit Gulwani and Judith BishopTeaching and Learning Programming and Software Engineering via Interactive GamingIn Proceedings of the 35th International Conference on Software Engineering (ICSE 2013), Software Engineering Education (SEE), San Francisco, CA, May 2013.http://people.engr.ncsu.edu/txie/publications/icse13see-pex4fun.pdf

ConclusionPexTest generation for .NETbased on dynamic symbolic executionEnables Parameterized Unit Testing

Pex4Fun on the web.NET: C#, Visual Basic, F#Execution, code analysis with PexCoding DuelsGreat for teaching, learning

http://pex4fun.comhttp://research.microsoft.com/pex

Testing Educational Gaming

secret

Training

Support

http://research.microsoft.com/pex/ http://pex4fun.com

Q & AThank you!

contact: taoxie@gmail.com

Acknowledgments: NSF grants CCF-0845272, CCF-0915400, CNS-0958235, CNS-1160603, a Microsoft Research SEIF Award, and a Microsoft Research Award.