Post on 31-Mar-2015
transcript
Tutorial
Pex4Fun:Teaching and Learning
Computer Sciencevia Social Gaming
http://pex4fun.com
Nikolai Tillmann, Jonathan de Halleux, Judith Bishop, Michal Moskal
Microsoft Research
Tao XieNCSU
http://bit.ly/ioa4qf
Pex4Fun:Teaching and Learning
Computer Sciencevia Social Gaming
http://pex4fun.comAgenda
• Background:Code Analysis byDynamic Symbolic Execution
• Writing Code in a Browser• Coding Duels• Social Experience• Teaching Platform• Conclusion
DYNAMIC SYMBOLIC EXECUTIONIN THE CLOUD
Pex – The Pex4Fun Engine – History
Before the cloud…
http://research.microsoft.com/pex/
void CoverMe(int[] a){ if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug");}
a.Length>0
a[0]==123…
TF
T
F
Fa==null
T
Constraints to solve
a!=null a!=null &&a.Length>0
a!=null &&a.Length>0 &&a[0]==123456890
Input
null{}
{0}
{123…}
Execute&MonitorSolve
Choose next path
Observed constraints
a==nulla!=null &&!(a.Length>0)a==null &&a.Length>0 &&a[0]!=1234567890a==null &&a.Length>0 &&a[0]==1234567890
Generates test data systematically
Done: There is no path left.
Dynamic Symbolic Execution in Pex
http://pex4fun.com/HowDoesPexWork
Dynamic Symbolic Execution Exercises
• ArrayIndexLengthPex knows about all implicit, exception-throwing control-flow branches
• ArrayHeapPex models the heap
• Assert, Assert123Assertions connect code coverage and correctness
http://pex4fun.com/DynamicSymbolicExecutionExercises
Limitations
• 16K characters of code (single file / editor window) *
• Single-threaded only• No environment interactions *• No non-determinism *• Try to avoid floating-point computations
* Limitation only applies to pex4fun.com, but not Pex tool
Dynamic Symbolic ExecutionSummary
• “Ask Pex” sends code to cloud• Code is compiled and analyzed in cloud• Dynamic Symbolic Execution automatically
finds relevant interesting test inputs that achieve high code coverage
• Results are shown in browser
WRITING CODEIN A BROWSER
Code Auto-Completion
Auto-Completion
Writing Code in BrowserExercise
1. Go to http://pex4fun.com2. Click “New”3. Write some code with public static “Puzzle” method
using System;class Vector { int x, y; public Vector(int x, int y) { this.x = x; this.y = y; } public int Length() { return (int)Math.Sqrt(x*x+y*y); } public Vector Normalize() { return new Vector(x/Length(), y/Length()); } public bool Equals(Vector other) { return this.x==other.x && this.y == other.y; }}public class Program { public static void Puzzle(Vector v) { var n1 = v.Normalize(); var n2 = n1.Normalize(); if (!n1.Equals(n2)) throw new Exception(“bug in normalize”); }}
4. “Ask Pex!” http://bit.ly/lOgPZJ
http://pex4fun.com/ParameterizedUnitTesting
Writing Code in BrowserSummary
• Code is compiled in cloud• Code is executed in cloud• Auto-completion via cloud/Javascript
in browser
CODING DUELSFun and Engaging Serious Game – Win by Writing Code
Coding Duels
Coding Duels
• Pex computes “semantic diff” in cloud code written in browser vs. secret reference implementation
• You win when there are no differences
secret
Coding Duelsclass VisibleProgram { public static int Puzzle1(int x) { return x;} }class SecretProgram { public static int Puzzle2(int x) { return x * x;} }
public class MetaProgram { public static void Puzzle(int x) { if (VisibleProgram.Puzzle1(x) != SecretProgram.Puzzle2(x)) throw new System.Exception("mismatch!");} }
Coding DuelsTeaching Debugging/Problem Solving Skills
• From Hypothesis to Theorem– Step 1: Inference of hypothesized (delta) requirement• When delta: Inference of failure-inducing conditions• If difficult: forced generation of new data points
– By adding data-point-fitting conditionals
– Step 2: Writing of requirement-satisfying solution• When delta:
– Derivation of hypothesized faulty line(s) of code– Derivation of patches on the faculty line(s) of code
– Step 3: “Ask Pex” to validate the hypothesis
Initial almost-complete (delta) working implementation helps teach debugging skills
Coding DuelsTeaching Abstraction Skills
• Step 1: Inference of hypothesized (delta) requirement from observed data points– Pattern recognition– Generalization from data points to formula– Inversion of constraint solving
Teaching Programming, Algorithm, and Program Understanding Skills
• Step 2: Writing of requirement-satisfying solution
• Many choices, e.g. recursion vs. loop implementation
Teacher can give guidance:• Providing hints in comments • Providing initial almost-complete
working implementation
Coding DuelsExercise
1. Go to http://pex4fun.com; optionally: sign in with Windows Live ID2. Click “New”3. Write secret implementation with public static “Puzzle” method
using System;public class Program { public static int Puzzle(int x, int y) { return x + y – 42; }}
4. “Ask Pex!” 5. Enter Coding Duel Name6. Click “Turn This Puzzle Into A Coding Duel”7. Open Permalink8. Edit Description9. Click “Permalink”10. Use it, or create ulta-short version, e.g. with http://bit.lyhttp://pex4fun.com/CreatingAndPublishingCodingDuels
Coding DuelsFun and Engaging
• Iterative gameplay• Adaptive• Personalized• No cheating• Clear winning criterion
Coding DuelsSummary
• Coding Duel: write function that matches specification given by another implementation
• Semantic equivalence checked in cloud• Student’s tasks:– Inference of hypothesized (delta) requirement– Writing of requirement-satisfying solution
http://pex4fun.com/CreatingAndPublishingCodingDuels http://pex4fun.com/TipsAndTricksForGreatDuels
SOCIAL EXPERIENCEConnecting Students – Competitive yet Self-Paced Environment
Social Experience
Social ExperienceSummary
• Community• High score lists, leaderboard• Live feed
http://pex4fun.com/Community.aspxhttp://pex4fun.com/Livefeed.aspx
PEX4FUN, A TEACHING PLATFORMPages – Courses – Automated Assessments
Teaching
TeachingSummary
• How to become a teacher:– Sign in, choose nickname– Send us email with nickname
• Teachers can– Create courses– Reuse existing or author existing pages– Get automated assessments of students– See students code
http://pex4fun.com/TeachingWithPexForFun
Outlook I
• Analyzing code submissions– Detecting duplicate submissions– Ranking submissions (how correct, how beautiful)– If faulty, give user guidance on “the next step”.
Outlook II
• Better code development experience– Debugging– Dynamic input/output table– More languages
• Beyond the web– Phone
More ReadingNikolai Tillmann, Jonathan de Halleux, and Tao Xie
Pex for Fun: Engineering an Automated Testing Tool for Serious Games in Computer Science
Microsoft Research, Technical reportMSR-TR-2011-41, March 2011
http://research.microsoft.com/apps/pubs/?id=147143
Conclusion• Taking programming into the browser, cloud– .NET: C# (with Intellisense), Visual Basic, F#– Execution, Code analysis with Pex
• Fun learning experience with Coding Duels• Social experience: Live feed, sharing duels• Teaching with automatic grading
• Create your own courses!
http://pex4fun.com