Special Course in Software TechniquesDirections for Future Research

on Algorithm VisualizationAalto UniversityJune 7-23, 2011

Clifford A. ShafferDepartment of Computer Science

Virginia Tech

What makes for a good AV?

"Good AVs are all alike; every failed AV fails in its own way."

-- Leo Tolstoy

2

“State of the Field” Rationale

• Many faculty wish to use AVs

• Hundreds of visualizations are freely available on the Internet

• Studies on the effectiveness of AVs– Many studies show no significant difference– But AVs have been shown to help in some

implementations– One conclusion is that creating/using effective AVs

is possible but not easy

Research Questions

• What AVs are available?• How do educators find AVs?• Who makes them?• Is there adequate coverage?• What is their quality?• Is the field active, and improving?• Is there adequate infrastructure for storing and

disseminating AVs?

What AVs are Available?

• A collection of links available at http://algoviz.org• Links to over 500 visualizations• Nearly all AVs now written in Java

– Applets vs. applications• Stand-alone vs. collections

Who Makes Them?

• Single authors, one-off implementations (1-5)– 30%

• Small shops, sustained over a few years– Typically a faculty member and a few students– 5-10 visualizations– 10%

• Larger teams, longer term investment– Team built, maybe funded– 25%

• Major Projects– integrated package or shared look-and-feel– 35%

Is There Adequate Coverage?• No

– Sorting, search trees, and linear structures overwhelmingly dominate

– Coverage for more advanced topics is spotty

134

60

4638

29

14 13 12

4

0

20

40

60

80

100

120

140

Sort A

lgor

ithm

s

Searc

h Stru

ctur

es

Linea

r Stru

ctur

es

Graph

Algo

rithm

s

Other

Alg

orith

ms

Other

Dat

a Stru

cture

s

Compr

essio

n Alg

s

Searc

h Algo

rithm

s

Mem

ory M

anag

emen

t

What Is Their Quality?

• A majority have no pedagogical value– These give the user no understanding of how the data

structure or algorithm works– Will be of little use in the classroom

• We would recommend less than one quarter of what we have seen for any purpose

• Even the better visualizations usually have serious deficiencies– Animation only: Users are passive observers– Tree structure visualizations tend to show what

happens, but not how– Limited interactivity

Is the Field Improving?

• Pros:– A growing body of literature on best practices

to create effective AVs– Community starting to organize (AlgoViz)

• Cons:– Recent projects are no more in tune with

coverage gaps than old projects– No apparent movement in creating

repositories

Is the Field Active?• Appears to be a reduction in “one-off”

development. (Drop in student projects)– Fewer CS students– Less interest in Java– But these trends might reverse

• But steady activity in the larger groups.

AVs: The Problem• AVs have high faculty and student

favorability ratings

• But most faculty don’t use them much in courses

Informal Survey Results• Warning: Self-selected responders

• Are AVs useful?– Strongly Agree: 12– Agree: 17– Neutral: 1

• A (bare) majority indicated that they used some sort of visualization with class

Survey: Impediments to Use• Lack of knowledge/time to find good

AVs: 13

Survey: Impediments to Use• Lack of knowledge/time to find good

AVs: 13

• Time to make good AVs: 2

• Difficulty integrating in class: 9

• Lack of time within class constraints: 2

• Uncertainty about quality outcomes: 1

• Content not relevant to my classes: 1

Overcoming Impediments

• Reassurance about what AVs are good

• Ideas on how to use AVs

• Reassurance about how a given AV can be used successfully in class

• Ability to connect to developers

AVs: The Solution is Community• http://algoviz.org/

– Build a community of users/developers– Better disseminate best practices

information

• Project Support– NSF CCLI grant– NSF NSDL grant– Connections to NSDL/Ensemble project

AlgoViz.org• A collection of links to over 500 AVs

• Annotated bibliography of over 500 research papers

• Forums, field reports

• OpenAlgoViz

Phase I: The Passive Side• What problem do we solve here?

– Information? We provide that (to a limited degree)

• Nearly done!

AlgoViz Usage StatisticsTraffic on AlgoViz Portal

Month

Jul-

09

Au

g-0

9

Sep

-09

Oct

-09

No

v-09

Dec

-09

Jan

-10

Feb

-10

Mar

-10

Ap

r-10

May

-10

Jun

-10

Jul-

10

Au

g-1

0

Sep

-10

Oct

-10

No

v-10

Dec

-10

Jan

-11

Feb

-11

Vis

it

200

400

600

800

1000

1200

1400

1600

1800

2000

Portal visit countRegression line

Phase II: The Active Side• Key problem: How to help instructors

through the impediments to using AVs

• We identified community as a solution

• How do we build an online educational community?

New Paradigm for Digital Library• Create a new model of “dissemination” to

lower barriers to access• Move away from the “digital library” model

of users coming to collections• Focus on “community-driven” content

development– Discussion, review, ratings– Think Amazon, but we have critical mass issues

Community Value Added• Catalog entry ratings

• Catalog entry reviews

• Field reports

• Forum discussions

• Interactions with developers

Community Building: AlgoViz Awards

• Exemplars of good AVs

• Solicited votes from the public on a slate of nominees

• 2010: Six winners

• Hall of Fame: 6+1– Sorting out Sorting

• 2011: Two winners

Notification

• Connect to community by informing members of site changes– News posts,– New AVs, bibliography, field reports, forum posts

• Email, FaceBook, Twitter, RSS feeds• Fine grained• User control of information stream

– User profiles

Registration and Access (1)

• Site registration requirements deter users from participating

• Link spammers and other malicious entities can overwhelm a site

• So we need a balance– Minimum registration requirements possible

• Goals:– Link together a given user’s site activities– Be able to block a user

Registration and Access (2)

• Solutions:– Allow unrestricted access to view – Minimize registration burden– FaceBook connect, etc.– Initially moderate, then declare user to be “trusted”

• Social Capital– Ranking, badging

Lessons Learned

• Building site infrastructure and collection is relatively easy– Developers are motivated to provide

information

• Building community infrastructure is relatively easy

• Building community is hard

Are AVs of Pedagogical Value?

• Instructors generally think so

• Students usually say they “like” them

28

Metastudy: 2002

• Reviewed 24 prior studies on pedagogical effectiveness related to AVs– Generally of an individual system or AV

• Results of 24 studies:– 11 found significant (positive) results– 10 did not find a significant result– 2 entangled prediction with visualization– 1 study found a negative result!

29

Classification

• The studies represented a wide range of activities and methods

• Looking deeper, reclassify the independent variables:– Epistemic Fidelity: 10– Cognitive Constructivism: 14– (others too few to measure)

• CC has the highest percentage of positive studies

30

Results

• CC: 71% statistically signficant

• EF: 30% statistically significant

31

CC Activities

• Construct own input sets

• Make predictions about future states

• Program the algorithm

• Answer questions about the algorithm

• Construct own visualization

32

Level of Effort

• Compared when the two treatments required similar “cognitive effort” vs. different levels of effort– Equivalent effort: 33% significant– Not equivalent: 71% significant

• Viz vs. reading, exercises maybe on either side• Construct algorithm/visualization takes time?• Time vs. effort vs. engagement considerations?

33

Procedural vs. Conceptual Knowledge

• Procedural only: 67% [10/15]

• Procedural and Conceptual: 67% [2/3]

• Conceptual only: 38% [3/8]

34

Study Measures

• Post-test only: 54%

• Pre- to Post-test difference: 78%– But most of these studies came from one

source

35

Study Conclusions

• How students use AV is more important than what they see

• Pre-test/post-test experiments on procedural knowledge show most improvement

• Technology is effective when it is used for active engagment

36

Engagement Taxonomy

• Naps Working Group 2002– No viewing– Viewing– Responding– Changing– Constructing– Presenting

• Relates to Bloom’s Taxonomy37

Extended Engagement Taxonomy

• Myller, et al.– No viewing* (textbook)– Viewing* (video)– Controlled Viewing (slideshow)– Entering Input (Define the input to execute)– Responding* (answer questions)– Changing* (direct manipulation)– Modifying (??)– Constructing* (create the AV)– Presenting* (Teach the material)– Reviewing (Give a review of AV)

38

List of Features

From Saraiya, et al. (2004)• Ease of use (one time only)• Appropriate feedback• State changes made explicit• Multiple views• User control• Canned examples vs. data input• Pseudocode

2009 EvaluationUrquiza-Fuentes/Velazquez-Iturbide

• 33 successful evaluations• Evaluation:

– Usability

– Learning outcomes

• Many studies compared Viewing, Changing, or Constructing vs. Non-Viewing

• A few compared Changing or Constructing vs. Viewing

• Learning improvements in 75% of studies40

What is a Hypertextbook?

• It means different things to different people• We mean a close integration of three aspects:

– Static text and graphics– Dynamic content: animated slideshows, interactive activities

and visualizations, simulations, etc.– Assessment activities: questions, proficiency exercises,

programming exercises, etc., with immediate feedback on progress

• If it does not have lots of these three things working together, then we do not consider it to be a hypertextbook.

Why Hypertextbooks?

• Improve exposition– Take advantage of the computer to depict dynamic

action

• Increase engagement– Take advantage of the computer to support interactive

learning with feedback

Data Structures and Algorithms

• In particular, Data Structures and Algorithms topics are good candidates for hypertextbook treatment– Students have trouble “getting” this material because it

is inherently dynamic, but often portrayed with static methods.

43

What’s new?

• In one sense, nothing– People have wanted this for decades, and often

in much the same terms– But they haven’t made it yet

• We have some advantages– Past experience– Technology: right place at right time– A plan for getting the necessary resources

Won’t this Fail for the Same Reason that AVs Fail?

• AVs have been something of a disappointment– AVs are everywhere these days– Instructors and students “like” them– But instructors do not use them so much

• Impediments– Lack of knowledge about good AVs– Hard to get them integrated into classes

Hypertextbooks vs. AVs

• Conclusion: It is easier to put a full unit into your course than to “fit in” a new form of pedagogy

• The problems of AVs can be overcome by:– A well-known set of resources

– Complete resources, ready to plug into a course

– Either complete topics or whole courses

– Mechanisms to pick an choose what you want, and to make modifications if necessary

From Marc Brown’s Dissertation

Much of the success of the BALSA system at Brown is due to the tight integration of its development with the development of a textbook and curriculum for a particular course. BALSA was more than a resource for that course – the course was rendered in software in the BALSA system.

Prior Work (1)

• Content efforts:– Ginder et al (2002): Theory of computation

– Rodger: JFLAP software supplements standard textbook

– Crescenzi, et al (2007): courseware, tied to a textbook

– JHAVE, TRAKLA2, Animal all include tutorials for various topics

– Titterton, et al (2010): Lab-centric courseware for CS1

– Alharbi et al. (2010): Operating Systems interactivities and questions

– Kim and Jung (2010): Korea eTextbooks

Prior Work (2)

• Definition and Infrastructure• Ross (2008): Perl/Dreamweaver: a few chapters

• Karavirta (2009): Xaal

• Roling and Vellaramkalayil (2009): AVs and Moodle�• Rossling et al (2006): ITiCSE Working Group on �

VisCoSH

Case Study

• Virginia Tech Hashing• Tutorialhttp://research.cs.vt.edu/AVresearch/hashing

• In each of two semesters– One section with lecture/textbook

– One section with “closed lab” tutorial

• ANOVA shows significant difference between treatment groups, in favor of the online tutorial

• This gives some evidence that a hypertextbook can work, pedagogically

Technology

• A potential objection to our project is the fact that prior efforts haven’t succeeded due to technical obstacles

• One advantage now is that technology has advanced in significant ways– Ubiquitous access to computing: laptops,

mobile devices, Internet, classrooms– HTML5

Project Vision

• Create a hypertextbook– text/graphics

– dynamic/interactive exposition

– assessment with feedback

• Complete materials for Data Structures and Algorithms courses

• Creative Commons: Pick and Choose

• Broad contribution from AV community

Levels of Interaction

• Text and Graphics• “Animated slideshow”

– for dynamic exposition• Interactive proficiency exercises

– TRAKLA inspired• Interactive simulations/explorations

– From probing equations to running performance simulations

• Assessment questions– Keep student engaged and on track

Creating the Hypertextbook

We imagine a multistage process

1.Get an initial body of text/graphics up (this is easy since we have a textbook)

2.Build the Storyboard

3.Convert the Storyboard to the actual hypertextbook

The Storyboard

• The storyboard carefully details every point where an AV or other interactive/dynamic piece is needed, along with every assessment activity

• Begin by augmenting basic text with text specifications

• Over time, augment these specifications with existing AVs, mockups, whatever works

• Gradually replace the mockups with real implementations

• The Storyboard is a management tool, recruiting tool, and progress guide

Implementation Considerations for Dynamic and Interactive Content

• Animated slideshows could be implemented using many standard packages– PowerPoint, Beamer, OO Impress, KeyNote

• None are portable

• Conversion to PDF doesn’t really work for animation

• Flash and Java need plugins– Which leaves out iPad, mobiles

• HTML5/JavaScript appears to solve all of our technical problems

Implementation Considerations for Creative Commons

• Connexions (http://cnx.org)

• LeMill (http://lemill.net)

• Both are large creative commons for educational resources with infrastructure for authoring and reuse of material

• We have plenty of text!

Assessment Activities

• Questions– Multiple choice, short answer, etc– Could be occasional, could be at end of section

• TRAKLA-style proficiency exercises• VILLE-style questions

– Write a function– Trace a program

• How to assess correctness?

Implementation Considerations for Assessment

• This is going to be hard!

• What we need:– Storing, selecting, retrieving, and delivering

questions– Processing student answers– Storing, organizing by class, and retrieving

student assessment results– Making decisions about progress

Management Considerations

• Will need a lot of contributors

• Will need review mechanisms for contributed material

• Will need a process for handling derived/modified works

AV Development Library

• A nice technical issue ;)

• A set of functions to help developers build AVs

• Its existence will help consistency in look-and-feel