Broader ImpactsResearch you can use

Judith OlsonUniversity of California Irvine

Today I’m going to cover…. Three recent events that inspired this

“call to arms” Your role in this discussion What it means to have impact Kinds of impact

Recap of what it means to have impact Scope, Cost, Timeline…

Your pledge about making an impact

① Collaboration Success Wizard

Theory based on The literature on teams Own own observations

and interviews of over 50▪ Science Collaborations▪ Corporate virtual teams

To verify theory Need data Online survey with advice to motivate

participation They get the help and we get the data

① Collaboration Success Wizard

Web accessible assessment tool

Assesses Strengths Challenges How to

overcome the challenges

We are having an impact

NSF Had us give a talk to Federal

funders in general “I have needed this for the last 10

years! Thank you.” Teams who were assessed

welcomed advice “It drew out patterns in the way

our members work that we were not conscious of, confirmed some of our impressions, and allowed us to hear frankly from our members.

…useful as an independent evaluation tool not tied to a funding agency or other review panel”.

② Clinical and Translational Science Awards (CTSA)

Object of study

…”to speed the translation of laboratory discoveries into treatment for patients.”

“from bench to bedside.”

National Institute of Health: 60 CTSA awards in 30 states plus DC

② Clinical and Translational Science Awards

National Institutes of HealthSince 2006$733 M

③ NSF “Dear Colleague”

“While most researchers know what is meant by Intellectual Merit, experience shows that many researchers have a lessthan clear understanding of the meaning of Broader Impacts.”

Impact

Many of us came to this field to change the digital world Technology had gone awry

Many early people attracted to HCI were “Children of the 60s”

Then…

Our careers were caught up in the reward structures Industry▪ Create new products▪ Disincentive to make

findings available to others Academia▪ Publish new findings▪ Stay on topic, build a reputation

Worry…..

Where have all the impacts gone Long time passing

How we will proceed….

I will describe what I think it means to have impact

I will list a number of ways we do and can have an impact

You pledge…

The card on your seat▪ What other ways can you have impact▪ How are you going to have an impact▪ Collected by SVs at the door as you leave

What it means to have impact What “counts”

Theory gets used Downloads/views Profits Degrees/Education Technologies Lives changed …..

Who is impacted? Students Developers Consultants

Specific populations The general public

What it means to have impact

Scopes differ You affect some people directly▪ Interventions, teaching

You enable others to be better at making better products▪ Toolkits

You set policy▪ Affect a large number of people

What it means to have impact Time scales differ

Now▪ e.g. Action research

1-3 years▪ e.g., Publications

20-30 years▪ e.g., Theory Assessment Tools

40-50 years▪ e.g., Cyberinfrastructure development

?▪ e.g., Policy (like SOPA/PIPA)

What it means to have impact

Access? Free▪ Toolkits…▪ Wizard▪ ….

Fees▪ Commercial Assessment Tools▪ Products▪ Educational degree▪ ….

Kinds of Impacts

TheoriesAssessment tools Technological innovations Guidelines, templates, patterns, toolkits

and standards Policies New media dissemination Action research Teaching and teaching materials …

What else?Which kinds of impacts will YOU make?

Theories

“There is nothing so practical as a good theory” Kurt Lewin

“He who loves practice without theory is like the sailor who boards ship

without a rudder and compass and never knows where he may cast”

Leonardo Da Vinci

Theories

Who Other researchers Consultants Tool developers

How Read and build on/test theory

Scope Small at first

Time scale 1-3 or more years

Access Free

Kinds of Impacts

TheoriesAssessment tools Technological innovations Guidelines, templates, patterns,

toolkits and standards Policies New media dissemination Action research Teaching and teaching materialsWhat else?

Which kinds of impacts will YOU make?

Theories delivered as Assessment Tools

Collaboration Success Wizard

Globesmart Myers-Briggs Personality Assessment CogTool …

Assessment Tools: GlobeSmart

Based on academic theories of cultural differences▪ David Matsumoto▪ Handbook of Culture and

Psychology Like the Wizard, they collect

data to adjust their assessments▪ Recent upgrade used data from

400,000 users from over 60 countries

Assessment Tools: Globesmart

Assessment Tools: GlobeSmart

Assessment Tools: GlobeSmart

Assessment Tools: GlobeSmart

Assessment Tools: Myers Briggs

Based on the work of Carl Jung Developed further by Myers

and Briggs

Like the GlobeSmart you can see differences in values and habits with people you interact with Dimensions of discussion Some professional help

Assessment Tools: CogTool Based on work of Bonnie

John▪ Based on Card, Moran, & Newell▪ GOMS and the Model Human

Processor

A general purpose UI prototyping tool It automatically evaluates

your design with a predictive human performance model▪ A “cognitive crash dummy”

Assessment Tools: CogTool

Assessment Tools: CogTool

▪ “You can compare expert use task time without recruiting participants…An excellent choice for completely new systems that don’t already have experts.”

Assessment Tools

Who General public

How Take the assessment

Scope Could be huge

Time scale Immediate

Access Some are free; some cost money

Kinds of Impacts

TheoriesAssessment toolsTechnological innovations Guidelines, templates, patterns,

toolkits and standards Policies New media dissemination Action research Teaching and teaching materialsWhat else?

Which kinds of impacts will YOU make?

Technical Innovations: Alice

3-D programming environment For telling a story Playing an interactive game

Teaching tool for introductory programming Formally shown to improve

learning and performance

Randy Pausch

Technical Innovations: Alice

Caitlin Kelleher,2006

Technical Innovations: Alice

Using storytelling to make computer programming attractive to middle school girls

Storytelling Alice users spent 42% more time programming were more than three times as likely

to sneak in extra time to continue working on their programs

Caitlin Kelleher

Technical Innovations: Alice

10% of the nation’s colleges now use Alice An accompanying textbook, lessons, test

banks

88% of “at risk” students who had Alice in a pre-CS1 course were retained through CS2 3.03 GPA

Which then inspired….

iMuse A requirements

engineering environment where both developers and stakeholders could understand the flow Kristina Winbladh

Technical Innovations: More...

Hypertext Transfer Protocol HTTP HTTP/1.1 spec▪ Fielding, Gettys, Mogul, Frystyk and Berners-

Lee WebDAV extension “Architecture of the Web”▪ Fielding and Taylor

Technical Innovations: More…. Aspect Oriented Programming

▪ Difference lies in the power, safetyand usability of the constructs provided

Original article downloaded6,681 times

16,600 articles in Google Scholar with “Aspect Oriented Programming”

Crista Lopes

Technical innovations

Who Students The general public Other developers

How Use the technology that makes things possible

Scope Huge

Time scale 5-10 years

Access Often free (though products cost money)

Kinds of Impacts

TheoriesAssessment tools Technological innovationsGuidelines, templates, patterns,

toolkits and standards Policies New media dissemination Action research Teaching and teaching materialsWhat else?

Which kinds of impacts will YOU make?

Guidelines, templates, and patterns

All provide conventions So there is little new to

learn

Where things go, what they look like

Sometimes task flow guide

Guidelines, templates, and patterns

What are they based on? Are they consistent?

(Human Interface Guidelines)

Guidelines, templates, and patterns

Principles, patterns and practices for improvinguse experience

Early instance:

Christian Crumlish & Erin Malone

Christopher Alexander

Guidelines, templates, and patterns

Their effectiveness depends on▪ The research they

are based on▪ The context in

which they arose▪ Their fit to the

context they are being applied to

Toolkits

UI Development environments With extra features▪ Highly interactive▪ Graphical▪ Direct manipulation▪ Automatic undo▪ Support for animation▪ Gesture recognition

Amulet - C++ Garnet – Common Lisp, X11, and Mac

Brad Myers

Standards

Keeping these up to date….

Who Developers End users

How Find and use relevant templates….

Scope Speeds development, makes software consistent

Time scale Immediate

Access Free

Guidelines, templates, patterns, toolkits and standards

Kinds of Impacts

TheoriesAssessment tools Technological innovations Guidelines, templates, patterns,

toolkits and standardsPolicies New media dissemination Action research Teaching and teaching materialsWhat else?

Which kinds of impacts will YOU make?

Policies

Stop Online Piracy Act (SOPA)/Protect Intellectual Property Act (PIPA)

Network neutrality Participatory design in Scandinavia Open access vs. commercial

production of educational materials Data sharing policies ……

Who Everyone

How Dictates what’s possible

Scope Huge

Time scale ?

Access Who gets to be in the conversation?

Policies

Kinds of Impacts

TheoriesAssessment tools Technological innovations Guidelines, templates, patterns,

toolkits and standards PoliciesNew media dissemination Action research Teaching and teaching materialsWhat else?

Which kinds of impacts will YOU make?

New Media Dissemination

Total views = 14,660,471

New Media Dissemination

All videos viewed 6,813,795 times

Hans Rosling

A Swedish medical doctor, academic, statistician and public speaker. He is Professor of International Health at Karolinska Institute[2] and co-founder and chairman of the Gapminder Foundation, which developed the Trendalyzer software system.

Who The public Students

How YouTube, TedTalks….

Scope Huge

Time scale Immediate

Access Free

New Media Dissemination

Kinds of Impacts

TheoriesAssessment tools Technological innovations Guidelines, templates, patterns,

toolkits and standards Policies New media disseminationAction research Teaching and teaching materialsWhat else?

Which kinds of impacts will YOU make?

Action Research

Helping teachers of autistic children assess behavioral incidents

Helping caretakers and clinicians of preterm infants monitor their movement and other key factors

Gilllian Hayes

Who Target population starting with a small

group How

New technologies to help critical situations

Scope Small at first, larger as results are

generalized Time scale

Immediate Access

Free

Action research

Kinds of Impacts

TheoriesAssessment tools Technological innovations Guidelines, templates, patterns,

toolkits and standards Policies New media dissemination Action researchTeaching and teaching materialsWhat else?

Which kinds of impacts will YOU make?

Teaching and Teaching Materials

Undergraduate teaching 6,970 students in a career

Ph.D. students ~40▪ A multiplier because they go on to teach

Teaching materials Books for classes Cases, exercises

Teaching and Teaching Materials

Online resourcesthat educate

UsabilityFirst.comHcibib.org

Useit.com

Teaching and Teaching Materials

Teaching or action kits

National Center for Womenin Information TechnologyNCWIT

Teaching and Teaching Materials

NCWIT

Teaching and Teaching Materials

Teaching and Teaching Materials

Teaching and Teaching Materials

David Evans &Sebastian Thrun

Who Students

How Exposed to lectures, exercises, assessments

Scope Digital media is the multiplier

Time scale 1-2 years

Access Sometimes free; sometimes requires tuition

Teaching and Teaching Materials

Kinds of Impacts

TheoriesAssessment tools Technological innovations Guidelines, templates, patterns, toolkits

and standards Policies New media dissemination Action research Teaching and teaching materials …

What else?Which kinds of impacts will YOU make?

Recap on what counts as impacts

Who is impacted How Scope Time Scale Access

Decisions you have to make…

What it means to have impact What “counts”

Theory gets used Downloads/views Profits Degrees/Education Technologies Lives changed …..

Who is impacted? Students Developers Consultants

Specific populations The general public

What it means to have impact

Scopes differ You affect some people directly▪ Interventions, teaching

You enable others to be better at making better products▪ Toolkits

You set policy▪ Affect a large number of people

What it means to have impact Time scales differ

Now▪ e.g. Action research

1-3 years▪ e.g., Publications

20-30 years▪ e.g., Theory Assessment Tools

40-50 years▪ e.g., Cyberinfrastructure development

?▪ e.g., Policy (like SOPA/PIPA)

What it means to have impact

Is it free? Yes▪ Khan Academy▪ Open Knowledge▪ Standards, toolkits, patterns▪ Wizard

No▪ Udacity▪ Meyers Briggs▪ GlobeSmart▪ Textbooks▪ Degree programs▪ Products

A continuing dialog

How to translate our research to have broader impacts?

How to guarantee quality? E.g. evidence based medicine

How to make it accessible?

How to evaluate impact?

Indirect but important impact How careers are advanced now

Product innovation Publications

Future + Impact

It takes the evaluators to change the system Promotion policy

What impact will you make?

TheoriesAssessment Tools Popular technologies that become

standards Guidelines, templates, patterns, toolkits and

standards Policies New media dissemination Action Research Teaching and teaching materials … Student Volunteers will collect on the way out

Thank you

In the interest of potential impact A video of this will appear on the ACM

website and the ACM-W website jsolson@uci.edu