Applying the Scientific Method to Teaching Science

Remarkable effort; remarkable successDr. Douglas Duncan

University of Colorado, Boulder

We’ve been teaching for a long time…..

Today

2000 years ago

How effective are we?

How effective could we be?

FTEP – Using Clickers in the Classroom

Discipline-based Education Research @CU a $15M, 10+ yr. effort!

Faculty Collaborators: Michael Dubson PhysicsSteve Pollock “Noah Finkelstein “Kathy Perkins “X Carl Wieman “Valerie Otero EducationMike Klymkowsky BiologyxMichelle Smith “xLeilani Arthurs GeologyStefanie Molburn SociologyDoug Duncan AstronomySeth Hornstein Astronomy

X= Now are faculty at other universities 2

Ph.D. Students: Angel HoekstraChandra Turpen

Charles Baily Lauren Kost Ben Spike Kara Gray Heidi Iverson May Lee Mike Ross Robert Talbot Colin Wallace Bethany Wilcox

Postdocs: Angel Hoekstra Steve Goldhaber Laurel Mayhew Noah Podolefsky Rachel Pepper Wayne Schlingman

Dispel two common misconceptions:

1. Why do science departments have to do education research? Shouldn’t it be done by the School of Education?

• The Ed School will not teach kinematics, E & M, thermodynamic equilibrium.

• But we care how well our students understand these topics.

• So we take the education and success of our students into our own hands.

2. Don’t fall into the logical trap, “This way of teaching worked for me, it should work for my students.” That confuses necessary and sufficient.

It washes out students who aren’t like you.

Don’t expect them to all be “mini-me’s.”

Just over 50% of university students who start with a physics major switch to something else. Why?

A. They were not as good at mathB. The were less smartC. The were less motivatedD. They were less satisfied

• 7 universities and colleges• 600 hours interviews with roughly 400 science students• Surveys of thousands• Data on SAT and GPA (grade ave.)

Just over 50% of university students who start with a physics major switch to something else. Why?

A. They were not as good at mathB. The were less smartC. The were less motivatedD. They were less satisfied

• 7 universities and colleges• 600 hours interviews with roughly 400 science students• Surveys of thousands• Data on SAT and GPA (grade ave.)

Mentoring helps!

What was the #1 complaint of students who switched out of physics?

A. Too much workB. Don’t like my classmatesC. Poor teachingD. Something else

• 7 universities and colleges• 600 hours interviews with roughly 400 students• Surveys of thousands• Data on SAT and GPA (grade ave.)

What was the #1 complaint of students who switched out of physics?

A. Too much workB. Don’t like my classmatesC. Poor teachingD. Something else

• 7 universities and colleges• 600 hours interviews with roughly 400 students• Surveys of thousands• Data on SAT and GPA (grade ave.)

What was the #1 complaint of students who remained physics majors?

A. Too much workB. Don’t like my classmatesC. Poor teachingD. Something else

• 7 universities and colleges• 600 hours interviews with roughly 400 students• Surveys of thousands• Data on SAT and GPA (grade ave.)

What was the #1 complaint of students who remained physics majors?

A. Too much workB. Don’t like my classmatesC. Poor teachingD. Something else

• 7 universities and colleges• 600 hours interviews with roughly 400 students• Surveys of thousands• Data on SAT and GPA (grade ave.)

Now being redone!

If you are going to believe the results I present, you have to believe we can accurately measure

student’s understanding of science topics.

If you are going to believe the results I present, you have to believe we can accurately measure

student’s understanding of science topics.

What is the best way to measure how deep and thorough students’ understanding is? What is the best way to measure how deep and thorough students’ understanding is?

Over the last 20 years, the following methodology has been developed to study the success of science teaching and learning…

• Hundreds of interviews with students• Determine right and common wrong answers

(misconceptions) to important questions• Construct a multiple-choice test where the wrong

answers are commonly believed misconceptions• Give the test to thousands of students

Over the last 20 years, the following methodology has been developed to study the success of science teaching and learning…

• Hundreds of interviews with students• Determine right and common wrong answers

(misconceptions) to important questions• Construct a multiple-choice test where the wrong

answers are commonly believed misconceptions• Give the test to thousands of students

Force Concept Inventory (FCI)

Almost entirely conceptual

Mazur (1997; 2004)

(b) …

(b) …

conventional conceptual

49% 69%

R. Hake, ”…A six-thousand-student survey…” AJP 66, 64-74 (‘98).

<g> = post-pre 100-pre

traditional lecture

Force Concept InventoryIn a traditional lecture class, students master about 25% of the concepts (that they don’t already know).

Traditional Model of Education

Instruction viatransmissionIndividual Content

Students brains are not empty.A cautionary tale….

• about a violin….

From Carl Weiman’s*

“Physics of Everyday Life” class.

*Nobel prize winner AND good teacher

(b) Only 10% of students gave the correct answer.

Fifteen minutes later in the same lecture!

Teaching by telling is surprisingly ineffective

- if you want students to master concepts.

Students minds must be active to learn.

“Peer discussion” of conceptual questions forces students to talk and reason during class. Clickers hold them accountable.

Anything that gets students to think and

explain is good!

Consider a tiny acorn, and a giant oak tree.

An old, dead, dry log from that tree weighs 10,000x as much as the acorn.

Where does MOST of

the mass come from?

A. Sunlight

B. Water

C. Dirt

D. Minerals in the soil

E. The airOld, dead, dry logs

6 CO2 + 12 H2O → C6H12O6 + 6 O2 + 6 H2O Carbon dioxide water glucose (sugar) oxygen water

Photosynthesis

The correct answer is “The Air.”

Carbon dioxide from the air!

Notice the excitement!

You discover something new to you,

…and benefit from peer discussion.

A lot of the teaching at CU is now

like this.

<g> = post-pre 100-

pre

red = traditional, blue = interactive engagement

Physics learning at the University of Colorado leads the US!

The graph shows the fraction of everything taught students learn thoroughly during the semester.Red and blue histogram bars are for 52 classes throughout the US.

F01 F99 F04F05

S05 S04F07

© Copyright Steven Pollock 2007

Traditional lecture (popular professor)Clickers with student-student discussion

Clickers + tutorials

What difference does interactive engagement make?

This presupposes effective use of peer discussion and clickers.

Questions need to be challenging.

Student need to actively participate.

(not just listen)

- otherwise you will not realize the learning gains

I see plenty of ineffective clicker use.

Bloom’s Taxonomy

Don’t get stuck at the lowest

level of questioning!

(factual recall)

Variation in Types of QuestionsVariation in Types of Questions

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 6.5 7 7.5 8 8.5 9m

Time Elasped in mins.

Fra

cti

on

of

CQ

s w

ith

in D

elt

a t

Ra

ng

e Trad &PER instructors

More experienced instructors vary the complexity of questions.

More experienced users spend more time discussing incorrect responses

More experienced users spend more time discussing incorrect responses

0%

10%

20%

30%

40%

50%

60%

70%

80%

Trad PER PER Trad

Professor

Perc

en

t o

f q

uesti

on

s w

here

in

co

rrect

op

tio

ns w

ere

d

iscu

ssed

Most of the time individual answers are solicited before peer discussion.

It is enormously valuable to wander around and listen to your students

arguments while they debate a provocative question.

Peer instruction reveals your students’ assumptions and arguments, which often are not what you expect.

Fish is Fish…

Peer Instruction and Clicker use means more work for students than just taking notes.

It means they have to talk to the person next to them. Who they may think is an idiot.

Without explanation students protest.

Real student quote: “I expected you to teach me, I didn’t expect to have to learn!”

You must explain why they are doing this, or they will protest!

-- They should discuss what it means to learn

Metacognition

Peer instruction and clickers are relatively easy to implement.

Even upper division classes are using them.

But remember – it’s not the technology that counts, it’s what you do with it.

If you hear someone say, “Do clickers work?” “Do tutorials?” Tell them…

A general principle: The more they think about the technology, the less they think

about the science…..(cognitive load)

What the Most Successful technology implementers do:

1. Decide what we want our students to do

2. Find the best technology to encourage that behavior

e.g. “encourage discussion in class.”

Or: more personal engagement

Determine your

goal first!

Tutorials in Introductory PhysicsTutorials in Introductory Physics

Reconceptualize Recitation Sections• Materials• Classroom format / interaction• Instructional Role

Tutorial vs. Trad'l RecitationTutorial vs. Trad'l Recitation

You already know what they will find difficult!

CU-Boulder LA Model Emulation

Auburn UniversityBlack Hills State UniversityBoise StateBoston UniversityCalifornia Polytechnic StateCalifornia State University - LBCornell UniversityFlorida International UniversityIndiana University (UTA Program)James Madison (Planned 11/12)Louisiana State UniversityMarshall University (Planned 11/12)New Jersey Institute of TechnologyNorth Dakota State UniversityRutgers UniversitySeattle Pacific UniversitySouth Dakota StateTowson UniversityUNC Chapel HillUniversity of ArizonaUniversity of ArkansasUniversity of MaineUniversity of MarylandUniversity of Minnesota - St PaulUniversity of OklahomaUniversity of Texas - AustinUniversity of Texas - El PasoUtah State UniversityVirginia TechWestern Kentucky University

red = traditional, blue = interactive engagement

Physics learning at the University of Colorado leads the US!

The graph shows the fraction of everything taught students learn thoroughly during the semester.Red and blue histogram bars are for 52 classes throughout the US.

F01 F99 F04F05

S05 S04F07

© Copyright Steven Pollock 2007

Traditional lecture (popular professor)Clickers with student-student discussion

Clickers + tutorials

What difference does interactive engagement make?

Colin Wallace.

From relativity to discipline-based education research.

The professor is still key!

What student voices reveal about using clickers

It is important to use clickers often enough and to include a variety of question types

“In this class, no, they do not help us learn class material. It feels like she uses them just for attendance purposes and then doesn’t really fully go over them. It’s mostly just a waste of time… My physics professor used them very well… let us discuss them with our classmates, and then went over the right answer, thoroughly explained [the clicker question], and then told us why the other options were wrong, that really helped.”

(journalism student)

Univ. of ColoradoUpper-Level Course Transformation

Longitudinal

After upper div. E&M. (Only students who took intro without Tutorials)

Upper division majors’ BEMA scores

S. Pollock, 2007 PERC, and Phys. Rev STPER 5 (2009)

Univ. of ColoradoUpper-Level Course Transformation

Longitudinal

BLUE: students who took freshman E&M with Tutorials

Upper division majors’ BEMA scores

S. Pollock, 2007 PERC, and Phys. Rev STPER 5 (2009)

Univ. of ColoradoUpper-Level Course Transformation

Longitudinal

Yellow: students who had been E&M LAs

(3.1 ±.1) (3.0 ±.1)

(3.3 ±.1)(3.2)Grade in course

S. Pollock, 2007 PERC, and Phys. Rev STPER 5 (2009)

Upper division majors’ BEMA scores

ISSUES OF TECHNOLOGY AND “MULTI-TASKING” ARE CRITICAL, AND WILL BECOME MORE SO….

PHeT (phet.colorado.edu)

I think I’m a great multi-tasker.

Am I?

I bet the professor thinks

we’re taking notes….

Technology is in your class more than you think!

INTERVIEW QUOTES “I would say that I probably send

16 text messages in a class, definitely.”

“When you go to class, it’s a time to sit down and text.”

Sometimes people are texting all the time and it really gets annoying.

“I don’t really have any set rules as to how I conduct myself with my phone [in class]… I find text messaging is not very intrusive.

You are competing for the attention of your students. To best benefit learning, instructors should set clear policies – for on-task clicker use and off-task cell phone use, and use regular meta-narrative explaining “appropriate device behavior” to students.

Never 1-2 times 3-5 times Over 5 times

Do not own

Did not vote

Not present

60

65

70

75

80

853 Astronomy Classes (N=328)

Spring 2012

Frequency of Cell Phone Use (per class)

Fin

al G

rad

e (

perc

en

t)

Avg. Score no phone – Avg. Score phone users = 4.7 ± 1.4 %

Tamir and Mitchell (Harvard) FMRI study: talking about yourself on Facebook activates the brain’s reward system, like gambling does for many people. Your students find this hard to give up. They are “high” on it!

30% 70%

FTEP – Using Clickers in the Classroom

Canadian Study of Laptop use

Students were asked to use laptops to take notes on a lecture.

½ the students – randomly assigned – were also asked to look up stuff “when you feel you can spare some time….”

What do you think was the effect on grades?

64

Those sitting in view of a laptop did even worse!

…It’s weird sitting in class taking notes and somebody’s on Facebook looking at their pictures from a party they had two nights ago. Sometimes the guys are sitting there watching a snowboard video. I’ve seen people Skyping, I don’t know how they get away with that in class, I don’t know why you’d have just the video feed. It’s just distracting in general if somebody’s playing a game.

NOBEL LAUREATE DANIEL KAHNEMAN:

You can do several things at once, but only if they are easy and undemanding.

You have a limited budget of attention – cognitive load – and if you go beyond that you fail. It is a mark of effortful activities that … you cannot do more than one at a time.

My speculative warning: if students can multi-task in your class and succeed, in the future your teaching will be replaced by a machine.

Where is the wisdom we have lost in knowledge?

Where is the knowledge we have lost in information?

T. S. Elliot

Experts don’t just know a lot of stuff….

Experts don’t just know a lot of stuff….

• They know which knowledge to access for a given type of problem

• How to apply it• They self-check (metacognition)

• They know which knowledge to access for a given type of problem

• How to apply it• They self-check (metacognition)

Metacognition is critically important, yet often overlooked.

Thinking explicitly about reasoning, problem solving, and testing one’s ideas.

Metacognition is essential for effective learning in complex situations

A new wrinkle – deciding if you can multi-task.