Applying the Scientific Method to Teaching Science
Remarkable effort; remarkable successDr. Douglas Duncan
University of Colorado, Boulder
We’ve been teaching for a long time…..
Today
2000 years ago
How effective are we?
How effective could we be?
FTEP – Using Clickers in the Classroom
Discipline-based Education Research @CU a $15M, 10+ yr. effort!
Faculty Collaborators: Michael Dubson PhysicsSteve Pollock “Noah Finkelstein “Kathy Perkins “X Carl Wieman “Valerie Otero EducationMike Klymkowsky BiologyxMichelle Smith “xLeilani Arthurs GeologyStefanie Molburn SociologyDoug Duncan AstronomySeth Hornstein Astronomy
X= Now are faculty at other universities 2
Ph.D. Students: Angel HoekstraChandra Turpen
Charles Baily Lauren Kost Ben Spike Kara Gray Heidi Iverson May Lee Mike Ross Robert Talbot Colin Wallace Bethany Wilcox
Postdocs: Angel Hoekstra Steve Goldhaber Laurel Mayhew Noah Podolefsky Rachel Pepper Wayne Schlingman
Dispel two common misconceptions:
1. Why do science departments have to do education research? Shouldn’t it be done by the School of Education?
• The Ed School will not teach kinematics, E & M, thermodynamic equilibrium.
• But we care how well our students understand these topics.
• So we take the education and success of our students into our own hands.
2. Don’t fall into the logical trap, “This way of teaching worked for me, it should work for my students.” That confuses necessary and sufficient.
It washes out students who aren’t like you.
Don’t expect them to all be “mini-me’s.”
Just over 50% of university students who start with a physics major switch to something else. Why?
A. They were not as good at mathB. The were less smartC. The were less motivatedD. They were less satisfied
• 7 universities and colleges• 600 hours interviews with roughly 400 science students• Surveys of thousands• Data on SAT and GPA (grade ave.)
Just over 50% of university students who start with a physics major switch to something else. Why?
A. They were not as good at mathB. The were less smartC. The were less motivatedD. They were less satisfied
• 7 universities and colleges• 600 hours interviews with roughly 400 science students• Surveys of thousands• Data on SAT and GPA (grade ave.)
Mentoring helps!
What was the #1 complaint of students who switched out of physics?
A. Too much workB. Don’t like my classmatesC. Poor teachingD. Something else
• 7 universities and colleges• 600 hours interviews with roughly 400 students• Surveys of thousands• Data on SAT and GPA (grade ave.)
What was the #1 complaint of students who switched out of physics?
A. Too much workB. Don’t like my classmatesC. Poor teachingD. Something else
• 7 universities and colleges• 600 hours interviews with roughly 400 students• Surveys of thousands• Data on SAT and GPA (grade ave.)
What was the #1 complaint of students who remained physics majors?
A. Too much workB. Don’t like my classmatesC. Poor teachingD. Something else
• 7 universities and colleges• 600 hours interviews with roughly 400 students• Surveys of thousands• Data on SAT and GPA (grade ave.)
What was the #1 complaint of students who remained physics majors?
A. Too much workB. Don’t like my classmatesC. Poor teachingD. Something else
• 7 universities and colleges• 600 hours interviews with roughly 400 students• Surveys of thousands• Data on SAT and GPA (grade ave.)
Now being redone!
If you are going to believe the results I present, you have to believe we can accurately measure
student’s understanding of science topics.
If you are going to believe the results I present, you have to believe we can accurately measure
student’s understanding of science topics.
What is the best way to measure how deep and thorough students’ understanding is? What is the best way to measure how deep and thorough students’ understanding is?
Over the last 20 years, the following methodology has been developed to study the success of science teaching and learning…
• Hundreds of interviews with students• Determine right and common wrong answers
(misconceptions) to important questions• Construct a multiple-choice test where the wrong
answers are commonly believed misconceptions• Give the test to thousands of students
Over the last 20 years, the following methodology has been developed to study the success of science teaching and learning…
• Hundreds of interviews with students• Determine right and common wrong answers
(misconceptions) to important questions• Construct a multiple-choice test where the wrong
answers are commonly believed misconceptions• Give the test to thousands of students
Force Concept Inventory (FCI)
Almost entirely conceptual
R. Hake, ”…A six-thousand-student survey…” AJP 66, 64-74 (‘98).
<g> = post-pre 100-pre
traditional lecture
Force Concept InventoryIn a traditional lecture class, students master about 25% of the concepts (that they don’t already know).
Students brains are not empty.A cautionary tale….
• about a violin….
From Carl Weiman’s*
“Physics of Everyday Life” class.
*Nobel prize winner AND good teacher
Teaching by telling is surprisingly ineffective
- if you want students to master concepts.
Students minds must be active to learn.
“Peer discussion” of conceptual questions forces students to talk and reason during class. Clickers hold them accountable.
Anything that gets students to think and
explain is good!
Consider a tiny acorn, and a giant oak tree.
An old, dead, dry log from that tree weighs 10,000x as much as the acorn.
Where does MOST of
the mass come from?
A. Sunlight
B. Water
C. Dirt
D. Minerals in the soil
E. The airOld, dead, dry logs
6 CO2 + 12 H2O → C6H12O6 + 6 O2 + 6 H2O Carbon dioxide water glucose (sugar) oxygen water
Photosynthesis
The correct answer is “The Air.”
Carbon dioxide from the air!
Notice the excitement!
You discover something new to you,
…and benefit from peer discussion.
A lot of the teaching at CU is now
like this.
<g> = post-pre 100-
pre
red = traditional, blue = interactive engagement
Physics learning at the University of Colorado leads the US!
The graph shows the fraction of everything taught students learn thoroughly during the semester.Red and blue histogram bars are for 52 classes throughout the US.
F01 F99 F04F05
S05 S04F07
© Copyright Steven Pollock 2007
Traditional lecture (popular professor)Clickers with student-student discussion
Clickers + tutorials
What difference does interactive engagement make?
This presupposes effective use of peer discussion and clickers.
Questions need to be challenging.
Student need to actively participate.
(not just listen)
- otherwise you will not realize the learning gains
I see plenty of ineffective clicker use.
Variation in Types of QuestionsVariation in Types of Questions
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 5.5 6 6.5 7 7.5 8 8.5 9m
Time Elasped in mins.
Fra
cti
on
of
CQ
s w
ith
in D
elt
a t
Ra
ng
e Trad &PER instructors
More experienced instructors vary the complexity of questions.
More experienced users spend more time discussing incorrect responses
More experienced users spend more time discussing incorrect responses
0%
10%
20%
30%
40%
50%
60%
70%
80%
Trad PER PER Trad
Professor
Perc
en
t o
f q
uesti
on
s w
here
in
co
rrect
op
tio
ns w
ere
d
iscu
ssed
Most of the time individual answers are solicited before peer discussion.
It is enormously valuable to wander around and listen to your students
arguments while they debate a provocative question.
Peer instruction reveals your students’ assumptions and arguments, which often are not what you expect.
Peer Instruction and Clicker use means more work for students than just taking notes.
It means they have to talk to the person next to them. Who they may think is an idiot.
Without explanation students protest.
Real student quote: “I expected you to teach me, I didn’t expect to have to learn!”
You must explain why they are doing this, or they will protest!
-- They should discuss what it means to learn
Metacognition
Peer instruction and clickers are relatively easy to implement.
Even upper division classes are using them.
But remember – it’s not the technology that counts, it’s what you do with it.
If you hear someone say, “Do clickers work?” “Do tutorials?” Tell them…
A general principle: The more they think about the technology, the less they think
about the science…..(cognitive load)
What the Most Successful technology implementers do:
1. Decide what we want our students to do
2. Find the best technology to encourage that behavior
e.g. “encourage discussion in class.”
Or: more personal engagement
Determine your
goal first!
Tutorials in Introductory PhysicsTutorials in Introductory Physics
Reconceptualize Recitation Sections• Materials• Classroom format / interaction• Instructional Role
Tutorial vs. Trad'l RecitationTutorial vs. Trad'l Recitation
You already know what they will find difficult!
CU-Boulder LA Model Emulation
Auburn UniversityBlack Hills State UniversityBoise StateBoston UniversityCalifornia Polytechnic StateCalifornia State University - LBCornell UniversityFlorida International UniversityIndiana University (UTA Program)James Madison (Planned 11/12)Louisiana State UniversityMarshall University (Planned 11/12)New Jersey Institute of TechnologyNorth Dakota State UniversityRutgers UniversitySeattle Pacific UniversitySouth Dakota StateTowson UniversityUNC Chapel HillUniversity of ArizonaUniversity of ArkansasUniversity of MaineUniversity of MarylandUniversity of Minnesota - St PaulUniversity of OklahomaUniversity of Texas - AustinUniversity of Texas - El PasoUtah State UniversityVirginia TechWestern Kentucky University
red = traditional, blue = interactive engagement
Physics learning at the University of Colorado leads the US!
The graph shows the fraction of everything taught students learn thoroughly during the semester.Red and blue histogram bars are for 52 classes throughout the US.
F01 F99 F04F05
S05 S04F07
© Copyright Steven Pollock 2007
Traditional lecture (popular professor)Clickers with student-student discussion
Clickers + tutorials
What difference does interactive engagement make?
What student voices reveal about using clickers
It is important to use clickers often enough and to include a variety of question types
“In this class, no, they do not help us learn class material. It feels like she uses them just for attendance purposes and then doesn’t really fully go over them. It’s mostly just a waste of time… My physics professor used them very well… let us discuss them with our classmates, and then went over the right answer, thoroughly explained [the clicker question], and then told us why the other options were wrong, that really helped.”
(journalism student)
Univ. of ColoradoUpper-Level Course Transformation
Longitudinal
After upper div. E&M. (Only students who took intro without Tutorials)
Upper division majors’ BEMA scores
S. Pollock, 2007 PERC, and Phys. Rev STPER 5 (2009)
Univ. of ColoradoUpper-Level Course Transformation
Longitudinal
BLUE: students who took freshman E&M with Tutorials
Upper division majors’ BEMA scores
S. Pollock, 2007 PERC, and Phys. Rev STPER 5 (2009)
Univ. of ColoradoUpper-Level Course Transformation
Longitudinal
Yellow: students who had been E&M LAs
(3.1 ±.1) (3.0 ±.1)
(3.3 ±.1)(3.2)Grade in course
S. Pollock, 2007 PERC, and Phys. Rev STPER 5 (2009)
Upper division majors’ BEMA scores
ISSUES OF TECHNOLOGY AND “MULTI-TASKING” ARE CRITICAL, AND WILL BECOME MORE SO….
PHeT (phet.colorado.edu)
I think I’m a great multi-tasker.
Am I?
I bet the professor thinks
we’re taking notes….
Technology is in your class more than you think!
INTERVIEW QUOTES “I would say that I probably send
16 text messages in a class, definitely.”
“When you go to class, it’s a time to sit down and text.”
Sometimes people are texting all the time and it really gets annoying.
“I don’t really have any set rules as to how I conduct myself with my phone [in class]… I find text messaging is not very intrusive.
You are competing for the attention of your students. To best benefit learning, instructors should set clear policies – for on-task clicker use and off-task cell phone use, and use regular meta-narrative explaining “appropriate device behavior” to students.
Never 1-2 times 3-5 times Over 5 times
Do not own
Did not vote
Not present
60
65
70
75
80
853 Astronomy Classes (N=328)
Spring 2012
Frequency of Cell Phone Use (per class)
Fin
al G
rad
e (
perc
en
t)
Avg. Score no phone – Avg. Score phone users = 4.7 ± 1.4 %
Tamir and Mitchell (Harvard) FMRI study: talking about yourself on Facebook activates the brain’s reward system, like gambling does for many people. Your students find this hard to give up. They are “high” on it!
30% 70%
FTEP – Using Clickers in the Classroom
Canadian Study of Laptop use
Students were asked to use laptops to take notes on a lecture.
½ the students – randomly assigned – were also asked to look up stuff “when you feel you can spare some time….”
What do you think was the effect on grades?
64
…It’s weird sitting in class taking notes and somebody’s on Facebook looking at their pictures from a party they had two nights ago. Sometimes the guys are sitting there watching a snowboard video. I’ve seen people Skyping, I don’t know how they get away with that in class, I don’t know why you’d have just the video feed. It’s just distracting in general if somebody’s playing a game.
NOBEL LAUREATE DANIEL KAHNEMAN:
You can do several things at once, but only if they are easy and undemanding.
You have a limited budget of attention – cognitive load – and if you go beyond that you fail. It is a mark of effortful activities that … you cannot do more than one at a time.
My speculative warning: if students can multi-task in your class and succeed, in the future your teaching will be replaced by a machine.
Where is the wisdom we have lost in knowledge?
Where is the knowledge we have lost in information?
T. S. Elliot
Experts don’t just know a lot of stuff….
Experts don’t just know a lot of stuff….
• They know which knowledge to access for a given type of problem
• How to apply it• They self-check (metacognition)
• They know which knowledge to access for a given type of problem
• How to apply it• They self-check (metacognition)