Post on 14-Dec-2015
transcript
David DiBattista, Ph.D.Brock University
Psychology Department
Encouraging Higher-order Thinking Using Multiple-choice Questions
Toronto, OntarioFebruary, 2012
©D. DiBattista 2012
OVERVIEW Using MCQs for summative purposes The Revised Bloom’s Taxonomy Getting beyond remembering with MCQs Using MCQs for formative purposes $ The rationale $ Some strategies
©D. DiBattista 2012
©D. DiBattista 2012
What is Summative Assessment?
A summative assessment is a test that is given primarily:a) to assess a student’s achievement in a course, andb) to assign a grade that reflects the student’s level of achievement.
MCQs are the most commonly used form ofselected-response technique on classroom tests.
Selected quotes from the media• All who take part in MC tests are wasting their time.• MC tests are a stupid way to test what students know.• MC testing is not a good indicator of conceptual
understanding.• [MC tests] are a lazy, one-size-fits-all substitute for
actually assessing what students have learned.• Results [of MC tests] tend to depend on luck, not on
intelligence and hard work.• Multiple choice tests are games with a premium on
strategic guessing.
©D. DiBattista 2012
MC questions: Controversial!
• The real value of MC items…is their applicability in measuring higher-level objectives, such as those based in comprehension, application, and analysis. (Burton, et al., 1991)
• [Constructed-response items] take more examinee time and resources to measure essentially the same thing as [MCQs, but they do so] more poorly than the latter. (Wainer & Thissen, 1993)
©D. DiBattista 2012
But on the other hand…
Let's look at one of the most common criticisms of MCQs.
“It seems quite evident from many studies that the basic behavior evaluated (by MCQs) is pure recall of information.” (Srivastava, et al., 2004)
Overemphasis on memory-based MCQs
Poor content-related validity of the test
andPromotion of surface learning
at the expense of deep learning
©D. DiBattista 2012
We need a framework that lets us discuss this issue in depth.
OVERVIEW Using MCQs for summative purposes The Revised Bloom’s Taxonomy Getting beyond remembering with MCQs Using MCQs for formative purposes $ The rationale $ Some strategies
©D. DiBattista 2012
Factual
Knowledge Dimension
Conceptual Procedural Metacognitive
CognitiveProcess
Dimension
Remember
Understand
Apply
Analyze
Evaluate
Create
The Revised Bloom’s Taxonomy Anderson and Krathwohl, 2001
©D. DiBattista 2012
This chart is in today’s handout!
Factual
Knowledge Dimension
Conceptual Procedural Metacognitive
CognitiveProcess
Dimension
Remember
Understand
Apply
Analyze
Evaluate
Create
The Revised Bloom’s Taxonomy Anderson and Krathwohl, 2001
These are all ACTION verbs—i.e., things students can DO with their knowledge.
©D. DiBattista 2012
Good tests allow us to determine what our students are capable of.
…but they can be used effectively to assess all of the other cognitive processes.
MC questions are notuseful for assessing creativity…
Factual
Knowledge Dimension
Conceptual Procedural Metacognitive
CognitiveProcess
Dimension
Remember
Understand
Apply
Analyze
Evaluate
Create
The Revised Bloom’s Taxonomy Anderson and Krathwohl, 2001
Can you remember this?
Can you understand this?
Can you apply this?
Can you evaluate this?
Can you create this?
Can you analyze this?
©D. DiBattista 2012
These are all ACTION verbs—i.e., things students can DO with their knowledge.
Factual
Knowledge Dimension
Conceptual Procedural Metacognitive
CognitiveProcess
Dimension
Remember
Understand
Apply
Analyze
Evaluate
Create
The Revised Bloom’s Taxonomy Anderson and Krathwohl, 2001
©D. DiBattista 2012
Some thoughts about REMEMBER…
ALL assessment tasks involve usingmemory to at least some degree.
BUT“If assessment tasks are to tap higher-order cognitive processes, they must require that students cannot answer them correctly by relying on memory ALONE.”
—Anderson and Krathwohl, 2001, page 71
A simple, unfortunate fact:Creating MC items that rely on memory alone is far easier than creating higher-level items.
©D. DiBattista 2012
OVERVIEW Using MCQs for summative purposes The Revised Bloom’s Taxonomy Getting beyond remembering with MCQs Using MCQs for formative purposes $ The rationale $ Some strategies
©D. DiBattista 2012
In classical conditioning, what name is given to a stimulus that elicits a particular response even in the absence of any prior training?A. conditioned stimulusB. unconditioned stimulusC. activational stimulusD. discriminative stimulus
“Do you remember…?”
©D. DiBattista 2012
Right after a rat smells menthol, it is always given Drug X, which causes osmometric drinking. Eventually, the rat drinks whenever it smells menthol, even when it is not injected with Drug X. In this situation, what is the role of Drug X?A. conditioned stimulusB. unconditioned stimulusC. activational stimulusD. discriminative stimulus
Note the important role of NOVELTY.
©D. DiBattista 2012
To learn about the attitudes of Canadian voters toward a government bill limiting auto emissions, Gary gave a questionnaire to 200 members of a group called Friends of a Cleaner Environment. He found that every single one of them strongly supported the bill. Based on this research, which of the following is the most appropriate conclusion?
©D. DiBattista 2012
A. The questionnaire is extremely reliable.B. The questionnaire items are not written
very clearly.C. The majority of Canadian voters strongly
support the bill.D. The results tell us little about the attitudes
of Canadian voters.
©D. DiBattista 2012
This item deals with sampling bias,but sampling bias is never mentioned.
Note again the importance of novelty.
Your friend Anne reads that census data indicate that people are having fewer children nowadays than they did 40 years ago. Anne does not believe this because the young couple who live next door to her are both under 30 and already have four children. If Keith Stanovich were told about this, what might you reasonably expect him to say?
©D. DiBattista 2012
A. The census data must be wrong.B. Anne’s comment illustrates valid probabilistic reasoning.C. Anne’s comment illustrates the use of person-who statistics.D. The young couple provide an exception that actually serves to prove the rule.
©D. DiBattista 2012
Suppose that you have performed a Gram stain on a mixed culture of Gram-positive and Gram-negative cells. Afterwards you realize that you omitted the iodine step. What do you expect to see when you look at the slide under the microscope?
©D. DiBattista 2012
An MCQ from the test bank for Russell, et al., Biology: Exploring the Diversity of Life.
A. colourless Gram-positive cells and pink Gram-negative cells
B. colourless Gram-positive cells and colourless Gram-negative cells
C.pink Gram-positive cells andpink Gram-negative cells
D. purple Gram-positive cells andpink Gram-negative cells
©D. DiBattista 2012
And see the handout for more examples…Now let’s see how MCQs can
be used in a formative fashion.
OVERVIEW Using MCQs for summative purposes The Revised Bloom’s Taxonomy Getting beyond remembering with MCQs Using MCQs for formative purposes $ The rationale $ Some strategies
©D. DiBattista 2012
©D. DiBattista 2012
What is Formative Assessment?
A formative assessment is one that is primarily intended to promote students’ learning rather than to assign a grade.
©D. DiBattista 2012
“The key idea underlying our research is that frequent classroom testing (and student self-testing) can greatly improve education from kindergarten through university. This is a bold claim that runs counter to current wisdom in educational circles…. [We] emphasize testing as an aid to learning, a practice that should be part and parcel of a good educational system.”
Roediger, et al., 2006
Formative Uses of MCQs
©D. DiBattista 2012
MCQs during class
MCQs outside of class$ Aplia$ Online quizzes
The so-called “10-minute rule”
©D. DiBattista 2012
Attributed to Hartley & Davis, 1978
Some classic data on attention…
©D. DiBattista 2012
“More students will maintain interest for a longer time when the topic is interesting and delivered with clarity, enthusiasm, and drama, with a cognitive break now and then.”
Wilson & Korn, 2007
©D. DiBattista 2012
“Change grabs attention.”Dan Willingham
University of Virginia
“More students will maintain interest for a longer time when the topic is interesting and delivered with clarity, enthusiasm, and drama, with a cognitive break now and then.”
Wilson & Korn, 2007
©D. DiBattista 2012
PRESENT NEW MATERIAL
MCQPRESENT
NEW MATERIALDEMO
PRESENT NEW MATERIAL
MCQ
A role for clickers!
BUTClickers work well only when the
questions asked are used effectively.
Lecture time
62
55
18
0
20
40
60
80
Course 1 Course 2 Course 3
Per
cen
t ag
reei
ng
©D. DiBattista 2012
Because we use clickers, we havebetter discussions of class material.
Dol & DiBattista, 2011
©D. DiBattista 2012
82
55 57
0
10
20
30
40
50
60
70
80
90
100
Course 1 Course 2 Course 3
Per
cen
t ag
reei
ng
The questions asked when we use clickersencourage me to think more deeply about the material.
Dol & DiBattista, 2011
©D. DiBattista 2012
The clicker questions allowme to learn from my mistakes.
Dol & DiBattista, 2011
92
53
62
0
20
40
60
80
100
Course 1 Course 2 Course 3
Per
cen
t ag
reei
ng
©D. DiBattista 2012
• Retrieval-practice questions Remember or higher-level
• Fuzzy best-answer questions More than one defensible answer
• Student perspective questions Opinions and experiences
• Misconception questions Directly address errors of thought
Four Types of Clicker Questions
©D. DiBattista 2012
• Retrieval-practice questions Remember or higher-level
• Fuzzy best-answer questions More than one defensible answer
• Student perspective questions Opinions and experiences
• Misconception questions Directly address errors of thought
Which of the following defines microevolution?
Which of the following provides an example of microevolution?
Four Types of Clicker Questions
• Retrieval-practice questions Remember or higher-level
• Fuzzy best-answer questions More than one defensible answer
• Student perspective questions Opinions and experiences
• Misconception questions Directly address errors of thought
©D. DiBattista 2012
What research strategy would be most effective for answering the question that Burton has posed?
Four Types of Clicker Questions
• Retrieval-practice questions Remember or higher-level
• Fuzzy best-answer questions More than one defensible answer
• Student perspective questions Opinions and experiences
• Misconception questions Directly address errors of thought
©D. DiBattista 2012
Which of the following best describes what you believe about the origins of human beings?
Four Types of Clicker Questions
©D. DiBattista 2012
Would you expect the vapour pressure ofthese two flasks of water to differ? If so, how?
A B
• Retrieval-practice questions Remember or higher-level
• Fuzzy best-answer questions More than one defensible answer
• Student perspective questions Opinions and experiences
• Misconception questions Directly address errors of thought
Four Types of Clicker Questions
Formative Uses of MCQs
©D. DiBattista 2012
MCQs during class
MCQs outside of class$ Aplia$ Online quizzes
• Animations and videos clarify complex biological processes.
• Active learning opportunities include automatically graded questions and detailed explanations.
• Questions focus on data analysis, experimental and observational approaches to research, real-world applications, global problems, and more.
Outside of Class:Aplia
APLIA
©D. DiBattista 2012
Outside of Class:Online Quizzes
©D. DiBattista 2012
The course
$ PSYC 2F23: Statistics and Research Design
$ Full year, 4 hr/wk of contact
$ Required course; unpopular topic
$ Tests and exams: 78% of course grade
The online quizzes
$ Purpose: Progress check and exam prep
$ Both REMEMBER and HIGHER-LEVEL MCQs
$ Six quizzes via Sakai; 12 items/quiz
$ Low stakes (2 marks per quiz)
$ 10-day window for each quiz
$ Unlimited attempts
$ Only highest mark counts
$ Feedback after each quiz attempt
©D. DiBattista 2012
The research
$ Students completed an extensive survey at end of course.
$ Participation rate: 69% (287 out of 413 students)
$ Obtained self-reports re:Course liking, Quiz liking, Quiz Motivation, and Learning Value of Quizzes
©D. DiBattista 2012
Major findings
$ 94%: Quizzes cover important material.
$ 87%: Quizzes help me know what to study more.
$ 17%: Did not like having the quizzes.
$ 79%: Would like to have them in other courses.$ 53%: Would not even look at quizzes if they did not count for any marks.
©D. DiBattista 2012
$ Total quiz score and Exam grades: r = +0.32(Partialed out: GPA, Course Liking, Quiz Liking, Quiz Motivation)
$ 79%: Quizzes have Learning Value. The online quizzes help me to learn the material relating to…
Strongly related to Quiz Motivation
When I take an online quiz,I work hard to answer every single item correctly.
If the online quizzes did not count for any marks,I doubt that I would even look at them. (R)
©D. DiBattista 2012
Learning Value YES NO
QuizMotivation
Positive
Negative
$ 97% of those with positive Quiz Motivation agreed that the quizzes promoted their learning.
$ In contrast, only 65% of those with negative Quiz Motivation felt this way.
$ They also liked the course less, and they did less well on exams.
©D. DiBattista 2012
152
84 46
5
Two Major Conclusions
In a summative context, MCQs can be used to assess students’ learning both at the remember level and at higher cognitive levels.
Well-chosen MCQs can be used in a formative fashion to promote students’ learning both inside and outside of the classroom.
©D. DiBattista 2012