Date post: | 08-Jul-2015 |
Category: |
Documents |
Upload: | cindy-hmelo-silver |
View: | 104 times |
Download: | 0 times |
+
Seeing the elephant: Using mixed methods
to understand complex learning
Cindy E. Hmelo-Silver
Rutgers University
+Indian Parable:
The Blind Men and The ElephantThe First approached the Elephant,
And happening to fall
Against his broad and sturdy side,
At once began to bawl:
"God bless me! but the Elephant
Is very like a wall!"
The Second, feeling of the tusk
Cried, "Ho! what have we here,
So very round and smooth and sharp?
To me `tis mighty clear
This wonder of an Elephant
Is very like a spear!"
The Third approached the animal,
And happening to take
The squirming trunk within his hands,
Thus boldly up he spake:
"I see," quoth he, "the Elephant
Is very like a snake!” (Saxe, n.d.)
+Overview
Complex Learning
Design-based Research
Example 1: STELLAR
Example 2: VMC
Discussion
+What is Complex Learning?
“True knowledge −− understanding −− develops through
exploration, rumination, interpretation, judgment, and the
application of information.” (Hawkins, 1997)
“Complex learning aims at the integration of knowledge,
skills, and attitudes; the coordination of qualitatively different
constituent skills; and the transfer of what is learned to daily
life or work settings.” (van Merrienboer, Kirschner, & Kester,
2003)
+Complexity of Learning Environments
The strengths of design studies lie in testing theories in the crucible of
practice; in working collegially with practitioners, co-constructing
knowledge; in confronting everyday classroom, school, and community
problems that influence teaching and learning and adapting instruction
to these conditions; in recognizing the limits of theory; and in capturing
the specifics of practice and the potential advantages from iteratively
adapting and sharpening theory in its context. (Shavelson et al, 2003)
Its not just the technology….
Pedagogy
Curriculum
Participant structures
AND
Technology
+Answering the question “under what
circumstances”
“What works” is underpinned by a concern for “how, when, and
why” it works, and by a detailed specification of what, exactly, “it”
is. This intimate relationship between the development of theory
and the improvement of instructional design for bringing about
new forms of learning is a hallmark of the design experiment
methodology.
(Cobb et al, 2003, p. 13)
+Design Experiments (Brown, 1992)
+ Supporting new forms of learning
Conducted in a limited number of settings
Example of broader class of phenomena
Embodied conjecture (Sandoval, 2004)
+DBR Cycle
•Comparison of enactments
•Micro cycles within enactments
•Theory
•Embodied conjectures
•Aspects of design
•Documenting context
•Attend to unexpected
•Documenting Learning
• Embodied Conjecture
• “t”theory
What? How?
Why?Revise
?
+Mixing Methods
Drowning in data, e.g.,
Video
Artifacts
Pre and post tests,
Qualitative
Ethnography
Interaction Analysis
Grounded theory
Quantitative
Descriptive
Inferential
Statistical modeling
+
Example 1: STELLARWith Sharon Derry, Anandi Nagarajan, Ellina Chernobilsky
+Adapting PBL to Teacher Education
Initial implementation (Hmelo-Silver, 2000)
Paper cases
One wandering facilitator for 6-7 groups
Limitations
Cases were oversimplification
One wandering facilitator for 6-7 groups
Difficulty identifying fruitful learning issues
because of limited and variable prior knowledge
+STELLAR Learning Environment Goals:
Provide perceptually rich video cases of practice
Affordances for meshing conceptual knowledge with terrain of practice
Support students in identifying learning issues and beginning self-directed learning
Extend skilled facilitation resources
Implementation
Knowledge Web (KW): Learning sciences hypermedia designed according to Cognitive Flexibility Theory (Spiro et al, 1992)
Video cases indexed to KW
PBL online activity structure
+ Knowledge Web
+Videocase Library
+Example Problem
+Group
whiteboard
+Findings (Derry et al, 2006)
Pre-post across institutions using video analysis task
Demonstrated significant gains from pre- to post test for both sites, despite differences in implementations
Quasi-experimental design over 3 semesters
Participants: All pre-service teachers taking Educational Psychology
STELLAR PBL students
126 students from Educational Psych subject pool
Tracer concepts “Understanding”
Moderate to large effects over the three years on quantitative measures
Between group variability striking
+Visual Representations for
Contrasting Case Analysis (Hmelo-
Silver et al, 2008)Group 1
0
5
10
15
20
25
30
35
40
45
50
300 400 500 600 700 800 900 1000 1100
Lines
Codes
Facilitator
CHS
Ann
Fran
Cathy
Fauna
Luke
Other monitoring
SDL
Group monitoring
Individual monitoring
Grounded beliefs
Personal beliefs
Elaborations
Explanations
Transforming
Elaborated telling
Telling
Acknowledgement
Summary
Disagreements
Agreements
Modifications
New Ideas
Metacognitive questions
Explanation questions
Information questions
Personal talk
Concept talk
Tools as help
Tools as a problem
Task talk
View Other Proposals
Research Library
White Board
Discussion Board
Notebook
KW
Video
+Contrasting Cases (cont’d)
Group 2
0
5
10
15
20
25
30
35
40
45
50
450 550 650 750 850 950 1050 1150 1250 1350 1450
Lines
Codes
Facilitator
CHS
Matt
Bob
Carla
Caitlin
Liz
Helen
Other monitoring
SDL
Group monitoring
Individual monitoring
Grounded beliefs
Personal beliefs
Elaborations
Explanations
Transforming
Elaborated telling
Telling
Acknowledgement
Summary
Disagreement
Agreement
Modifications
New Ideas
Metacognitive questions
Explanation questions
Information questions
Personal talk
Concept talk
Tools as help
Tools as a problem
Task Talk
View Other Proposals
Research Library
White Board
Discussion Board
Notebook
KW
Video
+But what about other concepts?
Similar type of rubric developed to measure “transfer”
(Hmelo-Silver et al, 2009)
Components of transfer rubric:
1. requires understanding,
2. involves activating appropriate prior knowledge and
applying something learned in a new situation,
3. involves abstraction and cognitive flexibility,
4. can be near or far transfer, and
5. can be preparation for future learning.
+Scoring Scale
0 “Knows nothing.” No evidence that any aspect of the concept is understood or attended to, or
evidence that concept is rejected or not understood. Concept very unlikely to be used
correctly in planning or implementation unless student teacher receives and is open to
intensive assistance.
1 “Needs substantial scaffolding.” Indicate that there is some limited understanding and
acceptance of idea and that limited range of acceptable implementation of idea is occurring.
However, there are major omissions, weaknesses, or misunderstandings in relation to the idea,
and the student teacher will probably need substantial assistance to help design and
implement the idea successfully.
2 "Demonstrates early expertise.” Indicate idea is likely understood with some range and depth
and is being implemented with at least moderate success as conceptualized. However, there
are some weaknesses or omissions that should be addressed, and this part of the student
teacher’s work could be improved in important ways with some assistance.
3 “Expert.” Evidence that idea is well conceptualized in depth, detail, over a range of uses
(given limits of current assessment context—type of assignment, word limits, etc.) and is
being implemented successfully and reflectively with sophisticated understanding, even
though improvements might still be possible. Encouragement and positive feedback but little
assistance would be appropriate.
+Quantitative Results
Pretest and Posttest Scores by Class Type
Class N Pretest Posttest
STELLAR 33 0.71 (0.31) 2.02 (0.69)
Traditional 37 0.61 (0.36) 0.68 (0.34)
F (1,67) = 114.323, p < .001, d=2.55
+Qualitative results: Understanding why
Large variability in groups
Examined STELLAR whiteboards for contrasting cases
analyses
Engagement with “Transfer”
Group A
6 female students who had some difficulty
Mean gain= 1.40, SD=0.89
Group B
6 female students, rarely needed any assistance
Mean gain= 1.33, SD= 0.61
+
Group A
Discussed transfer in 3 of 4 problems
In Problem 1, Jenny proposed explanation for enduring
understanding that child in video developed:
“In the case of Brandon, he needed to have an understanding
of how and why he was able to solve the block problem in
order to transfer his ideas onto the pizza problem. "The first
factor that influences successful transfer is degree of mastery
of the original subject" (How People Learn, 53). Brandon was
able to continue to solve such a problem because of his
complete understanding of how he was able to arrive at the
solution for the block problem.”
+Group A: Encapsulating
Knowledge
In problem 2, Rina used the concept of transfer in thinking about assessment as she offered this proposal:
…The portfolio should have a final summary of the students' work and questions regarding the students' learning, so that the students can explain and evaluate their own thinking. (knowlege web [sic]) The students should be able to transfer their prior knowledge of concepts such as force and motion in order to create their vehicle, while also allowing the activity to expand on that knowledge. …another important facet of understanding is application (sic). Ms. Baker will know whether the students acquired enduring understanding by how much they can apply this knowledge to real world problems. One way of doing is to have Ms. Baker create another problem that will use the same concepts in a real world setting, and evaluating whether the students were able to apply the concepts they had learned.
+Group B: Getting started
All students mentioned transfer in P1; 2-4 students in subsequent problems
Kathy wrote:
…Second, Brandon was able to recognize a connection between the pizza problem and the tower problem that he did weeks earlier. Moreover, he made this connection relatively quickly and without much effort. He was able to show us, by using his chart and the manipulatives (blocks), exactly how the pizza problem mapped into the tower problem. His understanding of the pizza problem therefore facilitated a new, and deeper, understanding of the block problem; this process is called transfer. Brandon’s seemingly effortless use of transfer provides evidence that he understood the problem, because “transfer and wide application of learning are most likely to occur when learners achieve an organized and coherent understanding of the material.” (How People Learn, p. 238-239)…
+Group B: Moving Along
On later problems, fluid application of concept part of shared understanding
Micki writes:
…another way to look for enduring understanding would be the students' transfer and application of principles of force and motion especially to real world situations. This would show the student's understandings of information previously and transfer it to the problem at hand, which is a real world problem that allows students to work with hands-on material.
Mimi followed this up by incorporating Micki’s comment and a previous proposal from another group member:
To put these two ideas together, [t]he teacher could bring together individual explanation and transfer as evidence of enduring understanding. An activity could be created at the end of each project that would ask the individual members of the group to use the principles gained to explain a real world scenario. Likewise, an activity could be designed to facilitate transfer of the instructional objectives. For instance, one of the objectives was learning the scientific inquiry process. The teacher could present a real world problem that would require the students to use the same scientific process to solve. (This would also facilitate transfer)
+Conclusion
Quantitative methods can show what students learn Pre-post tests
Experimental, quasi-experimental designs
Quantifying verbal data can show how students learn CORDTRA representations can support interpretation
Other kinds of qualitative data analysis can explain how learner engagement affects what they learn
All of these are part of a program of design-based research
+
Example 2with Carolyn Maher, Marjory Palius, Grace Agnew, Robert Sigley,
Chad Mills
www.videomosaic.org
+Video Mosaic Collaborative (VMC)
Preserves a major video data collection on student reasoning From diverse schools settings
To be available as open source
From 40 doctoral dissertations
Makes available new tools for Teachers
Educators
Researchers
From longitudinal/cross sectional studies spanning 25 years
Videos following the same student cohort from elementary school through high school and beyond
Over 4500 hours of video
+Video Mosaic Collaborative
(VMC)
+Design-based Research for Teacher
Professional Development
In-service and pre-service teachers
Primary, middle and secondary mathematics
Counting and fractions strand
At all sites pre and post: Video assessment recognizing forms of reasoning
Content assessment
Belief assessment
At selected sites In-depth qualitative analysis
+Theoretical Perspective
Importance of making sense of students’
conversations and how tools mediate learning
(Hmelo-Silver, 2003)
Being aware of the contextual resources
(media, other Ss, prior experience) that Ss
use influence collaborative knowledge
construction (Arvaja et al., 2006)
Attending to social interactions in collaborative
knowledge construction (Palincsar, 1998)
35
+Results for Video Assessment:
Counting Strand
+An Online Enactment: Using
Resources for Reasoning
Graduate mathematics education hybrid course
Four online groups (three, 6 Ss; one, 7 Ss)
Two + week unit In class problem solving
Individual study of videosand related readings
Group discussion questions
Online Design eCollege CMS
Streaming video / linked papers
Minimal online instructor intervention
Data Postings from online
threaded discussions
Pre and post tests (math, Ss reasoning)
+Research Questions
To what extent do videos and readings promote online
discussion within and across groups?
How, if at all, do learners relate practice to online
discussion?
What evidence, if any, is there of Ss enjoyment in
studying readings and videos?
To what extent do Ss relate videos to readings in their
online discussions?
38
+Coding
All posts coded for comments related to
Video (V)
Readings (R)
Additional sub categories of comments relating
videos/readings to:
Own problem solving (PV/PR)
Others’ problem solving (OV/OR)
Earlier interventions (EV/ER)
Affect (AV/AR)
Practice (TV/TR)
39
+Video: Ankur’s Challenge
Shows two groups of 10th graders (2 in one & 3 in other) working on the problem:
How many different block towers can be built, four tall, selecting from three colors of blocks such that the towers have at least one block of each color?
Approximately 8 minutes
http://hdl.rutgers.edu/1782.1/rucore00000001201.Video.000062055
40
+
Romina’s Proof
+ Discussion Questions
(1) Describe Romina’s strategy for solving the Ankur’s
Challenge problem.
(2) In your opinion, is this solution a convincing one?
Why or why not?
(3) According to the Yackel & Hanna chapter, both von
Glaserfeld and Thompson equate reasoning with
learning (p. 227). From this perspective, in what
ways do explaining and justifying contribute to
learning mathematics?
+
+Summary of Quantitative Analysis
Across all groups: Studying videos generated reflections about own and
classmates’ problem solving
Studying videos of students’ reasoning was enjoyable
But still left us with question of ways in
which which Ss related resources to their
practice
+Connections
+
Bringing New Tools on
Board: The
VMCAnalytic
Allows users to create multimedia
artifacts using the VMC repository
Create narrative with video for purpose
(e.g., research, professional
development)
Used in a variety of ways by instructors,
researchers
Needed to be able to classify and
identify what differentiates high quality
and low quality “analytics”
Data sources: 27 VMCanalytics from
several different classes and
researchers
VMC Analytic by Hmelo-Silver (2011)
VMC Analytic by Horwitz (2011)
+What do different methods tell us?
Class Math Depth
LS Depth Clarity Coherence
Design-based Research 0.78 1.00 1.22 1.33
Intro to
Math Ed 1.96 1.80 2.36 2.20 Critical
Thinking 2.00 2.40 2.30 2.00
Practicum 3.00 2.25 2.63 2.63
No Class 1.17 1.67 3.00 3.00
Quantitative methods tells us about
broad strokes
+Qualitative methods
Looking deeper with contrasting case analysis
Example 1: Analytic illustrating how students can move from particular to general
Concepts from both learning sciences and mathematics clearly articulate
Indicative of designer's understanding of students’ learning trajectory
Example 2: Analytic illustrates teacher questioning during early algebra exploration
Students claims not supported by video segments selected
Textual descriptions of events were vague
Video not well chosen for intended purpose of relationship of teacher questioning and student engagement
+Word Clouds
Analytics coded for emergent
themes
Explored use of word clouds
as a learning analytic
Mathematics Education
ideas
Learning Sciences ideas
Allow us to see dominant
themes within the two areas of
interest
+Synergy
Quantitative and qualitative methods answer different questions
Help move closer to seeing the whole elephant
As part of design research, answer different questions for example,
What students learn
How enactments differ
Ways resources are used in different enactments
How tools are used
In all these questions, both have value
Providing the broad brush
Providing rich detail
Guidance for redesign of theory and learning environement
+Challenges in DBR in Complex
Learning Environments
By their very nature, design studies are complex, multivariate, multilevel, and interventionist, making warrants particularly difficult to establish (Shavelson et al, 2003)
Data overload and responding to emergent questions while trying to stay systematic and focused
Resource challenges
Need to expertise in range of research methods
Documenting design decisions and rationales in the heat of the moments
Staying accountable to both theory and practice