Data-Based Decision Making, Assessment for Learning and Diagnostic
Testing in Formative AssessmentAEA-Europe 2013 Paris
Fabienne Van der Kleij Jorine Vermeulen Kim Schildkamp
Theo Eggen
Formative assessment
…lack of a uniform definition
Formative assessment is an umbrella term
informed teaching
peer assessment
feedbackresponding to data
progress
reflectionauthentic
classroom dialogues
student ownership
diagnosis
observationrubrics
understandingattaining standards
self-regulation
targeted instruction
Evaluation or assessment
Different aggregation levels
Formative evaluation
the use of assessment data to make decisions concerning the quality of education at a higher aggregation level than the level of the learner or the class
Formative assessment
concerns decisions at the levels of the learner and the class to accommodate the pupils’ individual educational needs
Three approaches to formative assessment
Data-based decision making
DBDMAssessment for learning
AfL
Diagnostic testing
DT
Attaining results and targets
Focus on the quality of the
learning process
Detailed insights into learning
processes and outcomes
• Educational practice implements a mix of these approaches.
• Terminology and definitions are inappropriately used interchangeably theoretical ambiguity.
• Discussions tend to revolve around finding evidence of what works. “quick fixes” (Elwood, 2006).
• “Definition is important because if we can’t clearly define an innovation, we can’t meaningfully document its effectiveness” (Bennett, 2011, p. 8).
• Need for prescriptions on why, how, and when assessment should be used by learners, teachers, and schools to enhance learning.
Three approaches to FA
… lessons from the literature on formative assessment
• Initial exploration: Recognise the fundamental differences amongst these approaches blend these approaches in a meaningful way.
• Contribute to a more coherent research agenda.
Note: We do not intend to make any claims about which assessment approach is most effective.
Aims and scope
Comparative theoretical study
• What are the similarities and differences in the theoretical underpinnings of DBDM, AfL, and DT?
• What are the consequences of these similarities and differences for implementing DBDM, AfL, and DT in educational practice?
Research questions
Starting point for implementing a system-wide formative assessment approach: An understanding of the learning theories behind currently dominant approaches (Elwood, 2006)
• Neo-behaviourism• Cognitivism• Meta-cognitivism• Social cultural theory• (Social) constructivism
Learning theories and feedback
• Neo-behaviourism: Introduction of the formative concept. memorisation of facts, feedback is intended to reinforce correct recall of facts.
• Cognitivism: Highlights information processing and knowledge representation. Feedback is intended to correct incorrect responses.
• Meta-cognitivism: Emphasis on learning how to learn and regulating the learning process. Feedback is about how to learn, rather than about what to learn.
• Social cultural theory: Feedback in the form of scaffolding (ZPD) for acquiring knowledge and skills, focus on what students are able to learn.
• (Social) constructivism: Learning is a cyclic process in which new knowledge and skills are continuously built on prior ones. Active engagement, realistic contexts, learning in interaction, frequent elaborated feedback.
Learning theories in a nutshell
Theoretical underpinnings and implementation
Data-based decision making
DBDMAssessment for learning
AfL
Diagnostic testing
DT
• Schildkamp and Kuiper (2010): "systematically analyzing existing data sources within the school, applying outcomes of analyses to innovate teaching, curricula, and school performance, and, implementing (e.g., genuine improvement actions) and evaluating these innovations" (p. 482).
• Data in the context of schools is "information that is systematically collected and organized to represent some aspect of schooling" (Lai & Schildkamp, 2013, p. 10).
e.g., assessment results, student background characteristics, questionnaire results.
Theoretical underpinnings
DBDM
Process of data use in DBDM
Feedback loop
Data use
Identify data
Collect data
Analyse data
Interpret data
• Early initiatives: Based on neo-behaviourism and cognitivism.
• Move towards social cultural theory and constructivism.• DBDM for formative assessment:
Set learning goals, assess and monitor student achievement, and if necessary, adjust instruction.
• DBDM for formative evaluation Policy development, school improvement planning,
teacher development, and monitoring the implementation of the school's goals.
DBDM
Learning theories and processes
• Aggregation levelSchool, class, and student levels.
• Assessment methodsObjective data from standardised tests, e.g., from a PMS. Formally gathered data. Less frequently available. + Data from standardised assessments and (structured) observations.
• Feedback loopsFeedback based on assessment data. Educators transform data into actions for educational improvement. Length improvement cycles and feedback loops varies. Retroactive in nature.
Implementation of DBDMAggregation Level, Assessment Methods, and Feedback Loops
Theoretical underpinnings and implementation
Data-based decision making
DBDMAssessment for learning
AfL
Diagnostic testing
DT
• ARG "… the process of seeking and interpreting evidence for use by learners and their teachers to decide where the learners are in their learning, where they need to go and how best to get there" (2002, p. 2).
• Klenowski (2009) "part of everyday practice by students, teachers and peers that seeks, reflects upon and responds to information from dialogue, demonstration and observation in ways that enhance ongoing learning" (p. 264).
• Hargreaves (2005): two approaches within AfL
Theoretical underpinnings
AfL
Measurement Inquiry
• Underpinned by meta-cognitivism, social cultural theory, and social constructivism.
• Learning is a social activity; learning occurs through interaction.
• Students have a central role in the learning process; active participation.
• Increase learner autonomy, motivation, and reflection, by facilitating an inquiry-oriented and interactive classroom climate.
AfL
Learning theories and processes
• Aggregation levelClassroom and student levels.
• Assessment methodsPaper-and-pencil tests, dialogues, practical demonstrations, portfolios, peer assessment, or self-assessment. Qualitative & quantitative. Planned & unplanned. Formal & informal. Assessment quality actionable information for formative purposes over the short term.
• Feedback loopsContinuous dialogues and feedback loops, (immediate) feedback to direct further learning.Assessments integrated into the learning process, feedback loops are usually short. continuous adaptation.Interactive, sometimes retroactive or proactive.
Implementation of AfLAggregation Level, Assessment Methods, and Feedback Loops
Theoretical underpinnings and implementation
Data-based decision making
DBDMAssessment for learning
AfL
Diagnostic testing
DT
• How a task is solved is indicative of the developmental stage of the learner.
• Identify the learner's reasoning styles, and skipped or wrongly executed procedural steps caused by misconceptions and prior knowledge.
• Principles from cognitive psychology, subject pedagogy, and learning theories are combined. inferences about student learning based on a student's response patterns.
Theoretical underpinnings
DT
• Underpinned by principles from cognitivism.• Vygotsky's social cultural learning theory ZPD.
• Identify the learner's developmental stages by obtaining action-oriented, fine-grained assessment data (process data).
• Cognitive theories are used to interpret process data to identify misconceptions and knowledge associated with the learner's developmental stage.
• Small-grain size of the measurements in DT exceptionally useful for formative purposes.
DT
Learning theories and processes
• Aggregation levelStudent level.
• Assessment methods
Assessment tasks make inferences about how task behaviour relates to student thinking.
Inferential chain from empirical knowledge from information processing theories, cognitive psychology, and learning trajectories.
Assessment results: indicative of the developmental stage.
Use of CAT and dynamic assessments.• Feedback loops
Short feedback loops in are preferred.
Retroactive, sometimes interactive or proactive.
Implementation of DTAggregation Level, Assessment Methods, and Feedback Loops
Comparison of the three approaches
Theoretical underpinnings of DBDM, Afl, and DT
Approach
Theoretical aspect DBDM AfL DT
Learning theories
Neo-behaviourism Social cultural
theory (Social)
constructivism
Meta-cognitivism Social cultural theory Social constructivism
Cognitivism Social cultural theory
Goals Improve the quality of education and the quality of instruction by using data to monitor and steer practices to reach intended goals (e.g., increased student achievement).
Improve the quality of the learning process by engaging learners to evaluate and reflect on their own learning and steering the learning process through continuous feedback.
Collect fine-grained data about a student’s zone of proximal development, prior knowledge, and reasoning styles that can inform decisions on adapting the learning environment to the learner’s needs.
• Aggregation levels
• Assessment methodsDiversion in the goals: large variety in assessment methods associated with the nature of data, purposes and stakes regarding the use of these data.
• Feedback loopsTake place at various levels and frequencies.
Comparison of the three approaches
Implementation of DBDM, AfL, and DT
Student Class School
DBDM
AfL
DT
Assessment
Evaluation
• Original theoretical underpinnings of the approaches differ.
• Various assessment methods that are underpinned by different learning theories are needed to fully grasp the complexity of learning at all levels.
• Integrating the three assessment approaches can lead to more valid formative decisions.
• Potential to enable school staff to continuously provide feedback at the school, class, and individual levels, to guide and enhance student learning.
Conclusion
Differences, similarities, and possibilities
• DBDM: School, class and student levels: Monitor learning goals, group students, improve the quality of education. Feedback loops extend over a longer time span.
• AfL: Class and individual levels: Improve quality of the learning process by engaging learners. Continuous feedback in short cycles.
• DT: individual level: Collect fine-grained data that can inform decisions on adapting the learning environment. Feedback loops occur irregularly; depends on learner’s needs.
• At different points in the education process, retroactive, interactive, or proactive feedback loops can be used to optimise students’ learning processes.
Conclusion (continued)
Blending the approaches
Towards a balanced assessment system
Alignment of formative with summative practices
“The field will need to marry information from the different assessments to produce a union that a
stronger than any of the parts in isolation” (O’Malley, Lai, McClarty, & Way, 2013, p. 146).
Thank you for your attention!
or