Race to the TopAssessment
ProgramConsultation
Carol Campbell
Atlanta, GeorgiaNovember 17, 2009
2
Assessment Design Principles
EffectiveAssessment
Systems
Designed around a clearvision of purposeand learning goals
Align with and advance commonstandards, curriculumexpectations, learningobjectives and instructional strategies
Encompass and combineassessment of, for and as learning
Use and blend multiple sources of information and assessment approaches to identify, inform and report student learning
Embody high expectations to engage, motivate, support and stretch all students to progress and achieve
Build and developprofessional capacity
Assess a range of content, knowledge, skills and performancein authentic, applied and appropriate ways
Are part of a clear and flexible learning trajectory across subjects and over education/career pathways
Support data-informed decision-making for local and system improvement
Place qualityof assessment overquantity
3
1. Clear Vision of Purpose and Learning Goals - Specifying 21st century skills, college and career-readiness
Example: High school examinations in England combine:Subject/curricular knowledge with generic skills Functional skills – “essential for life, learning and work” – practical
skills in English, mathematics and information and communication technology” (ICT). Example – Math level 1 - “A member of a reprographics team is able to look at statistics and graphs detailing paper usage and identify trends that might suggest that equipment requires servicing”
Personal learning and thinking skills (PLTS) – “These are skills that will equip people for successful employment and lifelong learning…PLTS require learners to be:
Independent enquirers Creative thinkers Reflective learners Team workers Self-managers Effective participants”
4
Assessing 21st Century Skills – England, Ages 14 - 19
Assessment criteria for Functional Skills – “Assessment must, according the skill areas:
Provide realistic contexts, scenarios and problems Specify tasks that are relevant to the context Require application of knowledge, skills and understanding for a
purpose Require problem solving Assess process skills and the outcome of their application in
different contextsGenerally assessments externally defined (exam boards) and
internally assessed (teachers), including controlled assessments in classroom and projects. Combinations of open and fixed response items varies by subject area and level.
Performance assessments increasingly used in range of qualifications – coursework, projects, extended writing
Opportunities for accredited work experience in some qualifications, e.g. Diploma composite qualification includes 10 days work experience
5
Ontario, Canada - Achievement Chart – English, Grades 9-12 – Thinking SkillsThinking – The use of critical and creative thinking skills and/or processes
Categories 50-59% (L1) 60-69% (L 2) 70-79% (L 3) 80-100% (L4)
The student:
Use of planning skills (e.g., generating ideas, gathering information, focusing research, organizing information)
uses planning skills with limited effectiveness
… some effectiveness
… considerable effectiveness
… high degrees of effectiveness
Use of processing skills (e.g. drawing inferences, interpreting, analysing, synthesizing, evaluating)
uses processing skills with limited effectiveness
… some effectiveness
… considerable effectiveness
… high degrees of effectiveness
Use of critical/creative thinking processes (e.g. oral discourse, research, critical analysis, critical literacy, metacognition, creative process)
uses critical/creative thinking processes with limited effectiveness
… some effectiveness
… considerable effectiveness
… high degrees of effectiveness
6
2. Align with and advance common standards, curriculum expectations, learning objectives and instructional strategies
Example: Ontario, Canada Content Standards
Curriculum for Grades 1-12 for all subjects and courses Developed through extensive research, benchmarking,
and consultation with all stakeholders All teachers are required to teach the curriculum
expectations Performance Standards (criterion-referenced)
Criteria are described for student achievement of the curriculum expectations for Grades 1-12
Four levels of achievement Four categories of knowledge and skills: Knowledge and
Understanding; Thinking; Communication; Application Consistency:
Exemplars of student work for all curriculum Standard provincial report cards - aligned with content and
performance standards
7
3. Encompass and combine Assessment of, for and as learning
Assessment…Of learning - summative assessments to confirm
student knowledge, skills, understanding of curriculum, proficiency
For learning – formative assessments using information throughout the learning process to investigate student learning, inform instruction and provide feedback to support student progress
As learning – assessment as a process of meta-cognition involving student self-assessment and understanding of own learning processes and knowledge
8
Summative and Formative Assessments: Example – The National Assessment System, Scotland, UK
FORMATIVE
SUMMATIVE
INTERNAL EXTERNAL
Formative assessment
Personal learning planning
Involving learners, and parentsAnd other adults in the learningprocess
Local authority collection and analysis of information to inform provision and improvement
Inspection feedback and subject/quality/improving reports
Follow-through inspection activities
Teachers’ judgements and reports,With local moderation and NationalAssessments as part of understandingAnd sharing standards
Scottish Survey of Achievement P3, P5, P7, S2National Qualifications (SQA)International studiesHMIE inspections and reports on Authorities and schools
9
4. Use and blend multiple sources of information and assessment approaches to identify, inform and report student learning
Multi-faceted assessments used internationally include:
Assessments of knowledge (recall & analysis) and assessments of performance (demonstration of ability to apply knowledge in practice)
Multiple-choice, constructed-response, extended tasks and projects
On-demand and curriculum-embedded elements Externally-developed (by teachers and
developers) and classroom-developed/managed Scoring made consistent through training,
moderation, calibration, and auditing
10
Examples of Assessment Types: Ontario and England
Types of Task Example
External standardized test – multiple choice response
Approx. 50% of questions in Grade 9 and 10 provincial assessments, OntarioNot used in England
External standardized test – open-ended response
Approx. 50% of questions in Grade 9 and 10 provincial assessments, OntarioUsed for KS 2, parts of GCSE (40-100% of final mark) and A-level in England
Performance assessment, e.g. project Controlled assessments (coursework) for GCSE (up to 60% of overall mark) and Extended Project for A-level (equivalent to half an A-level)
Interaction and observations between teacher and student, e.g. discussion, conference, classroom and homework tasks and tests
Systematic Teacher assessments reported to parents/guardians annually and data collected nationally at key points (England, Ontario)
Cumulative assessments, e.g. portfolios National Vocational Qualifications and Diplomas (England)
11
Singapore GCE A-Level Examinations
12
Systematic Achievement Reports at the End of Key Stages, England, UK
Key Stage 1 (age 6-7): Classroom evidence is combined with centrally-developed open-ended tests and tasks in English and mathematics, taken when children are ready and marked by teachers, with moderation.
Key Stage 2 (age 8-11): Teachers’ summary judgments of student progress from classroom evidence are combined with open-ended tests in English, mathematics and science; externally marked and results collected nationally.
Key Stage 3 (age 14): External tests were recently abolished. Moderated teacher assessments are used to report achievement levels in all subjects, nationally collected.
No mandatory high school assessments – although most students will participate in GCSEs (age 15-16) and progress to other academic (A/AS level), vocational (diploma) and flexible pathways for progression
13
5. Embody high expectations to engage, motivate, support and stretch all students to progress and achieve
Example: England, UK: Making Good Progress Pilot:
Progression Targets to increase the number of pupils who make two levels of progress throughout a key stage
Progression Premium to reward schools which help students who entered a key stage behind national expectations and make good progress
14
The Assessment for Learning Strategy, England, UKOur aims are that: Every child knows how they are doing, and understands
what they need to do to improve and how to get there. They get support they need to be motivated, independent learners on a ambitious trajectory of improvement
Every teacher is equipped to make well-founded judgement about pupils’ attainment, understands the concepts and principles of progression, and knows how to use their assessment judgements to forward plan, particularly for pupils who are not fulfilling their potential
Every school has in place structured and systematic assessment systems for making regular, useful, manageable and accurate assessments of pupils, and for tracking their progress
Every parent and carer knows how their child is doing, what they need to improve, and how they can support the child and their teachers.
15
Applying Knowledge:A Literacy Task, Grade 10 Ontario
Task: Write a minimum of three paragraphs expressing an opinion on the topic below. Develop your main idea with supporting details (proof, facts, examples, etc.).
Purpose and Audience: an adult who is interested in your opinion
Length: The lined space provided for your written work indicates the approximate length of the writing expected.
Topic: Are today’s famous people good role models for young people?
16
6. Assess a range of content, knowledge, skills and performance in authentic, applied and appropriate ways
Example: General Certificate of Secondary Education (GCSE) (age 15-16), England
GCSEs are available in over 50 subjects: students choose. GCSEs normally take 2 years to complete. Many courses offer
assessment in each unit rather than only at the end. GCSEs use constructed response items, and usually include
structured classroom-based tasks that are more extended. Increased emphasis on extended writing and course work. Controlled assessments are taken under supervised conditions
and are either set by the awarding body and marked by teachers or set by teachers and marked by the awarding body. Teachers determine the timing of controlled assessments. Controlled assessments comprise up to 60% of a grade, depending on the subject.
Introduction of assessment of functional skills and personal learning and thinking skills across subjects
Cost of assessing one unit = $10 approx. Average costs of a GCSE course assessment = $40 approx.
17
ICT Controlled Assessment Task, GCSE EnglandLitchfield Promotions works with over 40 bands and artists to promote their
music and put on performances in the UK. The number of bands they have on their books is gradually expanding. Litchfield Promotions needs to be sure that each performance will make enough money to cover all the staffing costs and overheads as well as make a profit. Many people need to be paid: the bands; sound engineers; and, lighting technicians. There is also the cost of hiring the venue. Litchfield Promotions needs to create an ICT solution to ensure that they have all necessary information and that it is kept up to date. Their solution will show income, outgoings and profit.
Candidates will need to: Work with others to plan and carry out research to investigate how similar
companies have produced a solution (The company does not necessarily have to work with bands and artists or be a promotions company.)
Clearly record and display your findings Recommend a solution that will address the requirements of the task Produce a design brief, incorporating timescales, purpose and target
audience
18
ICT Task (cont.)Produce a solution, ensuring that the following are addressed:
it can be modified to be used in a variety of situations it has a friendly user interface it is suitable for the target audience it has been fully tested.
You will need to: Incorporate a range of: software features, macros,
modeling, and validation checks (used appropriately). Obtain user feedback Identify areas that require improvement, recommending
improvement, with justification Present information as an integrated document Evaluate your own and others’ work.
19
England: AS and A Levels - Changes to Assessments A broad range of question types to assess a wide range of skills Questions requiring extended answers give students the
opportunity to demonstrate the full breadth and depth of their understanding
Synoptic assessments that will test students’ understanding of the subject as a whole, including ability to: Develop a broader and deeper understanding of the
connections between the knowledge and understanding set out in the specifications as a whole
Draw together some of the key insights from their studies Make purposeful use of these in undertaking assessment tasks
Introduction of an extended project as a separate qualification for A level students to add to their study program.
20
7. Build and Develop Professional Capacity
Example: Teacher collaboration on assessment design and marking, Alberta, Canada
Test design: Identify student characteristics Assist in exam blueprint document Ensure curricular “fit” of the exam Pilot prototype multiple-choice and written response
forms Help develop writing assignments and their scoringItem building: Item development sessions held throughout the
province involve teachers in the creation of new multiple-choicce and written responses.
21
Teacher Collaboration – Alberta, Canada
Reviewing Tests: Each new examination form is reviewed by a
committee that includes classroom teachers. The committee examines both the written response
and multiple choice sections to ensure that the examination is fair, and demonstrates fidelity to the curriculum.
Confirming Standards: Before written responses are marked a committee of
teachers meets with examination branch staff to select student work for use in marker training (example papers, training papers and reliability review papers).
22
Teacher Collaboration – Alberta, Canada
The Marking Process: Each January and June approximately 1300
teachers meet to mark the written responses in a process with tight calibration to produce consistency in scoring.
Examination Advisory Committee: Once yearly representatives of various stakeholder
groups, including the Alberta Teachers’ Association meet with branch staff to review examination results from the previous school year, and to offer suggestions for the improvement of the examination program.
23
Scoring of Assessments - Ontario
Grade 9 Mathematics – raw score points = 46% multiple-choice, 54% open-response
Ontario Secondary School Literacy Test – raw score points = 48% multiple-choice, 52% open-response
Multiple choice items are machine scored Written open-ended responses systematically scored by qualified scorers
(mostly Ontario educators) Use of generic rubric to describe levels of performance, item-specific rubric
for each item, and anchors to illustrate descriptors for the score points to the rubric
Committee of educators define range of performance within each code of scoring rubric – ‘range finding’ development
Every scorer participates in extensive training System of checks and monitoring applied throughout process Quality assurance and statistical techniques to validate interrater reliability
and scoring validity Use of equating process to place student scores on common scale over two
adjacent years and to determine cut scores
24
Teaching-Learning Critical Pathway.Teaching-Learning Critical Pathway.
Gathering EvidenceGathering Evidence
Area of Greatest NeedArea of Greatest Need
Current PracticeCurrent Practice
Rubric, Data Wall, Culminating TaskRubric, Data Wall, Culminating Task
Professional Learning Community ActionProfessional Learning Community Action
Moderated MarkingModerated Marking
Share Findings, CommunicationShare Findings, Communication
Using Assessments to Inform Improvements:Example: Ontario
25
8. Are part of a clear and flexible learning trajectory across subjects and over education/career pathways
Example: Exam boards in England ensure consistency of standards within subjects while allowing flexibility of approach across subjects.
Five exam boards - governed by specifications, criteria, frameworks and oversight of Office of the Qualifications and Examinations (OFQUAL) Regulator
Exam boards: set out the course syllabus & specifications to meet
criteria set by the regulator oversee writing of exam questions and how exam papers
should be marked check coursework/controlled assessments provide training and quality control for examiners,
including use of ‘standardization meetings’ to compare marking across examiners
set grade boundaries to align marks and standards
26
National Qualifications Framework, England:Learner Pathways
27
9. Support data-informed decision-making for local and system improvement
Assessment for Learning:School and Classroom(Assessment Reform Group,1999)
Assessment for Learning and Data-Informed Decision Making: Systemic (Campbell and Levin, 2007)
Assessment for learning should be part of effective planning of teaching and learning.
Assessment of and for learning, plus relevant contextual, student and school data, is part of effective strategic and operational planning for state/province, district, school and classroom actions for improved teaching and learning.
Assessment for learning should focus on how students learn.
Assessment for learning focuses on how systems, schools and students improve their progress, achievements and outcomes.
28
Assessment for Learning:School and Classroom(Assessment Reform Group,1999)
Assessment for Learning and Data-Informed Decision Making: Systemic (Campbell and Levin, 2007)
Assessment for learning should be recognised as central to classroom practice.
Data-informed decision making, including assessment for learning, is recognised as central to educational practices at all levels of the system – province/state, district, school and classroom.
Assessment for learning should be regarded as a key professional skill for teachers.
Skill and capacity to use, understand and apply data to inform improved actions and outcomes should be regarded as a key professional skills for all educators, including teachers, principals, district staff and state/provincial officials.
Assessment for learning should be sensitive and constructive because any assessment has an emotional impact.
Careful consideration of which data to use, by whom and for what purpose is required to ensure the sensitive and constructive use of data, including assessment, to support improvement for all not to rank or judge unfairly.
29
Assessment for Learning:School and Classroom(Assessment Reform Group,1999)
Assessment for Learning and Data-Informed Decision Making: Systemic (Campbell and Levin, 2007)
Assessment for learning should take account of the importance of learner motivation.
The use of data, including assessment for learning, to generate motivation to improve should place an emphasis on developing respectful partnerships to engage educators at all levels of the system to work together.
Assessment for learning should promote commitment to learning goals and a shared understanding of the criteria by which they are assessed.
Shared goals for improvement, and indicators of success towards these goals, are developed and understood to generate a common commitment to improvement targets and learning outcomes.
Learners should receive constructive guidance about how to improve.
Building capacity for improvement involves providing feedback, strategies, resources and supports to enhance both professional learning and student learning.
30
Assessment for Learning:School and Classroom(Assessment Reform Group,1999)
Assessment for Learning and Data-Informed Decision Making: Systemic (Campbell and Levin, 2007)
Assessment for learning develops learners’ capacity for self-assessment so that they can become reflective and self-managing.
Data-informed approaches involve combining assessment of and for learning to balance external and internal accountability, while building the professional capacity for educators to implement self-evaluation, improvement planning and monitoring strategies.
Assessment for learning should recognise the full range of achievements of all learners.
Assessment for learning, and the use of related data, recognises the full range of achievements of all learners and schools, including not only achievement results but also progress over time and equity of outcomes for closing gaps in performance while also raising the bar overall.
31Students
School Boards/Authorities
Teachers, Principals & Administrators
FundingPolicy/ProgramsPublic Reporting
Ontario School Information System (OnSIS)
Staging Area
Board Validation & Sign-off by Director
MinistryValidation
Encryption&
Depersonalization
Elementary Secondary Data
Warehouse (ESDW)
Statistics Canada
Statistics Canada
EQAOEQAO
Other Sources
Other Sources
Data
Data
Data Data repository or data warehouse
Data repository or data warehouse
Student Management Systems (SMS)Student Management Systems (SMS)
Decision Support System
Decision Support System
EQAO
Stats Can
Other
Board Administrators
Pro
vin
ce-W
ide
In
form
atio
n
Data
Data
Data
MiSAGiARE
Info Info Info
Ministry of Education
32
10. Place quality of assessment over quantity
Examples: Australia – National standards (underway)
State Syllabi, State and Local Assessments(3 grades plus high school)
Canada – Provincial standards, syllabi & assessments(2 grades plus high school)
England – National standards & curriculum School-based assessments + national tasks (1 grade plus high school)Five secondary examination boards
Singapore – National standards and curriculumNational exams + school-based tasks(1 grade plus high school)
33
Common Practices Across Countries
Assessments are part of a tightly integrated system of standards, curriculum, instruction, assessment, and teacher development at the state or national (in small countries) level
Assessments include evidence of actual student performance on challenging tasks that evaluate 21st century standards of learning
Teachers are integrally involved in the development and scoring of assessments
Assessments are designed to continuously improve teaching and learning
Use of multiple-choice testing limited and combined with other forms of assessment
Emphasis on criterion-referenced and curriculum-embedded assessments
Not same approach for every student, every grade, every year Increasing use of teacher assessments in national reporting,
which external summative assessments in specific years/transition points
34
PrinciplesProposals Designed around a clear
vision of purpose and learning goals
Align with and advance common standards, curriculum expectations, learning objectives and instructional strategies
Start with the end in mind – clarify and confirm vision for US high school graduates as engaged citizens, active contributors within the global economy and future leaders. Requires combination of functional skills, generic and specialist knowledge, performance competencies, and personal learning strategies
Integrate assessment design with common standards for consistency of curriculum expectations and clarity of teaching and learning standards. Involves criterion-referenced and curriculum-embedded assessment items. Attention to consistency within curricular areas. Vertical alignment over grade levels to support progression of learning.
35
PrinciplesProposals Encompass and combine
assessment of, for and as learning
Use and blend multiple sources of information and assessment approaches to identify, inform and report student learning
Develop an assessment system focused on learning where summative assessments are one element of a wider repertoire of assessment information. Increase use and consistency of teachers’ assessments in reporting and monitoring student achievement and progress.
Use range of item-types and assessment approaches, including use of performance assessments, extended writing and project work. Use, but limit, time on standardized multiple-choice items and test taking.
36
PrinciplesProposals Embody high expectations to
engage, motivate, support and stretch all students to progress and achieve
Build and develop professional capacity
Range of assessments, including items and tasks that involve authentic experience, critical thinking and applied performance of students. Attention to student personal learning plans and progression targets. Growth and cohort analysis. Develop capacity for student self- and peer-assessment, including group work.
Educator capacity building is critical and requires time, resources (human, financial, material) and support. Teacher moderation is a powerful strategy to support teaching and learning throughout school year. Assessment items, constructs, interval data and other technical advice are required to be available and accessible to identify linkages between assessments, curriculum expectations, instructional strategies and interventions for specific students.
37
PrinciplesProposals Assess a range of content,
knowledge, skills and performance in authentic, applied and appropriate ways
Are part of a clear and flexible learning trajectory across subjects and over education/career pathways
Implement repertoire of formative and summative assessments with variety of approaches within and across grade levels to demonstrate knowledge, understanding and application.
Support clarity of learning pathways through high school and into college and career readiness, with accreditation options and credit accumulation requirements to indicate when a student is on or off track for success
38
PrinciplesProposals Support data-informed
decision-making for local and system improvement
Place quality of assessment over quantity
Implement data infrastructure (technology and systems) and culture (human capacity and data access) to enable collection, analysis and use of data at all levels of system
Shift emphasis from multiple-choice standardized tests as main approach for every student, every grade, every year. Combine external tests with coursework and other approaches to assess learning over time and in applied contexts. Develop consistency across assessment aligned with common standards, quality assurance mechanisms, and attention to building consistency in educator understanding and application of assessment strategies, including teacher moderation.