Post on 22-Feb-2016
description
transcript
1
Teacher Performance Assessment Consortium
November 10, 2010Andrea Whittaker and Nicole
MerinoStanford University
1
© Stanford Center for Assessment, Learning, and Equity, 2010
Three Sponsoring Organizations
• AACTE – overall project management, communication with programs
• Stanford University – assessment development and technical support
• Council of Chief State School Officers – policy development and support, communication with state education agencies
2
© Stanford Center for Assessment, Learning, and Equity, 2010
• Sign off from state education agencies
• Participation of preparation programs
• Up to 3 funded through grant
• Alignment and policy studies
• Adaptation of TPA to address future state program and policy needs
3
TPAC State Participation Requirements
4
© Stanford Center for Assessment, Learning, and Equity, 2010
Lineage of TPAC
• National Board for Professional Teaching Standards (NBPTS) portfolio assessments – accomplished teachers
• Connecticut BEST assessment – teachers at end of induction
• Performance Assessment for California Teachers (PACT) – pre-service teachers
source: SCALE @ Stanford University 5
© Stanford Center for Assessment, Learning, and Equity, 2010
APPLE CriteriaThe National Board for Professional Teaching
Standards
Administratively feasible
Publicly credibleProfessionally
acceptableLegally defensibleEconomically affordable
6
© Stanford Center for Assessment, Learning, and Equity, 2010 7
• Discipline Specific• Integrated Assessment • Student Centered• Analytic Feedback and
Support • Represents a Complex
View of Teaching: Multiple Measures 7
Design Principles: Educative Assessment
88
The TPAC Assessment System
Assessments Embedded in Local Programs
— examples —
Observation/Supervisory Evaluation & Feedback
Child Case Studies
Analyses of Student Learning
Curriculum /Teaching Analyses
The Capstone Assessment
TPAC AssessmentDemonstrates :• Planning Instruction and Assessment• Engaging Students and Supporting Learning •Assessing Student Learning‣Reflecting‣Academic Language
© Stanford Center for Assessment, Learning, and Equity, 2010 9
TPAC Assessment Records of Practice*
Instructional and Social ContextLearning Segment of 3 to 5 Hours/Days
Planning•Lesson Plans•Handouts, overheads, student work•Lesson Commentary
Engagement•Video clip(s)•Teaching Commentary
Assessment•Analysis of Whole Class Assessment •Analysis of learning of 2 students•Feedback to 2 students•Next steps in instruction
• Evidence of Academic Language Development• Daily Reflections and Reflective Commentaries
9
© Stanford Center for Assessment, Learning, and Equity, 2010 10
Guiding Questions and Analytic Rubrics
• PLANNING‣ Planning Focused,
Sequenced Instruction‣ Using Knowledge of
Students to Inform Teaching
‣ Planning Assessments to Monitor and Support Student Learning
• ENGAGING‣ Engaging Students in
Learning‣ Deepening Student
Learning During Instruction
10
• ASSESSMENT‣ Analyzing Student Work‣ Using Assessment to
Inform Instruction‣ Using Feedback to Guide
Further Learning
• REFLECTION‣ Monitoring Student
Progress and Adjusting Instruction
• ACADEMIC LANGUAGE‣ Understanding Language
Demands and Resources‣ Developing Students’
Academic Language Repertoire
10
© Stanford Center for Assessment, Learning, and Equity, 2010
Rubric Levels• Is the candidate ready for
independent teaching (i.e., to be the teacher of record)?
• Rubric Levels‣Level 1 – Some skill but needs
more practice to be teacher-of-record
‣Level 2 – Acceptable level to begin teaching
‣Level 3 – Solid foundation of knowledge and skills
‣Level 4 – Stellar candidate, in the top 5% of candidates
© Stanford Center for Assessment, Learning, and Equity, 2010
Questions and Answers
???12
© Stanford Center for Assessment, Learning, and Equity, 2010 13
PACT ScoresInter-rater Reliability
Level of Agreement Percent
Exact Match 46%± 1 point 34%
± 2 points or greater 10%
Sample Size · 2,580Spearman-Brown Reliability Estimate · 0.88
13
© Stanford Center for Assessment, Learning, and Equity, 2010 14
• Content validity‣ Development teams,
Program directors, Program faculty, & Leadership team
‣ TPE alignment study
• Concurrent validity‣ Evaluation of score
validity‣ Decision Consistency ·
Holistic vs. Analytic ratings
• Bias and fairness review
PACT Validity Studies
14
• Construct validity‣ Factor Analysis
(2002-03 Pilot Year):• Reflection
& Assessment• Instruction• Planning
• Predictive Validity (Carnegie/CT Study)
• Consequential Validity
© Stanford Center for Assessment, Learning, and Equity, 2010
Research Plans• Value-added analysis• Observations and interviews of
candidates completing assessment + first year of teaching
• Surveys and interviews with faculty and program administrators on use of assessment data
15
© Stanford Center for Assessment, Learning, and Equity, 2010
Assessment Development
• 2009-10 Small-scale tryout tasks & feedback from users
• 2010-11 Development of pilot prototypes based on feedback and piloting in at least 2 credential areas per institution. User feedback to guide revisions
• 2011-12 National field test of prototypes, producing a technical report with reliability and validity studies and a bias and sensitivity review. National standard setting.
• 2012-13 Adoption of validated assessment16
© Stanford Center for Assessment, Learning, and Equity, 2010
Overview of Tasks and Rubrics
• Artifacts and Commentaries ‣ What looks familiar?
• Level Three Rubric Descriptors‣ What gets assessed?
• Tips for implementation17
© Stanford Center for Assessment, Learning, and Equity, 2010
Handbook OverviewWhat looks like something we already do?
What looks new or different?
What will our candidates be good at now?
What might be a struggle?
18
© Stanford Center for Assessment, Learning, and Equity, 2010
Context Matters• Candidates describe what they know about
their students as learners:‣ Academic development‣ Academic Language development
‣ Social-emotional development
‣ Family, community and cultural assets‣ Special needs
• Candidates use information about students to plan appropriate instruction and assessment.
• Scoring rubrics examine extent to which needs of students are addressed
19
© Stanford Center for Assessment, Learning, and Equity, 2010 20
Guiding Questions and Analytic Rubrics
• PLANNING‣ Planning Focused,
Sequenced Instruction‣ Using Knowledge of
Students to Inform Teaching
‣ Planning Assessments to Monitor and Support Student Learning
• ENGAGING‣ Engaging Students in
Learning‣ Deepening Student
Learning During Instruction
20
• ASSESSMENT‣ Analyzing Student Work‣ Using Assessment to
Inform Instruction‣ Using Feedback to Guide
Further Learning
• REFLECTION‣ Monitoring Student
Progress and Adjusting Instruction
• ACADEMIC LANGUAGE‣ Understanding Language
Demands and Resources‣ Developing Students’
Academic Language Repertoire
20
© Stanford Center for Assessment, Learning, and Equity, 2010
EM1: Planning Focused, Sequenced Instruction
• Standards/objectives, learning tasks, and assessments are clearly aligned to a big idea or essential question. The learning tasks and assessments represent differing depths of understanding
• Candidate plans how to make clear connections among mathematical facts, computations/procedures, concepts, and reasoning.
• Learning tasks build on each other to promote an understanding of the designated mathematical concepts, computations/procedures, and reasoning skills. Learning tasks (or their adaptation) are justified by explaining their appropriateness for students with references to relevant research and/or theory.
21
© Stanford Center for Assessment, Learning, and Equity, 2010
EM2: Using Knowledge of Students to Inform
Teaching• Learning tasks and materials draw upon
students’ academic and social/emotional development, including strengths, as well as experiences and interests to help students reach the learning objectives.
• Planned support consists of strategically selected or modified tasks/materials and/or scaffolding of instruction that is closely tied to specific learning objectives. It is appropriate for specific individuals or subgroups.
22
© Stanford Center for Assessment, Learning, and Equity, 2010
EM3: Planning Assessments that Monitor and Support Student
Learning• The set of assessments will provide
evidence of student learning relative to the standards/objectives for each lesson. At least one lesson’s assessments provide evidence of student learning that extends beyond the formulaic application of computations or procedures.
• Assessments are aligned to clearly defined benchmarks or criteria for student performance.
• Assessments are modified or adapted to be appropriate for students having difficulty demonstrating their learning. 2
3
© Stanford Center for Assessment, Learning, and Equity, 2010
EM4: Engaging Students in Learning
• Strategies for intellectual engagement seen in the clip(s) offer structured opportunities for students to develop their own understanding of mathematical concepts through discourse. These strategies reflect attention to students’ academic or language development, social/emotional development, and/or cultural and lived experiences.
• Candidate identifies successful and unsuccessful teaching practices. The proposed improvements are reasonable and address the learning of a subgroup or individual students.
24
© Stanford Center for Assessment, Learning, and Equity, 2010
EM5: Deepening Student Learning During Instruction
• Candidates and/or other students build on what students are saying and/or doing, using reasoning to improve understanding of mathematical concepts.
• Candidate and/or other students prompt students to make connections between and among mathematical concepts and representations of content.
25
© Stanford Center for Assessment, Learning, and Equity, 2010
EM6: Analyzing Student Work
• Criteria are well-defined and reflect the depth of understanding stated in of the indicated standards/objectives from the learning segment.
• The analysis focuses on patterns of student errors, skills, and understandings in relation to standards and learning objectives. The analysis uses these patterns to understand student thinking.
• The analysis is supported by work samples and the summary of learning. Specific patterns are identified for individuals or subgroup(s) in addition to the whole class. 2
6
© Stanford Center for Assessment, Learning, and Equity, 2010
EM7: Using Assessment to Inform Instruction
• Next steps follow from an accurate analysis of student learning and aim at improving student understanding of important features of the standards/learning objectives.
• Next steps focus on improving student performance through targeted support to individuals and groups to address specific identified needs.
• Next steps are based on whole class patterns of performance and some patterns for individuals and/or subgroups and are described in sufficient detail to understand them.
27
© Stanford Center for Assessment, Learning, and Equity, 2010
EM8: Using Feedback to Guide Learning
• Specific and accurate feedback helps the student understand what s/he did well, and provides guidance for improvement.
• Candidate describes how students will use feedback to improve their work or their understanding.
28
© Stanford Center for Assessment, Learning, and Equity, 2010
EM9: Monitoring Student Progress and Adjusting
Instruction• Daily reflections identify what students
could or could not do within each lesson and consider the implications for meeting the standards/objectives at the end of the learning segment.
• Adjustments to instruction are appropriate and focused on addressing some individual and collective learning needs.
29
© Stanford Center for Assessment, Learning, and Equity, 2010
EM10: Understanding Language Demands and
Resources• Candidate describes academic language
strengths and needs of students at different levels of academic language proficiency.
• The language genre(s) discussed are clearly related to the academic purpose of the learning segment and language demands are identified. One or more linguistic features and/or textual resources of the genre are explicitly identified.
• Candidate identifies essential vocabulary for students to actively engage in specific language tasks. 3
0
© Stanford Center for Assessment, Learning, and Equity, 2010
EM11: Developing Students’ Academic Language Repertoire
• The candidate’s use of scaffolding or other support provides access to core content while also providing explicit models, opportunities for practice, and feedback for students to develop further language proficiency for selected genres and key linguistic features.
• Candidate articulates why the instructional strategies chosen are likely to support specific aspects of students’ language development for different levels of language proficiency.
31
© Stanford Center for Assessment, Learning, and Equity, 2010
Potentially New Competencies
• Academic language development
• Interpreting student work and identifying trends across the class
• Designing next steps in instruction based on assessment results
• Providing feedback to students 32
© Stanford Center for Assessment, Learning, and Equity, 2010
Issues to Think about for Spring 2011 Pilot
• Match with curriculum• Which faculty? Which credential
areas?• Candidate support• Timeline, from introducing
assessment through scoring• Preparation for videotaping• Recruiting scorers• Consideration of results 3
3
© Stanford Center for Assessment, Learning, and Equity, 2010
What did successful PACT programs do?
• Broad and collegial involvement in decision making
• Inquiry and program improvement rather than compliance
• Affirm and maintain program values and identity
• Broad involvement in assessment activities
• Regular practices and protocols for data analysis
• Learned from individual “case” studies
34
© Stanford Center for Assessment, Learning, and Equity, 2010
Questions and Answers
???35
© Stanford Center for Assessment, Learning, and Equity, 2010
Academic Language Overview
Three “F” Words…
36
© Stanford Center for Assessment, Learning, and Equity, 2010
Academic Language
Academic language development is making the language explicit to expand students’ control over language and improve their language choices according to the purpose/FUNCTION and audience for the message. Academic language also offers structures/FORMS for developing as well as expressing explanations, evaluations, and analyses.Developing students’ FLUENCY in academic language forms and functions provides access to the “language of school” and academic success 3
7
© Stanford Center for Assessment, Learning, and Equity, 2010
Academic Language
The purposes of Academic Language are to clearly and explicitly define, classify, analyze, explain, argue, interpret and evaluate ideas for distant audiences.
38
© Stanford Center for Assessment, Learning, and Equity, 2010
Academic LanguageThe purposes of Academic Language are to clearly and explicitly define, classify, analyze, explain, argue, interpret and evaluate ideas for distant audiences.
= FUNCTION39
© Stanford Center for Assessment, Learning, and Equity, 2010
Academic Language• Language for academic purposes differs
greatly from every day purposes• These differences include
‣ a better-defined system of FORMS (genres) with explicit expectations about how texts are organized to achieve academic purposes
‣ precisely-defined vocabulary to express abstract concepts and complex ideas
‣ more complex grammar in order to pack more information into each sentence
‣ a greater variety of conjunctions and connective words and phrases to create coherence among multiple ideas
‣ formatting conventions, graphics and organizational titles and headings to guide understanding of texts
40
© Stanford Center for Assessment, Learning, and Equity, 2010
Academic LanguageAcademic Language also includes instructional language needed to participate in learning and assessment tasks, including discussing ideas and asking questions, summarizing instructional and disciplinary texts, following and giving instructions, listening to a mini-lesson, explaining thinking aloud, giving reasons for a point of view, writing essays to display knowledge on tests.
41
© Stanford Center for Assessment, Learning, and Equity, 2010
Academic Language in TPAC
• Vocabulary
• Genre/forms with particular functions▪ Linguistic features (grammar,
organization)
▪ Textual resources (headers, table of contents, illustrations, graphs, charts)
42
© Stanford Center for Assessment, Learning, and Equity, 2010
Vocabulary• Technical vocabulary: triangle, metaphor,
metabolize
• Words whose technical meaning is different than everyday language: “balance” in chemistry, “plane” in mathematics, “ruler” in history/social science, “force” in science
• Connector words: and, but, because, therefore, however
43
© Stanford Center for Assessment, Learning, and Equity, 2010
Genres/Forms• Have a general structure – e.g., narratives,
explanations, arguments
• Structure of an explanation- Description of what is being explained- Statements of cause-effect relationships- Sometimes ending with an interpretation judgment
44
© Stanford Center for Assessment, Learning, and Equity, 2010
Subject-Specific Genres/FORMS
• Representing word problems mathematically
• Procedures for a science experiment
• Literary interpretation
• Argument proposing causes of an historical event
45
© Stanford Center for Assessment, Learning, and Equity, 2010
Structures of Argument
Genre/FORM• Mental verbs used to express opinions:
like, believe• Move from personal to impersonal voice• Connectives used for logical relations and
to link points‣Temporal connectives: first, next‣Causal conditional connectives: because‣Comparative connectives to introduce counterpoints: consequently, therefore
46
© Stanford Center for Assessment, Learning, and Equity, 2010
Structure of Arguments (from Knapp & Watkins, 2005)• Simple argument: point/proposition,
elaboration ‣ “I like The Simpsons because they make me laugh.”
• Argument with evidence: Proposition, argument, conclusion
• Discussion: statement of issue, arguments for, arguments against, recommendation
• Elaborated discussion: statement of issue, preview of pro/con, several iterations of point/elaboration representing arguments against, several iterations of point/elaboration representing arguments for, summary, conclusion
47
© Stanford Center for Assessment, Learning, and Equity, 2010
Characteristics of Advanced Academic Language
• Abstractions: government, electron, linear equation, acid
• Nominalizations, verbs or adjectives becoming nouns to enable more dense text or more cohesive text:
organize into…→ this organization……were revealed. The trigger for this
revelation was…
• More precise connector words and phrases, going beyond “and” or “but” to “in contrast” or “Given this, it follows that…”.
48
© Stanford Center for Assessment, Learning, and Equity, 2010
Academic Language Competencies Measured (Rubric
10)• Understanding language demands and
resources for instructional emphasis‣ Identification of linguistic features of a
genre/form/function addressed within instruction
‣ Relation of vocabulary identified to content and to students’ academic language proficiencies
‣ Description of student language strengths and needs
49
© Stanford Center for Assessment, Learning, and Equity, 2010
Academic Language Competencies Measured (Rubric
11)• Expanding students’ academic
language repertoires‣ Making key linguistic features related
to genre purpose (form and function) visible to students
‣ Modeling vocabulary and linguistic features and providing opportunities for practice (developing fluency)
‣ (at higher levels) appropriateness of models for students at different levels of language proficiency
50
© Stanford Center for Assessment, Learning, and Equity, 2010
• Analyze video for academic language using graphic organizer
http://www.learner.org/resources/series33.html
Activity
51
© Stanford Center for Assessment, Learning, and Equity, 2010
Academic Language Takeaways
• Academic language is different from everyday language. Some students are not exposed to this language outside of school.
• Much of academic language is discipline-specific.
• Unless we make academic language explicit for learning, some students will be excluded from classroom discourse and future opportunities that depend on having acquired this language. 5
2
© Stanford Center for Assessment, Learning, and Equity, 2010
Upcoming WebinarTuesday, November 16
1:30 Eastern Time
• To register contact Rachel Popham at
‣rpopham@aacte.org
53