Date post: | 17-Dec-2015 |
Category: |
Documents |
Upload: | augustine-hill |
View: | 216 times |
Download: | 1 times |
11
The Thinking BehindThe Thinking Behind
··PACT·PACT·PPerformanceerformance
AAssessment forssessment for CCaliforniaalifornia TTeacherseachers
The Thinking BehindThe Thinking Behind
··PACT·PACT·PPerformanceerformance
AAssessment forssessment for CCaliforniaalifornia TTeacherseachers
Raymond PecheoneRaymond Pecheone Stanford UniversityStanford University
April 16, 2008April 16, 2008
22
The PACT Assessment The PACT Assessment SystemSystem
The PACT Assessment The PACT Assessment SystemSystem
• A performance assessment for teacher A performance assessment for teacher candidates created in response to SB 2042, candidates created in response to SB 2042, with new subject matter standards, new with new subject matter standards, new program standards, and new assessment program standards, and new assessment standardsstandards
• Alternate assessments permitted must meet Alternate assessments permitted must meet California Quality Standards for California Quality Standards for reliability/validity reliability/validity (i.e., AERA/APA test standards).(i.e., AERA/APA test standards).
• Aligned with the California Teaching Aligned with the California Teaching Performance Expectations (standards) and Performance Expectations (standards) and California Content StandardsCalifornia Content Standards
• High stakes assessment designed to initially High stakes assessment designed to initially license beginning teacherslicense beginning teachers
22
33
PACT InstitutionsPACT InstitutionsPACT InstitutionsPACT Institutions
‣ UC BerkeleyUC Berkeley
‣ UC DavisUC Davis
‣ UC IrvineUC Irvine
‣ UCLAUCLA
‣ UC RiversideUC Riverside
‣ UC San DiegoUC San Diego
‣ UC Santa BarbaraUC Santa Barbara
‣ UC Santa CruzUC Santa Cruz
33
‣ Cal Poly — SLOCal Poly — SLO
‣ CSU Channel IslandsCSU Channel Islands
‣ CSU ChicoCSU Chico
‣ CSU Dominguez HillsCSU Dominguez Hills
‣ CSU Monterey BayCSU Monterey Bay
‣ CSU NorthridgeCSU Northridge
‣ Humboldt StateHumboldt State
‣ Sacramento State Sacramento State
‣ San Diego StateSan Diego State
‣ San Francisco StateSan Francisco State
‣ San Jose StateSan Jose State
‣ Sonoma StateSonoma State
‣ StanfordStanford
‣ Holy Names UniversityHoly Names University
‣ Mills CollegeMills College
‣ Notre Dame de Namur Notre Dame de Namur UniversityUniversity
‣ Pepperdine UniversityPepperdine University
‣ St. Mary’s College of St. Mary’s College of CaliforniaCalifornia
‣ University of the University of the PacificPacific
‣ University of San University of San DiegoDiego
‣ Antioch UniversityAntioch University
‣ USCUSC
‣ San Diego InternSan Diego Intern
444444
The PACT Assessment The PACT Assessment SystemSystem
The PACT Assessment The PACT Assessment SystemSystem
Assessments Embedded in Local Assessments Embedded in Local ProgramsPrograms
— — examples —examples —
Assessments Embedded in Local Assessments Embedded in Local ProgramsPrograms
— — examples —examples —
Observation/Supervisory Evaluation & Observation/Supervisory Evaluation & FeedbackFeedback
Observation/Supervisory Evaluation & Observation/Supervisory Evaluation & FeedbackFeedback
Child Child Case Case StudiesStudies
Child Child Case Case StudiesStudies
Analyses Analyses of Student of Student LearningLearning
Analyses Analyses of Student of Student LearningLearning
Curriculum Curriculum /Teaching /Teaching AnalysesAnalyses
Curriculum Curriculum /Teaching /Teaching AnalysesAnalyses
The Capstone The Capstone Teaching EventTeaching EventThe Capstone The Capstone
Teaching EventTeaching Event
Teaching EventTeaching EventDemonstrates :Demonstrates :
‣PlanningPlanning
‣InstructionInstruction
‣AssessingAssessing
‣ReflectingReflecting‣Academic Academic LanguageLanguage
Teaching EventTeaching EventDemonstrates :Demonstrates :
‣PlanningPlanning
‣InstructionInstruction
‣AssessingAssessing
‣ReflectingReflecting‣Academic Academic LanguageLanguage
55
Teaching Event Teaching Event Records of Practice*Records of Practice*Teaching Event Teaching Event
Records of Practice*Records of Practice*Instructional and Social ContextInstructional and Social Context
3 to 5 Days3 to 5 Days
PlanningPlanning•Lesson PlansLesson Plans•Handouts, Handouts, overheads, overheads, student workstudent work•Lesson Lesson CommentaryCommentary
InstructiInstructionon•Video Video clip(s)clip(s)•Teaching Teaching CommentaryCommentary
AssessmeAssessmentnt•Analysis of Analysis of Whole Class Whole Class Assessment Assessment •Analysis of Analysis of learning of 2 learning of 2 studentsstudents
ReflectioReflectionn•Daily Daily ReflectionsReflections•Reflective Reflective CommentaryCommentary
Evidence of Academic LanguageEvidence of Academic Language
55
* 24 Teaching Events in 13 credential areas
66
Teaching Event Teaching Event Subject AreasSubject Areas
Teaching Event Teaching Event Subject AreasSubject Areas
• Multiple Multiple SubjectsSubjects
‣ LiteracyLiteracy
‣ MathematicsMathematics
66
• Single Subject Single Subject
‣ AgricultureAgriculture
‣ English language English language artsarts
‣ History social History social sciencescience
‣ MathematicsMathematics
‣ ScienceScience
‣ ArtArt
‣ MusicMusic
‣ Physical EducationPhysical Education
‣ World languagesWorld languages
77
Guiding Questions and Guiding Questions and Analytic RubricsAnalytic Rubrics
Guiding Questions and Guiding Questions and Analytic RubricsAnalytic Rubrics
• PLANNINGPLANNING‣ Establishing a Balanced Establishing a Balanced
Instructional FocusInstructional Focus
‣ Making Content Making Content AccessibleAccessible
‣ Designing AssessmentsDesigning Assessments
• INSTRUCTIONINSTRUCTION‣ Engaging Students in Engaging Students in
LearningLearning
‣ Monitoring Student Monitoring Student Learning During Learning During InstructionInstruction
77
• ASSESSMENTASSESSMENT‣ Analyzing Student Work Analyzing Student Work
From an AssessmentFrom an Assessment
‣ Using Assessment to Using Assessment to Inform TeachingInform Teaching
• REFLECTIONREFLECTION‣ Monitoring Student Monitoring Student
ProgressProgress
‣ Reflecting on TeachingReflecting on Teaching
• ACADEMIC ACADEMIC LANGUAGELANGUAGE‣ Understanding Language Understanding Language
DemandsDemands
‣ Supporting Academic Supporting Academic Language DevelopmentLanguage Development
88
PACT Rubrics PACT Rubrics (one example)(one example)PACT Rubrics PACT Rubrics (one example)(one example)
88
ELEMENTARY LITERACY TEACHING EVENT ( 2004-05 PILOT)ELEMENTARY LITERACY TEACHING EVENT ( 2004-05 PILOT)GUIDING QUESTION:How does the candidate use analysis GUIDING QUESTION:How does the candidate use analysis of student learning to propose next steps in instruction?of student learning to propose next steps in instruction?
Level 1Level 1 Level 2Level 2 Level 3Level 3 Level 4Level 4
• Next steps are Next steps are vaguely related to vaguely related to or not aligned with or not aligned with the analysis of the analysis of student student misunderstandings misunderstandings and needs.and needs.
— OR — — OR —
• Next steps are not Next steps are not described in described in sufficient detail to sufficient detail to understand them.understand them.
— OR — — OR —
• Next steps are Next steps are based on inaccurate based on inaccurate conclusions about conclusions about student student development from development from the assessment the assessment analysis.analysis.
• Next steps focus on Next steps focus on improving student improving student performance performance through support through support that addresses that addresses student student misunderstandings misunderstandings or needs.or needs.
• Next steps are Next steps are based on broad based on broad patterns of patterns of performance on the performance on the assessment.assessment.
• Next steps focus on Next steps focus on improving student improving student performance performance through targeted through targeted support to support to individuals and individuals and groups to address groups to address specific specific misunderstandings misunderstandings or needs.or needs.
• Next steps are Next steps are based on analysis of based on analysis of whole class patterns whole class patterns of performance, of performance, some patterns for some patterns for individuals and/or individuals and/or subgroups and subgroups and general knowledge general knowledge of indvidiual of indvidiual students and/or students and/or subgroups.subgroups.
All components of All components of Level 3 plus:Level 3 plus:•Next steps Next steps demonstrate a strong demonstrate a strong understanding of both understanding of both the identfied content the identfied content and language and language standards and of standards and of individual students individual students and/or subgroups.and/or subgroups.
Level 2Level 2
•Next steps focus on improving Next steps focus on improving student performance through student performance through support that addresses student support that addresses student misunderstandings or needs.misunderstandings or needs.
•Next steps are based on broad Next steps are based on broad patterns of performance on the patterns of performance on the assessment.assessment.
99
2-Day Subject Specific 2-Day Subject Specific Scorer TrainingScorer Training
2-Day Subject Specific 2-Day Subject Specific Scorer TrainingScorer Training
• DAY 1DAY 1
‣ Overview of PACT Teaching Event and scoring Overview of PACT Teaching Event and scoring processprocess
‣ Discussion on biasDiscussion on bias
‣ Note taking and DocumentationNote taking and Documentation
‣ Understanding Level “2”Understanding Level “2”
•• DAY 2DAY 2
‣ Understanding Level “1”Understanding Level “1”
‣ Understanding Level “3”Understanding Level “3”
‣ Independently score a Calibration Independently score a Calibration Teaching Event & DebriefTeaching Event & Debrief
1100
PACT ScoresPACT ScoresInter-rater Reliability Inter-rater Reliability
PACT ScoresPACT ScoresInter-rater Reliability Inter-rater Reliability
Level of AgreementLevel of Agreement PercentPercent
Exact MatchExact Match 46%46%
± 1 point± 1 point 34%34%
± 2 points or greater± 2 points or greater 10%10%
Sample SizeSample Size · 2,580 · 2,580Spearman-Brown Reliability EstimateSpearman-Brown Reliability Estimate · · 0.880.88
1111
• Content validityContent validity
‣ Development teams, Development teams, Program directors, Program directors, Program faculty, & Program faculty, & Leadership teamLeadership team
‣ TPE alignment studyTPE alignment study
• Concurrent validityConcurrent validity
‣ Evaluation of score Evaluation of score validityvalidity
‣ Decision Consistency · Decision Consistency · Holistic vs. Analytic Holistic vs. Analytic ratingsratings
• Bias and fairness Bias and fairness reviewreview
PACT Validity StudiesPACT Validity StudiesPACT Validity StudiesPACT Validity Studies
1111
• Construct validityConstruct validity
‣ Factor AnalysisFactor Analysis(2002-03 Pilot Year):(2002-03 Pilot Year):
• Reflection Reflection & Assessment& Assessment
• InstructionInstruction
• PlanningPlanning
• Predictive Validity Predictive Validity (Carnegie/CT Study)(Carnegie/CT Study)
1122
What We Learn from What We Learn from the PACT Analysesthe PACT Analyses
What We Learn from What We Learn from the PACT Analysesthe PACT Analyses
•How our candidates do:How our candidates do:‣ On different aspects of teachingOn different aspects of teaching
‣ In different subject areasIn different subject areas
‣ In comparison to other institutionsIn comparison to other institutions
‣ Over timeOver time
‣ With different kinds of supportsWith different kinds of supports
1122
1133
Task Mean Item Scores by Campus
1.75
2.00
2.25
2.50
2.75
3.00
3.25
Total MISPlanningInstructionAssessment
Reflection
Acad Language
Category Title
Mean Item Score
A
B
C
D
E
F
G
H
I
J
K
Data Charts · 2003-04Data Charts · 2003-04Campus/Task ScoresCampus/Task Scores
Data Charts · 2003-04Data Charts · 2003-04Campus/Task ScoresCampus/Task Scores
1144
Task Mean Item Scores by Content Area
1.75
2.00
2.25
2.50
2.75
3.00
Total MIS
Planning MISInstruction MISAssessment MIS
Reflection MIS
Academic Language
Category Title
Mean Item Score
EL
EM
ELA
MTH
HSS
SCI
Data Charts · 2003-04Data Charts · 2003-04Content Area/Task ScoresContent Area/Task ScoresData Charts · 2003-04Data Charts · 2003-04Content Area/Task ScoresContent Area/Task Scores
1155
PACT Scores - PACT Scores - Assessment of Student Assessment of Student Learning (2003-2005)Learning (2003-2005)
PACT Scores - PACT Scores - Assessment of Student Assessment of Student Learning (2003-2005)Learning (2003-2005)
Student Learning Score Frequency 2003
0
50
100
150
200
250
1 2 3 4
Score
Frequency
Student Learning Score Frequency 2004
0
50
100
150
200
250
1 2 3 4
Score
Frequency
Student Learning Score Frequency 2005
0
100
200
300
400
1 2 3 4
Rubric Scores
Frequencies
1166
Faculty Learning & Faculty Learning & Program ImprovementProgram Improvement
Faculty Learning & Faculty Learning & Program ImprovementProgram Improvement
• Increased articulation across Increased articulation across courses, structures and rolescourses, structures and roles
• Changes in content of some Changes in content of some coursescourses
• Structural changes in Teacher Structural changes in Teacher Education Program Education Program
1177
PACT Teaching Event · PACT Teaching Event · DNADNA
PACT Teaching Event · PACT Teaching Event · DNADNA
• Documents teaching of learning Documents teaching of learning segment segment (3-5 lessons or hours of instruction)(3-5 lessons or hours of instruction)
• Subject specificSubject specific
• Standardized tasks & core questions Standardized tasks & core questions across programsacross programs
• Scored with common rubrics, passing Scored with common rubrics, passing standardstandard
• During student teachingDuring student teaching1177
1188
For More Information...For More Information...For More Information...For More Information...
• See Teaching Event See Teaching Event Handbooks and Rubrics Handbooks and Rubrics at www.pacttpa.org.at www.pacttpa.org.
1199
ActionsActionsActionsActions
ScorinScoring g
theTEtheTE
ScorinScoring g
theTEtheTE
CollaborativCollaborativeplanning eplanning
acrossUniveacrossUniversity &K-12 rsity &K-12
schoolsschools
CollaborativCollaborativeplanning eplanning
acrossUniveacrossUniversity &K-12 rsity &K-12
schoolsschools
ProfessionProfessional al
DevelopmeDevelopmentnt
ProfessionProfessional al
DevelopmeDevelopmentnt
PACT PACT AdvisoAdviso
rr
PACT PACT AdvisoAdviso
rr
Analysis Analysis of of
CandidatCandidate Worke Work
Analysis Analysis of of
CandidatCandidate Worke Work
Program Program MeetingMeeting
ss
Program Program MeetingMeeting
ss
2200
Total Mean Item Scores and Task Mean Item Scores by Campus (2003-04 Pilot Year)
1.50
1.70
1.90
2.10
2.30
2.50
2.70
2.90
3.10
3.30
Total MIS Planning Instruction AssessmentReflection
Acad Language
Mean Item Score
A
B
C
D
E
F
G
H
I
J
K
2211
The Research Base for The Research Base for Teacher Licensing TestsTeacher Licensing TestsThe Research Base for The Research Base for Teacher Licensing TestsTeacher Licensing Tests
• Weak relationship between traditional Weak relationship between traditional licensing tests and teacher effectiveness licensing tests and teacher effectiveness (NRC, 2001)(NRC, 2001)
‣ Strauss & Sawyer (1986)Strauss & Sawyer (1986)
‣ Ferguson (1991, 1998)Ferguson (1991, 1998)
‣ Ferguson & Ladd (1996) Ferguson & Ladd (1996)
‣ Clotfelter, Ladd, & Vigdor (forthcoming) Clotfelter, Ladd, & Vigdor (forthcoming)
‣ Goldhaber (2005, 2006) Goldhaber (2005, 2006)
• Effect sizes quite small in recent value Effect sizes quite small in recent value added research (.01 .06)added research (.01 .06)
2211
2222
Educative AssessmentEducative AssessmentEducative AssessmentEducative Assessment
• Teachers MatterTeachers Matter
• Subject Matter MattersSubject Matter Matters
• Preparation (support) Preparation (support) MattersMatters
• Authenticity MattersAuthenticity Matters
• Integration of Practice Integration of Practice MattersMatters
2222
2233
Total Mean Item Scores and Task Mean Item Scores by Campus (2003-04 Pilot Year)
1.50
1.70
1.90
2.10
2.30
2.50
2.70
2.90
3.10
3.30
Total MIS Planning Instruction AssessmentReflection
Acad Language
Mean Item Score
A
B
C
D
E
F
G
H
I
J
K
2244
Total Mean Item Score and Task Mean Item Score by Content Area (2003-04 Pilot Year)
1.5
1.7
1.9
2.1
2.3
2.5
2.7
2.9
3.1
Total MISPlanning MIS
Instruction MISAssessment MIS
Reflection MIS
Academic Language
Mean Item Score
EL
EM
ELA
MTH
HSS
SCI
2255
California Teaching California Teaching Performance Performance ExpectationsExpectations
California Teaching California Teaching Performance Performance ExpectationsExpectations
‣ TPE 1 · TPE 1 · Specific Pedagog-Specific Pedagog-ical Skills for Subject ical Skills for Subject Matter InstructionMatter Instruction
‣ TPE 2 · TPE 2 · Monitoring Student Monitoring Student Learning During InstructionLearning During Instruction
‣ TPE 3 · TPE 3 · Interpretation and Interpretation and Use of AssessmentsUse of Assessments
‣ TPE 4 · TPE 4 · Making Content Making Content AccessibleAccessible
‣ TPE 5 · TPE 5 · Student Student EngagementEngagement
‣ TPE 6 · TPE 6 · Developmentally Developmentally Appropriate Teaching Appropriate Teaching PracticesPractices
2255
‣TPE 7 · TPE 7 · Teaching English Teaching English LearnersLearners
‣TPE 8 · TPE 8 · Learning about Learning about StudentsStudents
‣TPE 9 · TPE 9 · Instructional Instructional PlanningPlanning
‣TPE 10 · TPE 10 · Instructional Instructional TimeTime
‣TPE 11 · TPE 11 · Social Social EnvironmentEnvironment
‣TPE 12 · TPE 12 · Professional, Professional, Legal, and Ethical Legal, and Ethical ObligationsObligations
‣TPE 13 · TPE 13 · Professional Professional GrowthGrowth
2266
What is Subject Specific What is Subject Specific about the Teaching about the Teaching
Event?Event?
What is Subject Specific What is Subject Specific about the Teaching about the Teaching
Event?Event?• Focus of learning segment & aligned to Ca. Focus of learning segment & aligned to Ca.
content standardscontent standards
• Teaching/learning tasks on video clip(s)Teaching/learning tasks on video clip(s)
• Additional prompts in some content areas Additional prompts in some content areas (e.g., misconceptions in science, (e.g., misconceptions in science, dispositions in mathematics, description of dispositions in mathematics, description of text in text in English/language arts)English/language arts)
• Common and subject specific rubricsCommon and subject specific rubrics
• BenchmarksBenchmarkswithin subject areaswithin subject areas 22
66
2277
ScoringScoringScoringScoring
• Trained and calibrated subject Trained and calibrated subject specific assessorsspecific assessors
• Campus based with central audits Campus based with central audits & regional scoring& regional scoring
• Rubric based scoring in real time Rubric based scoring in real time (web based platforms)(web based platforms)
• Organized around dimensions of Organized around dimensions of teaching (PIARA) and guiding teaching (PIARA) and guiding questionsquestions
• Sequentially Scored By PIARA TasksSequentially Scored By PIARA Tasks 2277
2288
PACT ScoresPACT ScoresAssessment of Student Learning (2003 - Assessment of Student Learning (2003 -
2005)2005)
PACT ScoresPACT ScoresAssessment of Student Learning (2003 - Assessment of Student Learning (2003 -
2005)2005)
Student Learning Score Frequency
0
100
200
300
400
1 2 3 4
— Score —
— Frequency —
2003
2004
2005