Educational Assessment
Sandra MacKinnon, Director
Assessment and Evaluation Branch (Anglophone Division)
Department of Education and Early Childhood Development
Atlantic Education International
July 22, 2014
0
0 .2
0 .4
0 .6
0 .8
1 .0
-3 -2 -1 0 1 2 3
b
Ab i l i ty
Pro
bab
ilit
y
Item Characteristic Curve: ITEM 0001
a = 0 .5 3 9 b = 0 .2 0 2
1. Educational Assessment in General
~Balanced Assessment
Large-Scale Assessment
2. The New Brunswick Provincial Assessment Program
~Mission
~Provincial Assessments (mandate, operational standards,
processes)
•English Second Language Assessment / Reassessment Reading Writing Oral Proficiency Listening
TOPICS
© 2005, Dynamic Measurement Group 3
GPS On Where are we?
You are Here
Port
Where do we want to be ?
How do we get there?
How are we doing?
ActualCourse
DesiredCourse
Assessment is the GPS for Educators
10
20
30
40
Dec.Scores
Feb.Scores
Jan.Scores
M archScores
AprilScores
MayScores
JuneScores
60
50
Rea
ding
Aimline
Balanced Assessment
• No single assessment approach is 'better' than another as each serves a slightly different role.
• All types of assessment must be planned with its purpose in mind.
Standardized testing, particularly, is justified as one resource to aid learning and should not be mistaken for a comprehensive verdict on learning or school quality. ~ Peter Grimmett & Paul Shaker
New Brunswick Assessment Program (NBAP)
Mission StatementThe mission for the Provincial Assessment and Evaluation Branch is to inform all stakeholders of student achievement by constructing valid, educationally beneficial, evaluations at critical points in the system. The purpose of the New Brunswick Assessment Program (NBAP) is to: determine if students demonstrate the skills and concepts outlined in New Brunswick curriculum documents; report provincially on how well students have achieved provincial standards at given points in their schooling; assist schools, districts, and the province in monitoring student learning and; coordinate the administration of national and international assessments.
New Brunswick Assessment Program
Operational Standards Processes How items are chosen
Standards for Educational and Psychological Testing
The Standards for Educational and Psychological Testing, 2008 edition are a collaborative publication from the American Educational Research Association, the American Psychological Association and the National Council on Measurement in Education. The intent of the Standards is to promote the sound and ethical use of tests and to provide a basis for evaluating the quality of testing practices.
The New Brunswick Assessment and Evaluation Branch adheres to the Standards when developing, administering, scoring and analysing large scale assessments within the scope of our mandate.
Operational Standards
Standards for Educational and Psychological Testing
The Assessment Cycle
There are four (4) stages within the overall Assessment Cycle for each of the 13 assessments; the (1) Development Stage, (2) Administration Stage, (3) Data Stage and the (4) Communication Stage.
Within each of the stages, there are several phases which require the expertise from a variety of staff.
Each of these stages require strict adherence to the Standards for Educational and Psychological Testing and the Principles for Fair Student Assessment in Canada.. These standards and principles will be referenced where appropriate.
Processes
0
0 .2
0 .4
0 .6
0 .8
1 .0
-3 -2 -1 0 1 2 3
b
Ab i l i ty
Pro
bab
ilit
y
Item Characteristic Curve: ITEM0001
a = 0 .5 3 9 b = 0 .2 0 2
1 - 7
8 - 14
15 - 21
22 - 28
29 - 35
36 - 42
43 - 48
Matrix Plot of Item Characteristic Curves for Book A
1 - 7
8 - 14
15 - 21
22 - 28
29 - 35
36 - 42
43 - 48
Matrix Plot of Item Characteristic Curves for Book A
English Second Language Assessment / Reassessment
ESLA/R
AGENDA - General
• ESLA Overview– 4 components– Summary of the data– What constitutes “Successful”– What is the standard for passing ESLA?– What are some important points that should be kept
in mind when writing the test?
Focus Areas
• Writing– How do we mark the writing?– Exemplars and Rationales
• Oral Proficiency Interviews– How long does the interview take?– Common topics and criteria for scoring– OPI Exemplars and Rationales
ESLA AT A GLANCE; GENERAL OVERVIEW
Test Specifications - OverviewEnglish Second Language Assessment (Grade 11)English Second Language Reassessment (Grades 12)
The English Second Language Assessment/Reassessment (ESLA/R) is an assessment of students’ English literacy abilities in all four language skills.
The ESLA/R is a requirement for students attending the Concord Colleges of Sino-Canada who wish to obtain a New Brunswick High School Diploma.
The ESLA/R is a criteria - standards based assessment where students are required to meet targets appropriate to two years of non-immersion English language instruction. Students in the Concord Colleges apply for acceptance to post-secondary English institutions.
English Second Language Assessment / Reassessment• 4 Components• Resembles the format of TOEFL except at a
high school level
ORAL
READING Weighted Equally
WRITING
LISTENING
OVERALL CALCULATION OF ESLA/ROral = H2*10
(Numeric oral value multiplied by 10)
Listening = L2*2.50
(Listening score multiplied by 2.50)
Overall score = SUM(I2,K2,M2,P2)/4 .
(overall score adding the four weighted scores, then dividing by 4.
Oral Ratings: ('U'=0) ('N'=3) ('B'=4) ('B+'=5) ('I'=6) ('I+'=7) ('A'=8) ('A+'=9) ('S'=10)
Writing Ratings: (DNW=*), ( BAA=1), (AA=2), and (SA=3)
cut point 43
0 - 43 Unsuccessful
ESLA/R READING
Reading Performance Levels ESLA
Reading Performance Levels
Reading Test Specifications
Book A --- Reading --- 90 minutes
7-9 Reading Passages
45-50 Number of items
Responding to Reading
Reading; Test difficulty and equating
• Unlike Writing and Oral Interviews, reading and listening are scored according to ‘cut points’.
• As the reading passages and items change, we need to ensure that the overall difficulty of the reading assessment is the same year over year – so that one cohort is being assessed on the same difficulty level as all other cohorts from previous administrations.
• So…how do we do this?
Test equating and cut point setting
ESLA/R WRITING
Performance Levels in Writing
Writing Test Specifications
Time allotment: Exercise I 60 minutesExercise II 60 minutes
The Writing component should be administered as either the first or last component. For Writing, students will answer directly in Booklet C. The Writing component consists of two writing activities: Exercise I and Exercise II.Writing prompts are provided on a separate sheet.
Marking the Writing Piece…
Writing is assessed using the Writing Achievement Standards – ESLA.
In order for a piece of writing to be deemed Appropriate Achievement or Strong Achievement it must possess all of the writing traits listed for that category.
ESLA/R LISTENING
Listening Stats
Listening Stats
Listening Test Specs
Time allotment: 30 minutes
Each booklet contains an information label with the student’s Chinese and English names as well as an identification number. Test administrators must ensure that each student receives the appropriate booklet.
Test administrators will read the items to the students using Instructions for Administrators of the Listening Assessment.
Students will respond to all Listening items directly in their booklet by circling either letter A, B, C, or D – whichever corresponds with the best response.
Book B - Listening consists of two components:
Part I - Answering Questions (Items 1-20)
Part II - Understanding Statements (Items 21-40)
Helpful hints for listening…
• Practice the “who, what, when, where, why” line of questions with the appropriate response.
• Warm-up activity to begin each class.
ESLA/R ORAL
Oral Interview Stats
OPI’s
Time allotment: 15 - 20 minutes
Interviews will be scheduled at twenty-minute intervals: the interview itself will be 10 - 15 minutes in duration. The interviewer will use the remaining time to determine and record the student's oral proficiency level and to prepare for the next interview.
FOCUS AREA; ORAL INTERVIEWS AND WRITING
OR
AL
INT
ER
VIE
W
ST
AN
DA
RD
S
LEVEL CRITERIA UNRATEABLE Can provide no intelligible answers including name
*Talks in own native language
NOVICE * Repeat/rephrase questions often Speaks in isolated words and phrases Not able to participate in conversations Some memorized material and ability to use it * Very limited vocabulary
BASIC
Survival Interviewer works hard to keep a conversation going
Can survive in a second language Can say a lot of simple things; buy basic food items, provide personal information, standard social exchange, school directions * Can get into, through and out of a situation Reactive, answering but not asking many questions – will ask if prompted Fluency slow, strained, except for routine expressions Pronunciation errors frequent / intelligible to someone who is used to ESL
BASIC PLUS
Some give and take in conversations
Can participate in short conversations, descriptions, narrations Shows more independence, spontaneity, asks questions but may require prompts to do so
INTERMEDIATE
More reciprocity - The interview begins to have a more conversational feel.
Can provide information Provide autobiographical information in some detail Give directions, accurate instructions, in area of personal experience or interest Describe / tell using past, present and future tenses Speak confidently, hesitates, uses paraphrases and fillers Pronunciation intelligible, often faulty Vocabulary sufficient to speak confidently
WR
ITIN
G S
TA
ND
AR
DS
ESLA Writing Exemplars
Package 1
Exemplars with rationales attached
QUESTIONS?