California K-12 Assessment Update

CALIFORNIA DEPARTMENT OF EDUCATIONTom Torlakson, State Superintendent of Public Instruction

California Education Research Association

November 30, 2012

Patrick Traynor, Ph.D.Director, Assessment Development and Administration Division

Eric Zilbert, Ph.D.Administrator, ADAD Psychometric Unit

California K-12 Assessment Update

TOM TORLAKSONState Superintendent of Public Instruction

Presentation Overview

• Recent and Upcoming SBAC Developments

• Scoring Technologies

• Alternate Assessment Participation (1% Population)

• Transitional Activities

2


Recent and Upcoming SBAC Developments

June-October 2012 Cognitive Labs

Oct 15-Nov 2, 2012 Small Scale Trials

October 9, 2012 Released Items

Nov 27, 2012-Jan 15, 2013

Draft Achievement Level Descriptors

Spring 2013 Pilot Test

Spring 2014 Field Test 3


SBAC Cognitive Labs

PurposeTo examine the impact of various item formats, tools, and accommodations on the ability of students to demonstrate in depth understanding of content measured.

Sampling– 700 students – ≈ 85 students in California

– Focus on SWD, ELL, and low SES students.

4


SBAC Cognitive Labs

Methodology •Research questions related to technology-based assessment

•A trained facilitator – administers test items– conducts a interview (≈ 90-120 minutes).

•Computer provided by the facilitator

•Labs (two approaches)– Think aloud as work through an item– First solve, then respond to questions about

approach

5


SBAC Small Scale TrialsPurposeTo inform automated and human scoring

Sampling•Stratified random sampling

– ≈ 900 schools across member states – ≈ 230 schools in California

•Goal: to be representative of SBAC students

•Schools – randomly select one to two classrooms– assigned grade (4, 7, or 11)

6


SBAC Small Scale Trials

Methodology

•Each assessment – 15-18 selected-response and constructed response

items – ≈ 60-90 minutes of student time

•School computers

•Each participating school designates a school coordinator who is trained to administer the assessment

•Schools are not required to assess all selected students at the same time or on the same day

7


Smarter Balanced Pilot Testing

• Smarter Balanced will conduct a Pilot computer-based administration of their assessment system beginning in February 2013.

• Items will be aligned to the Common Core State Standards and will include selected response, constructed response, and performance tasks

• Participation in the Pilot Test will be open to all schools in the Consortium and will be administered to students in grades 3–8 and grade 11.

8


Smarter Balanced Pilot Testing

• The Pilot Test entails two approaches (or components) in its implementation:

1) “Volunteer” component that is open to all schools in Smarter Balanced states and will ensure that all schools have the opportunity to experience the basic functionality of the system

2) “Scientific” component that targets a representative sample of schools and yields critical data about the items developed to date, as well as how the system is functioning

9


Pilot Participation Important Dates

The table below presents upcoming key activities related to the Smarter Balanced Pilot Test:

Activity Dates

Interested schools volunteer to participate via the online survey located at: https://www.surveymonkey.com/s/SmarterBalancedPilot

November 2012 – January 2013

Scientifically selected schools contacted to confirm their participation in the Pilot Test

Late-November 2012 – January 2013

Online training test is available Mid-January 2013

Smarter Balanced Pilot Test window

February 20 – May 10, 2013

10

https://www.surveymonkey.com/s/SmarterBalancedPilot

https://www.surveymonkey.com/s/SmarterBalancedPilot


Smarter Balanced Draft Achievement Level

Descriptors (ALDs)

• “Describe” four levels of achievement: “deep command,” “sufficient command,” “partial command,” or “minimal command” of knowledge, skills, and processes in both English–language arts/literacy and mathematics

• First draft ALDs for public comment from November 27, 2012 through January 15, 2013

• A full description of the ALDs and an online survey for providing feedback are available on the Smarter Balanced achievement level descriptors Web page at http://www.smarterbalanced.org/achievement-level-descriptors-and-college-readiness/

• 11

http://www.smarterbalanced.org/achievement-level-descriptors-and-college-readiness/



Smarter BalancedSample Items and Performance Tasks

12

Purpose of Sample Items and Performance Tasks

• Demonstrate rigor and complexity of ELA/literacy and mathematics questions

• Showcase variety of item types:• Selected response• Constructed response• Technology enhanced• Performance tasks

• Help teachers to begin planning for the shifts in instruction

Page 13

Overall Claim for Grades 3-8

Overall Claim for Grade 11

Claim #1 - Reading

Claim #2 - Writing

Claim #3 - Speaking and Listening

Claim #4 - Research/Inquiry

Claims for the ELA/Literacy Summative Assessment

Page 14

Overall Claim for Grades 3-8

Overall Claim for Grade 11

Claim #1 - Concepts & Procedures

Claim #2 - Problem Solving

Claim #3 - Communicating Reasoning

Claim #4 - Modeling and Data Analysis

Claims for the Mathematics Summative Assessment

Page 15

Exploring the Sample Items

16

Sample Items and Tasks Landing Page to Launch Oct. 9

Page 17

Sample Items and Tasks Navigation

View mathematics or ELA/literacy items

Advance to next item, or go back to previous

Page 18


Content Claim

Grade band

Page 19


Filter by item type, themes

Page 20

Item Metadata

About this item

Page 21

Exploring the Sample Items

Selected response and technology enhanced items are machine scorable

Page 22

Accessibility and Accommodations

• Sample items do not include accessibility and accommodations features

• As part of the development of accessibility and accommodations policies, the Consortium commissioned research on best practices for assessing English language learners and students with disabilities

• When fully operational, Smarter Balanced will be providing translation accommodation options for all math items

• For the Pilot, Smarter Balanced will be providing full Spanish translations and translated Spanish pop-up glossaries that are customized at the item-level for a specific form for each of three grades (almost 70 items each). Each grade-level (elementary, middle, and high school) will have one grade with the special form.

• The Consortium will need ELLs to participate in the Pilot to make sure they are represented in the student sample.

23

Accessibility and Accommodations

• Full range of accessibility tools and accommodations options under development guided by:– Magda Chia, Ph.D., Director of Support for Under-Represented

Students– Accessibility and Accommodations Work Group– Students with Disabilities Advisory Committee

• Chair: Martha Thurlow (NCEO)– English Language Learners Advisory Committee

• These teams of experts will ensure that the assessments provide valid, reliable, and fair measures of achievement and growth for both English learners and Students with Disabilities

• Smarter Balanced will also continue working with educators and experts in the field to design and test the assessment system

• Learn more online:– http://www.smarterbalanced.org/parents-students/support-for-

under-represented-students/24


Scoring Technology• Templates• Optical Scanning

– Scantron

– Electronic image based scoring (E.g. Pearson e-Pen)

– Scan to Score

• Traditional machine scoring– Dichotomous (correct/incorrect) scoring most common

– Exact word, number, or grid matches

– No partial credit

• Automated Scoring– Allows scoring of short answer and essay questions

– Require set of human scored papers to develop the scoring model

– Can give partial credit, or multiple point scores

25


How Automated Scoring Works

• Uses a set of human scored examples to develop a statistical model used to analyze answers

• Generally examine overall form and specific combinations of words

• Has an extensive library of possible meanings for words

26


What can be scored?

• Written responses– Prompt Specific Essays– Prompt Independent Essays– Short Answers– Summaries

• Spoken language – Correctness– Fluency

• Responses to simulations– Diagnosis of a patient’s illness– Landing a plane

27


How good is automated scoring?

• Consistent with the scores from expert human graders

• The way automated scores are produced is understandable and meaningful

• Fair• Validated against external measures in the

same way as is done with human scoring• The impact of automated scoring on reported

scores is understood

According to ETS, Pearson and the College Board in the recent, report “Automated Scoring for the Common Core Standards:”

28


Autoscoring Performance Response Assessment

Prompt Material

N Machine-Human

Correlation

Human-Human

Correlation

Source

Written 81 published essay prompts (grade 6-12)

400 0.89 0.86 Prentice Hall

18 research-leveled essay

prompts (grade 4-12)

635 0.91 0.91 MetaMetrics

5 synthesizing memos from

multiple sources

1239

0.88 0.79 Council for Aid to

Education

Spoken 2000 spoken English items

50 0.97 0.98 Balogh & et al. (2005)

3000 spoken Arabic items

134 0.97 0.99 Bernstein et al. (2009)

9 Oral Reading Fluency Passage

Grades 1-5

248 0.98 0.99 Downey et al. (2011)

Source: Streeter et. al. Pearson’s Automated Scoring of Writing, Speaking, and Mathematics, Pearson, May 2011 29

Example Essay Feedback

Source: Streeter et. al. Pearson’s Automated Scoring of Writing, Speaking, and Mathematics, Pearson May 2011

30

Data Requirements for Various Types of Automated Scoring

31

I tem Type Response Length in

Words

Typical Data Requirements for

development

Measures Returned

Prompt-Specific Essays

100-500 200-250 double-scored student essays

Overall score, trait scores, grammar & mechanics feedback

Prompt I ndependent Essays

(general models)

100-500 Approximately 1000 essays per grade

Overall score, select trait scores, grammar & mechanics feedback

Short Answers ~10-60 500 double-scored student answers

Total or partial-credit content score

Summaries 50-250 Readings to be summarized divided by major sections

Content coverage score for each section; checks copying, length, redundancy and irrelevance.

Source: Streeter et. al. Pearson’s Automated Scoring of Writing, Speaking, and Mathematics, Pearson, May 2011


Alternate Assessment Participation

• California recently joined the National Center and State Collaborative (NCSC) as a Tier II state

• Representing a Tier II state, the California team will:

– Dedicate a staff member to coordinate the work– Work directly with members of the Special Education Administrators of

County Offices of Education (SEACO) and with directors of special education local plan areas (SELPA) to build a community of practice

– Meet directly with the field implementers every other month with technology supported meetings in between and as needed

– Deliver electronically to California stakeholders the comprehensive curriculum, instruction, and professional development modules available from the NCSC on the CCSS expected by fall 2012

32


STAR In-Transition Activities

• Considerations– SBAC– CAT– ELA and Math Only– System – Formative, Interim, and

Summative

• Specific Planned Activities

33


Re-take option

Optional Interim assessment system—

Summative assessment for accountability

Last 12 weeks of year*

DIGITAL CLEARINGHOUSE of formative tools, processes and exemplars; released items and tasks; model curriculum units; educator training; professional development tools and resources; scorer training modules; and teacher collaboration tools.

Scope, sequence, number, and timing of interim assessments locally determined

* Time windows may be adjusted based on results from the research agenda and final implementation decisions.

Source: http://www.ets.org

Computer AdaptiveAssessment andPerformance Tasks

PERFORMANCETASKS

• Reading• Writing• Math

END OF YEARADAPTIVE

ASSESSMENT

English Language Arts and Mathematics, Grades 3–8 and High School

Computer AdaptiveAssessment andPerformance Tasks

BEGINNING OF YEAR

END OF YEAR

INTERIM ASSESSMENT INTERIM ASSESSMENT

34

Formative Assessments


35

Computer Adaptive Testing


36

Assessments to Consider (by content area and grade level)


Relationship of Assessments

37

Perie, Marion, Gong, Wurtzel, 2007


3838

STAR In-Transition Activities

Alignment to Common Core State Standards

•Planning to provide CCSS-aligned results on STAR Student Reports

•Planning to align CST released test questions (RTQs) with California’s Common Core State Standards (CCSS)


39

Proposed Web Site RedesignHome Page


40

Proposed Web Site RedesignCurrent Testing Standards—Filtering Feature


41

Proposed Web Site RedesignCurrent Testing Standards—Information About the Standard


42

Proposed Web Site RedesignCurrent Testing Standards—Sample Question


43

Proposed Web Site RedesignCurrent Testing Standards—Related CCSS Information


44

Proposed Web Site RedesignCommon Core—Filtering Feature


45

Proposed Web Site RedesignCommon Core—Standards Information


46

Proposed Web Site RedesignCommon Core—Comparison to Current Testing Standards


47

Proposed Web Site RedesignCommon Core—Example Related Sample Question


Contact Information• Patrick Traynor, PhD, Director

Assessment Development and Administration Division – E-mail: [email protected]

• Jessica Valdez, Administrator, Transition OfficeAssessment Development and Administration Division– Phone: 916-319-0332– E-mail: [email protected]

• Jessica Barr, SBAC and Reauthorization Lead Consultant, Transition OfficeAssessment Development and Administration Division– Phone: 916-319-0364– E-mail: [email protected]

48

Date post:	03-Jan-2016
Category:	Documents
Upload:	grizelda-thomas
View:	25 times
Download:	0 times

California K-12 Assessment Update

Documents