Transitioning to New Assessments

CALIFORNIA DEPARTMENT OF EDUCATIONTom Torlakson, State Superintendent of Public Instruction

Bilingual Coordinator’s NetworkTransitioning to New Assessments

STAR and SBACSeptember 20, 2012

Patrick Traynor, Ph.D.Director, Assessment Development & Administration Division

Transitioning to New Assessments

TOM TORLAKSONState Superintendent of Public Instruction

California Current Assessments

• Standardized Testing and Reporting Program (STAR)• California Standards Tests (CSTs)• Early Assessment Program (EAP)• California Alternate Performance Assessment (CAPA)• California Modified Assessment (CMA)• Standards-based Tests in Spanish (STS)

• California High School Exit Exam (CAHSEE)• California English Language Development Test (CELDT)• General Educational Development Test (GED)• California High School Proficiency Exam (CHSPE)• National Assessment of Educational Progress (NAEP)• Physical Fitness Test (PFT)



California Education Code Section 60604.5 requires the State Superintendent of Public Instruction (SSPI) to:•Develop recommendations addressing 16 areas toward reauthorization of the assessment system•Develop a plan for transitioning to new “high quality assessments”•Consult with specific stakeholders:

• The State Board of Education• Public Schools Accountability Act (PSAA) Committee• Measurement experts from California’s private and public universities• Experts in assessing students with disabilities and English learners• Teachers, administrators, and governing boards from California’s local educational

agencies• Parents

•Recommendations and Plan due to the Legislature fall 2012

3


“High-quality assessment”

• Means an assessment designed to measure a pupil’s knowledge of, understanding of, and ability to apply critical concepts through the use of a variety of item types and formats, including, but not limited to, items that allow for open-ended responses and items that require the completion of performance-based tasks

4


16 Areas of Consideration

The recommendations shall consider including all of the following elements in the reauthorized assessment system:(1) Aligning assessments to standards(2) Implementing common assessments developed by state collaborative (3) Conform to ESEA reauthorization

5


16 Areas of Consideration (Cont.)

(4) Measurement of achievement at a point in time and over time for groups and subgroups of pupils and for individual pupils.(5) Allow for comparison from one year to the next as a reflection of growth over time

(6) Valid, reliable, and fair for all students including English learners (ELs) and students with disabilities

6



(7) Assessments of ELs using primary language assessments

(8) Ensure no bias with respect to race, ethnicity, culture, religion, gender, or sexual orientation

(9) Incorporate a variety of item types including open-ended and performance-based tasks

7



(10) Generating multiple measures of pupil achievement, which, when combined with other measures, can be used to determine the effectiveness of instruction and the extent of learning

(11) Assess science and history–social science in all grade levels at or above grade 4

(12) Assess understanding and ability to use technology necessary for success in the 21st century classroom and workplace

8


9


(13) Formative and interim assessments that provide timely feedback for purposes of continually adjusting instruction to improve learning

(14) Use test administration and scoring technologies that will allow the return of test results to parents and teachers as soon as possible

(15) Minimizing testing time(16) Including options for diagnostic

assessments for pupils in grade 2


Types of Assessment

• Diagnostic — for learning

• Formative — for learning

• Interim — for learning and of learning

• Summative — of learning

10


Diagnostic Assessment• Ed Code 60603 defines diagnostic assessment as:

“ . . . assessment of the current level of achievement of a pupil that serves both of the following purposes:

1) The identification of particular academic standards or skills a pupil has or has not yet achieved.

2) The identification of possible reasons that a pupil has not yet achieved particular academic standards or skills.”

11


Formative Assessment

Ed Code 60603 defines formative assessment as:

“Assessment tools and processes that are embedded in instruction and are used by teachers and pupils to provide timely feedback for the purposes of adjusting instruction and to improve learning.”

12


Interim Assessments

Ed Code 60603 defines interim assessments as:“ . . . an assessment that is given at regular and specified intervals throughout the school year, is designed to evaluate a pupil’s knowledge and skill relative to a specific set of academic standards, and produces results that can be aggregated by course, grade level, school, or local education agency in order to inform teachers and administrator at the pupil, classroom, school, and local education agency levels.”

13


Summative Assessment

Ed Code 60603 defines “achievement test” as:

“ . . . any standardized test that measures the level of performance that a pupil has achieved in the core curriculum areas.”

14


Relationship of Assessments

15

Perie, Marion, Gong, Wurtzel, 2007


16

• Summative Assessments• Information for student, parent, teacher• Information for school and district • State and Federal Accountability

• Interim Assessments• Information for student, parent, teacher• Information for school and district

• Formative Processes and Tools• Information for student, parent, teacher

Sharing Assessment Information


Common Assessment Formats

• Paper and pencil testing– Most common type of format– Used for all assessments

• Computer based testing (CBT)– Uses fixed form but is administered using a

computer

• Computer adaptive testing (CAT)– Presents harder or easier questions, depending on

how the student performs as the test proceeds– Can provide greater accuracy with fewer questions

17


1818

Computer Adaptive Testing

• The system will adjust the difficulty of items throughout the assessment

Correct or Incorrect

Difficulty of Next Item

Correct More Difficult

Incorrect Easier


19

Computer Adaptive Testing20 Items


20

Six Item Types

• Selected Response• Short Constructed Response• Extended Constructed Response• Performance Tasks• Technology-Enabled• Technology-Enhanced


The Smarter Balanced Assessment Consortium

(SBAC)

• Area of Consideration 2 (of 16): Implementing common assessments developed by state collaborative

• Assessments for grades 3 - 8 and 11 aligned to the Common Core State Standards (ELA and Math)

• Operational across Consortium states in the 2014–15 school year

21


2222

25 states educating approximately 20 million of the nation’s public K to 12 students

21 governing, 4 advisory states

An Ever Changing Landscape


SBAC Procurements*

RFP ID RFP Name Contractor RFP Work Released Start DateRFP-00 Project Management Partner WestEd 06/04/2010 10/01/2010 RFP-01 Communications Services Provider GMMB 02/25/2011 06/03/2011 RFP-02 Technology Readiness Tool Pearson 05/20/2011 12/07/2011 RFP-03 IT Systems Architecture Measured Progress 05/27/2011 09/15/2011 RFP-04 Item Specifications Measured Progress/ETS 07/27/2011 12/01/2011 RFP-05 Psychometric Services ETS 10/07/2011 02/06/2012 RFP-06 Accessibility and Measured Progress 09/22/2011 01/04/2012 Accommodations Policy Guidelines RFP-07 Item Authoring/Item Pool Pacific Metrics 12/09/2011 04/03/2012 RFP-08 Item/Task Materials Development Measured Progress 10/14/2011 02/06/2012 RFP-09 Test and CAT Specifications ETS 10/25/2011 02/06/2012 RFP-11/ Test Engine Development for Pilot 03/07/2012 08/20/2012 and Field TestRFP-18/ and CAT SimulationsRFP-20 RFP-12 Initial Achievement Level CTB/McGraw-Hill 03/14/2012 07/13/012 Descriptors*Status current as of 8/22/12


SBAC Procurements (cont.)

RFP ID RFP Name Contractor RFP Work Released Start Date RFP-13 Translations 06/15/2012 07/15/2012 RFP-14 Item/Task Writing/Review — Pilot CTB/McGraw-Hill 12/13/2011 03/28/2012 RFP-15 Report Development 04/18/2012 Sept. 2012 RFP-16 Item/Task Development — Field Test Sept. 2012 Jan. 2013 RFP-17 Scoring Nov. 2012 Feb. 2013 RFP-19 Test Administration 04/13/2012 Sept. 2012 RFP-21 Standard Setting Jan. 2014 Apr. 2014 RFP-22 Sustainability Task Force 04/12/2012 Sept. 2012 RFP-23 Formative Assessment Practices Aug. 2012 Nov. 2012 and Professional Learning


Logical Components

Re-take option

Optional Interim assessment system—

Summative assessment for accountability

Last 12 weeks of year*

DIGITAL CLEARINGHOUSE of formative tools, processes and exemplars; released items and tasks; model curriculum units; educator training; professional development tools and resources; scorer training modules; and teacher collaboration tools.

Scope, sequence, number, and timing of interim assessments locally determined

* Time windows may be adjusted based on results from the research agenda and final implementation decisions.

Source: http://www.ets.org

Computer AdaptiveAssessment andPerformance Tasks

PERFORMANCETASKS

• Reading• Writing• Math

END OF YEARADAPTIVE

ASSESSMENT

English Language Arts and Mathematics, Grades 3–8 and High School

Computer AdaptiveAssessment andPerformance Tasks

BEGINNING OF YEAR

END OF YEAR

INTERIM ASSESSMENT INTERIM ASSESSMENT

26


2727

School-level Technology Issues

• Technology-enhanced test items will require compatibility with latest browser technologies

• Test security will require device to run in “kiosk mode”

• Devices must support text-to-speech software

• Test items with multimedia content will require sufficient internal and Internet network bandwidth


28

Technology Readiness Tool• Assesses current capacity and compares

to the technology that is expected to be required for new SBAC assessments

• Areas evaluated: devices, devices to tester ratio, network infrastructure, and staff and personnel

• Performs inventory, gap analysis and recommended areas where LEAs will need to upgrade technology


29

Technology Readiness Tool Data Collection

• First data collection window was from March 20, 2012 to June 30, 2012

• Subsequent data collections will occur each spring and fall through 2014

• Input information for computers expected to be available for online testing in 2014.


3030

Technology Readiness Tool Stats (Cont.)

• 600,000 devices entered into system• Operating systems inventory

– Windows XP: 52%

– Windows 7: 20%

– Mac OS X: 20%

– iOS 5.x: 3%

– Others (Vista, Linux, Windows – Other, iOS 4.x, Google Chrome, Android)


3131

Technology Readiness Tool Stats (cont.)

• Hardware types– Desktops: 67%

– Laptops: 19%

– Netbooks: 6%

– Tablets: 4%

– Thin Clients: 4%


32

Guidelines for Purchasing New Hardware

• Inform current and future purchases

• Covering the vast majority of commercially available computers and tablets

• Full guidelines: http://www.smarterbalanced.org/smarter-balanced-assessments/technology


California Work with SBAC: Opportunities for Teacher

Involvement• Feedback on draft content and item

specifications• Writing and reviewing of test items and tasks

(2012-13 pilot test; 2013-14 field test)• Range-finding and score validation• Collaborate on designing score reports and

Web tools for digital libraries (2013-14)• Scoring of performance tasks (2014-15 and

beyond)

33


Current CDE Participation in SBAC Work Groups

• SBAC Work Groups

– Assessment Design: Item Development

– Assessment Design: Performance Tasks

– Assessment Design: Test Design

– Accessibility and Accommodations

– Technology Approach

– Formative Assessment Practices and Professional Learning

• Deputy Superintendent Deb Sigman executive committee member

34


SBAC Resources• California Department of Education/SBAC

presentations be found on the CDE SBAC Web page at http://www.cde.ca.gov/ta/tg/sa/smarterbalanced.asp

• Join the SBAC CDE electronic mailing list by sending a blank e-mail to: [email protected]

35


36

Assessments to Consider (by content area and grade level)



California Education Code Section 60604.5 requires the State Superintendent of Public Instruction (SSPI) to:•Develop recommendations addressing 16 areas toward reauthorization of the assessment system•Develop a plan for transitioning to new “high quality assessments”•Consult with specific stakeholders:

• The State Board of Education• Public Schools Accountability Act (PSAA) Committee• Measurement experts from California’s private and public universities• Experts in assessing students with disabilities and English learners• Teachers, administrators, and governing boards from California’s local educational

agencies• Parents

•Recommendations and Plan due to the Legislature fall 2012

37


Statewide Assessment Reauthorization Timeline

38

March 21–22: Work Group

Meeting

April 17–18: Work Group

Meeting

May 22–23: Work Group

Meeting

June 12–14: Work Group

Meeting

July 25–26: Work Group

Meeting

Regional Public Meetings: 5 meetings between

March and April

Launch of Online Survey

Focus Groups: July -

September

Recommendations to the Legislature in

fall 2012

September 6: Work Group

Meeting

March 2012

April 2012

May 2012

June 2012

July 2012

August 2012

September 2012

October 2012

November 2012

Online Survey closes August

31

State Board of Education Meeting

Report writing and review

E-mail Account opened


Common Developing Themes• Diagnostic assessments: ELA and Math in grade 2 and

grades 3-12 as needed.• Formative Assessment Tools:

– SBAC: ELA and Math– Explore the state providing formative tools and processes for science and

History/Social-Science

• Interim assessments:– SBAC: ELA and Math Grades 3-8, 11– ELA and Math for all students in grade 2, 9-10– Science/HSS – state provided computerized interim assessments similar

to summative

• Summative: – SBAC assessments in grades 3-8 and 11 (ELA and Math). – Summative assessments for science and HSS in grades 3-12, for ELA

and math in grade 12, and writing in grades 3-12

39


Common Themes (Cont.)• Matrix sampling in some grades/subjects, specifically

science and history–social science (HSS)• Reduce linguistic complexity of assessments to more

accurately measure what students know and can do• Items that evaluate critical thinking, problem solving,

communication, collaboration, creativity, and innovation• Other measures: student engagement survey, parent

survey, graduation rates, attendance• Multiple measures: within classroom, (e.g., tests,

quizzes, projects, essays, journaling, class work, portfolios)

• Using results to satisfy the high school graduation requirement

40


41

SBAC TimelineField testing of

summative assessment,

training school- and district-level staff in formative

tools

Teams of teachers evaluate formative

assessment practices and

curriculum resources

Full implementatio

n of assessment

system

Formative Processes, Tools, and Practices Development Begins

Pilot Testing of Summative and Interim Items/Tasks Conducted

Writing and Review

Items/Tasks for Field Testing

(throughout the school year)

Technology readiness

tool available

Formative tools available

to teachers

Content and Item

Specifications Development

Writing and Review of Pilot Items/Tasks (including Cognitive

Labs and Small-Scale Trials)


Contact Information• Patrick Traynor, Director

Assessment Development and Administration Division – E-mail: [email protected]

• Jessica Barr, Education Programs ConsultantAssessment Development and Administration Division– Phone: 916-319-0364– E-mail: [email protected]

• Kristen Brown, Education Programs Consultant (SBAC Coordinator)

Assessment Development and Administration Division– Phone: 916-319-0334– E-mail: [email protected]

• Reauthorization E-Mail Account – [email protected]

42

Date post:	22-Feb-2016
Category:	Documents
Upload:	maine
View:	26 times
Download:	0 times

Transitioning to New Assessments

Documents