Post on 26-Dec-2015
transcript
April 2008 Getting Involved in STAR 1
STARSTAR Item Writing, Item Writing,
Assessment Review Panels, Assessment Review Panels, Standard Setting,Standard Setting,and Field Testingand Field Testing
WebcastWebcastApril 23, 2008
START TIME 9:00 a.m.Presenters
Jonathan Halling - ETSStephanie Sartain Herrera - ETS
Mary Anne Arcilla - ETSMark McLean - CDE
April 2008 Getting Involved in STAR 2
AgendaAgenda
1. Life of a Test Item2. Item Writer Training3. Assessment Review Panels (ARPs)4. Standard Setting5. Field Testing6. Open Forum
April 2008 Getting Involved in STAR 4
Study state guiding documents
ARP & CDEdevelop blueprints
SBE action
Develop item writer
materials
CDE review of item writer materials
Train item writers
Receive itemsPrepare, review, and revise items
ARP & CDE review items
Revise itemsSPARreview
Create field test sets with CDE review
Conduct field tests
Score and process data from field test
ARP & CDE review data
from field test
Create operational forms
ARP & CDE review forms
Administer operational test
April 2008 Getting Involved in STAR 5
Study state guiding documents
Develop item writer
materials
CDE review of item writer materials
Train item writers
Receive itemsPrepare, review, and revise items
ARP & CDE review items
Revise itemsSPARreview
Create field test sets with CDE review
Conduct field tests
Score and process data from field test
Create operational forms
ARP & CDE review forms
Administer operational test
ARP & CDE review data
from field test
ARP & CDEdevelop blueprints
SBE action
April 2008 Getting Involved in STAR 6
Study state guiding documents
ARP & CDEdevelop blueprints
SBE action
Develop item writer
materials
Create field test sets with CDE review
Conduct field tests
Score and process data from field test
Create operational forms
ARP & CDE review forms
Administer operational test
ARP & CDE review data
from field test
CDE review of item writer materials
Train item writers
Receive itemsPrepare, review, and revise items
ARP & CDE review items
Revise itemsSPARreview
April 2008 Getting Involved in STAR 7
Study state guiding documents
ARP & CDEdevelop blueprints
SBE action
Develop item writer
materials
CDE review of item writer materials
Train item writers
Receive itemsPrepare, review, and revise items
ARP & CDE review items
Revise itemsSPARreview
Create field test sets with CDE review
Create operational forms
ARP & CDE review forms
Administer operational test
Conduct field tests
Score and process data from field test
ARP & CDE review data
from field test
April 2008 Getting Involved in STAR 8
Study state guiding documents
ARP & CDEdevelop blueprints
SBE action
Develop item writer
materials
CDE review of item writer materials
Train item writers
Receive itemsPrepare, review, and revise items
ARP & CDE review items
Revise itemsSPARreview
Create field test sets with CDE review
Conduct field tests
Score and process data from field test
Create operational forms
ARP & CDE review data
from field test
ARP & CDE review forms
Administer operational test
April 2008 Getting Involved in STAR 10
Study state guiding documents
ARP & CDEdevelop blueprints
SBE action
Develop item writer
materials
CDE review of item writer materials
Train item writers
Receive itemsPrepare, review, and revise items
ARP & CDE review items
Revise itemsSPARreview
Create field test sets with CDE review
Conduct field tests
Score and process data from field test
CDE & ARP review data
from field test
Create operational forms
ARP & CDE review forms
Administer operational test
April 2008 Getting Involved in STAR 11
Item WriterItem WriterTrainingTraining
• What will be covered by ETS in the training session?– What makes a good item– The item-writing process– How to write items specifically to meet California
standards
• Who should attend? Persons who:– Possess knowledge of California content
standards – Are current or recent teachers– Have at least three (3) years teaching experience
April 2008 Getting Involved in STAR 12
Item WriterItem WriterTrainingTraining
• How many are trained? – 20–30 attendees per content area in CAPA, CST,
CMA, and STS
• Where does training take place?– 2008: Long Beach, CA
• When does the 2008 training occur?– July 15–16: CAPA items – July 17–18: CST, CMA, and STS items
April 2008 Getting Involved in STAR 13
Item WriterItem WriterTrainingTraining
• Benefits– Professional development– Increased familiarity with California content
standards– Potential for becoming professional item writer
• Expenses– $150 per day or substitute teacher reimbursement– Travel and lodging expenses paid by ETS
April 2008 Getting Involved in STAR 15
Assessment Review PanelsAssessment Review Panels
Panel members meet in an advisory capacityto ETS and CDE to review:
• Item specifications and blueprints• Proposed new test items• Data collected on field-tested items • Proposed new operational test forms• Proposed released test questions (RTQs)
– CST only
• Differential Item Functioning (DIF)
April 2008 Getting Involved in STAR 16
Study state guiding documents
ARP & CDEdevelop blueprints
SBE action
Develop item writer
materials
CDE review of item writer materials
Train item writers
Receive itemsPrepare, review, and revise items
ARP & CDE review items
Revise itemsSPARreview
Create field test sets with CDE review
Conduct field tests
Score and process data from field test
CDE & ARP review data
from field test
Create operational forms
ARP & CDE review forms
Administer operational test
April 2008 Getting Involved in STAR 17
Assessment Review PanelsAssessment Review PanelsItem Specifications and BlueprintsItem Specifications and Blueprints
• Help define recommended blueprints for new tests– Develop foundation for future item and test
form development – Examples:
• Revised General Mathematics– Conducted April 2, 2008
• California Modified Assessment (Grades 9–11)
April 2008 Getting Involved in STAR 18
Study state guiding documents
ARP & CDEdevelop blueprints
SBE action
Develop item writer
materials
CDE review of item writer materials
Train item writers
Receive itemsPrepare, review, and revise items
ARP & CDE review items
Revise itemsSPARreview
Create field test sets with CDE review
Conduct field tests
Score and process data from field test
CDE & ARP review data
from field test
Create operational forms
ARP & CDE review forms
Administer operational test
April 2008 Getting Involved in STAR 19
Assessment Review PanelsAssessment Review PanelsItem ReviewItem Review
• Advise if the items developed are:– Measuring the standard – Grade-level appropriate– Free from bias
• Ethnic • Socioeconomic• Gender• Geographical
– Interesting and appropriate to students
• Suggest ways in which an item may be adjusted to meet the above criteria, if necessary
• Do NOT rewrite items
April 2008 Getting Involved in STAR 20
Study state guiding documents
ARP & CDEdevelop blueprints
SBE action
Develop item writer
materials
CDE review of item writer materials
Train item writers
Receive itemsPrepare, review, and revise items
ARP & CDE review items
Revise itemsSPARreview
Create field test sets with CDE review
Conduct field tests
Score and process data from field test
CDE & ARP review data
from field test
Create operational forms
ARP & CDE review forms
Administer operational test
April 2008 Getting Involved in STAR 21
Assessment Review PanelsAssessment Review PanelsData ReviewData Review
• Recommend whether field-tested items should be included in the “bank” and hence be available to be selected for inclusion on future operational tests– Review statistics on the item’s performance
as a field-test item– No editing of items is performed at this
stage
April 2008 Getting Involved in STAR 22
Assessment Review PanelsAssessment Review Panels Differential Item Functioning (DIF)Differential Item Functioning (DIF)
ReviewReview• Occurs when item performance is significantly
different for members of reference groups and comparison groups
• Participants representing various demographic groups review test questions to determine whether or not items are appropriate
• One-day meeting, conducted after field testing
Note: DIF review panel consists of different participants from other ARPs
April 2008 Getting Involved in STAR 23
Study state guiding documents
ARP & CDEdevelop blueprints
SBE action
Develop item writer
materials
CDE review of item writer materials
Train item writers
Receive itemsPrepare, review, and revise items
ARP & CDE review items
Revise itemsSPARreview
Create field test sets with CDE review
Conduct field tests
Score and process data from field test
CDE & ARP review data
from field test
Create operational forms
ARP & CDE review forms
Administer operational test
April 2008 Getting Involved in STAR 24
Assessment Review PanelsAssessment Review PanelsForms ReviewForms Review
• Last stage in the process of building the test
• Assembled operational forms:– Contain a range of difficulty of items– Match the blueprint and statistical
specifications– Avoid clueing
• No editing of items
April 2008 Getting Involved in STAR 25
Assessment Review PanelsAssessment Review PanelsReleased Test Question (RTQ) Released Test Question (RTQ)
ReviewReview• After use on operational test forms,
some questions are released for review and use by the public– Contain a range of item difficulty– Are proportional to the blueprint – Released annually
• No editing of items
April 2008 Getting Involved in STAR 27
2008 ARP Meeting Schedule2008 ARP Meeting ScheduleCSTCST
• Item Review – April 2–5
• Data Review– July 29–30
• Forms Review– September 23
• Released Test Questions– October 29
Note: ARP members are expected to attend meetings at ALL stages of development
April 2008 Getting Involved in STAR 28
2008 ARP Meeting Schedule2008 ARP Meeting Schedule CMA CMA
• Item Review– Grades 3–8
• September 9–12
• Data and Forms Review– Grades 6–8
• November 19–20
• Data Review – Grades 3–5
• To be scheduled early 2009
Note: ARP members are expected to attend meetings at ALL stages of development
April 2008 Getting Involved in STAR 29
2008 ARP Meeting Schedule2008 ARP Meeting ScheduleCAPACAPA
• Item Review – May 13–14
• Data Review– September 25
• Forms Review– To be scheduled early 2009
Note: ARP members are expected to attend meetings at ALL stages of development
April 2008 Getting Involved in STAR 30
2008 ARP Meeting Schedule2008 ARP Meeting Schedule STS STS
• Item Review– RLA grades 8–11, Algebra I, and Geometry
• April 8–11– RLA and Math grades 2–7
• July 8–11
• Data Review– RLA and Math for grades 2–7
• October 8–9
• Forms Review– Grades 2–4
• October 10
Note: ARP members are expected to attend meetings at ALL stages of development
April 2008 Getting Involved in STAR 32
The ProcessThe Process
1. Review the ARP member requirements2. Obtain application form:
• E-mail star@ets.org• Call Kristin Elizalde 925-808-2009
3. Submit a completed application form4. Application reviewed by:
• ETS content experts• CDE Standards and Assessment staff• State Board of Education staff
5. Application approved or denied
Note: Process can take several weeks
April 2008 Getting Involved in STAR 33
ARP Members Needed!ARP Members Needed!
• California Standards Tests– Science (Grades 5 and 8)
– History–Social Science (Elementary and Middle School)
– ELA (All grades)
• California Modified Assessment– English–Language Arts (ELA)
– Mathematics
– Science
Note: CMA – Middle and High School especially needed!
April 2008 Getting Involved in STAR 34
ARP Members Needed!ARP Members Needed!
• California Alternate Performance Assessment– English–Language Arts (ELA)– Mathematics– Science
• Standards-based Tests in Spanish– Reading/Language Arts (RLA)– Mathematics, Algebra I, and Geometry
Note: STS – Middle and High School especially needed!
April 2008 Getting Involved in STAR 35
Benefits of Being an ARP Benefits of Being an ARP MemberMember
• Become more familiar with the testing program
• Professional development– Learn more about what attributes a high
quality test item should display– Learn more about assessing how an item
performs on a test
• Contribute to the development of high-quality tests
• All meeting attendance expenses covered
April 2008 Getting Involved in STAR 37
ARP Member Requirements ARP Member Requirements All Panels All Panels
• Educators from– Urban areas– Rural areas– Suburban areas
• Experienced in teaching K–12 students from diverse populations
April 2008 Getting Involved in STAR 38
ARP Member RequirementsARP Member RequirementsCST, CMA, CAPA, and STSCST, CMA, CAPA, and STS
• Bachelor’s or higher degree in content area or related subject
• Knowledge of California content standards• Familiarity with student capabilities• K–12 teachers must also have:
– Three or more years teaching courses in grades K–12• ELA, math, science, or history–social science
• County, district, or university content specialists must also have:– Three or more years as a district or county content
specialist or university instructor• ELA, math, science, or history–social science
April 2008 Getting Involved in STAR 39
Additional ARP Member Additional ARP Member Requirements Requirements
for CMAfor CMA• English–language arts, mathematics, and science only• K–12 teachers
– Three or more years of experience teaching ELA, math, or science in grades K–12 and experience working with special education students
– Experience with more than one type of disability
• School administrators, district/county content/program specialists, or university educators– Three or more years of experience as a school administrator,
district/county content/program specialist, or university instructor in the area of special education or areas related to English–language arts, mathematics, or science
April 2008 Getting Involved in STAR 40
Additional ARP Member Additional ARP Member Requirements Requirements
for CAPAfor CAPA• English–language arts, mathematics, and science only• K–12 teachers
– Experience with students with severe cognitive disabilities – Experience with more than one type of disability– Three or more years of K–12 experience teaching special
education or teaching ELA, math, or science in a general education classroom
• School administrators, district/county content/program specialists, or university educators– Three or more years of experience as a school administrator,
district/county content/program specialist, or university instructor in the area of special education or areas related to English–language arts, mathematics, or science
April 2008 Getting Involved in STAR 41
Additional ARP Member Additional ARP Member Requirements Requirements
for STSfor STS
• Reading/language arts (RLA) and mathematics only
• Bilingual and biliterate in Spanish • Three or more years experience
teaching students in Spanish• Bachelor’s or higher degree in Spanish
or a subject area related to bilingual education, reading/language arts, or mathematics
April 2008 Getting Involved in STAR 44
What is Standard Setting?What is Standard Setting?• Process used to recommend the
minimum test score needed for a student to be considered performing at a certain performance level– For example, the minimum test score needed to be
at the proficient level
• Bookmark standard setting methodology – Participants review test items that have been
sequenced from the easiest to most difficult item– Participants identify the items that likely exceed the
ability of a student at each of the California performance levels
– The difficulty level of these identified items form the basis for the minimum test scores needed to reach each performance level
April 2008 Getting Involved in STAR 45
Standard SettingStandard SettingCAPA Levels I–V CAPA Levels I–V
(ELA, Math, and Science) (ELA, Math, and Science)
• Three-day meeting – September 16–18, 2008
• Participant requirements same as ARP members
April 2008 Getting Involved in STAR 46
Standard SettingStandard SettingCMA Grades 3–5CMA Grades 3–5
(ELA, Math, and Science) (ELA, Math, and Science)
• Two three-day meetings – ELA
• September 24–26, 2008
– Math and Science• October 1–3, 2008
• Participant requirements same as ARP members
April 2008 Getting Involved in STAR 47
Standard SettingStandard SettingSTS Grades 2–4STS Grades 2–4RLA and Math RLA and Math
• Three-day meeting – RLA
• January/February 2009
– Math• January/February 2009
• Participant requirements same as ARP members
April 2008 Getting Involved in STAR 48
Standard SettingStandard SettingMember Requirements Member Requirements
• Educators from:– Urban, rural, and suburban areas– Large and small districts– Geographically dispersed throughout California
• Experienced in teaching students in appropriate grades/content areas and from diverse populations
• Community members– Parent Teacher Association (PTA)– School board– Business
April 2008 Getting Involved in STAR 49
Standard SettingStandard SettingFor CAPA, CMA and STSFor CAPA, CMA and STS
• Application Form– E-mail star@ets.org
or– Call Kristin Elizalde 925-808-2009
• Reimbursement for:– Travel and accommodation expenses– Substitute teacher for school/district
Apply now!
April 2008 Getting Involved in STAR 51
Field TestField TestCalifornia Modified Assessment California Modified Assessment
(CMA)(CMA)
• September 15–29, 2008• Eligible students
– IEP– Not taking CAPA– Administered to grades 5, 7, 8, and 9
• ELA (grades 7, 8, and 9)• ELA writing (grades 5 and 8)• Mathematics (grades 7 and 8)• Science (grade 9)
– Student may test in one or more content areas– 2008 CST results
• Far below basic• Below basic
April 2008 Getting Involved in STAR 52
QuestionsE-mail: star@ets.org
Subject: 2008 CMA Field TestTelephone: STAR TAC 800-955-2954
CMA Field TestCMA Field Test
April 2008 Getting Involved in STAR 53
Questions
E-mail: star@ets.org
Subject: April Webcast
Open ForumOpen Forum