2/15/2006

2/15/2006

Online Testing Initiatives in Online Testing Initiatives in North CarolinaNorth Carolina

Presenters: Presenters: Randy CravenRandy CravenDirector for Information TechnologyDirector for Information TechnologyTechnical Outreach for Public Schools (TOPS), NC State Technical Outreach for Public Schools (TOPS), NC State UniversityUniversity

Scott RagsdaleScott RagsdaleProject Manager, North Carolina Computer Skills Project Manager, North Carolina Computer Skills Assessments +Assessments +Technical Outreach for Public Schools (TOPS), NC State Technical Outreach for Public Schools (TOPS), NC State UniversityUniversity

2/15/2006

Modern science has been a voyage into Modern science has been a voyage into the unknown, with a lesson in humility the unknown, with a lesson in humility waiting at every stop. Many passengers waiting at every stop. Many passengers would rather have stayed home.would rather have stayed home.

Carl SaganCarl Sagan

2/15/2006

The purpose of this session is to The purpose of this session is to update the audience on NCDesk, the update the audience on NCDesk, the Online Test of Computer Skills, and Online Test of Computer Skills, and other online assessments used in North other online assessments used in North Carolina testing.Carolina testing.

2/15/2006

What Initiatives?What Initiatives?

• Online Test of Computer SkillsOnline Test of Computer Skills• End-of-Course Online Physics Field Test (mandatory)End-of-Course Online Physics Field Test (mandatory)• End-of-Course Tests (optional 06-07)End-of-Course Tests (optional 06-07)• Writing Online Special Study (Grades 7 and 10)Writing Online Special Study (Grades 7 and 10)• Online formative assessments (i.e., ClassScape)Online formative assessments (i.e., ClassScape)• Other online resourced assessmentsOther online resourced assessments

– NCEXTEND1NCEXTEND1– NCCLASNCCLAS

• Other web applications supporting test Other web applications supporting test development/maintenance:development/maintenance:– Online Item Writing SystemOnline Item Writing System– Experimental Section Review (ESR)Experimental Section Review (ESR)

2/15/2006

Online Assessments SuccessOnline Assessments Success

• The Online Test of Computer Skills was The Online Test of Computer Skills was successful in its initial administration year of successful in its initial administration year of 2005-06 and continues to be successful this 2005-06 and continues to be successful this year.year.

• NCCLAS online data collection was successful NCCLAS online data collection was successful in 2005-06 and continues to work this year.in 2005-06 and continues to work this year.

• Online Physics Field Test was administered to Online Physics Field Test was administered to over 3,700 students during the fall administration over 3,700 students during the fall administration with no significant issues.with no significant issues.

• For 2006-07, For 2006-07, NCEXTEND1NCEXTEND1 students have been students have been matched to their assessors and have had their matched to their assessors and have had their Decision Tree processes completed.Decision Tree processes completed.

2/15/2006

Elements of Online AssessmentsElements of Online Assessments

• More interactiveMore interactive• Test Simulations required to be availableTest Simulations required to be available• More data can be collectedMore data can be collected• Opportunity to expand the use of innovative Opportunity to expand the use of innovative

itemsitems– Virtual/performance-based itemsVirtual/performance-based items– Short and/or extended responseShort and/or extended response

• Quicker data processing and reportingQuicker data processing and reporting• Reduction of materials managementReduction of materials management• Students have a positive attitude for online Students have a positive attitude for online

testingtesting

2/15/2006

Elements of Online Assessments Elements of Online Assessments (continued)(continued)

• TechnologyTechnology– Internet aware applications (NCDesk)Internet aware applications (NCDesk)

• Java versionJava version

– Browser based (EOC & Physics)Browser based (EOC & Physics)• Session State CookiesSession State Cookies• JavascriptJavascript• Flashpaper / Interactive Flash Items / Flash playerFlashpaper / Interactive Flash Items / Flash player• VersionsVersions

2/15/2006

Performance ComponentsPerformance Components

• Online Test of Computer SkillsOnline Test of Computer Skills– 50% multiple choice50% multiple choice– 50% performance50% performance

• Physics OnlinePhysics Online– Mostly multiple choiceMostly multiple choice– Small number of performance-based itemsSmall number of performance-based items

• EOC OnlineEOC Online– All multiple-choiceAll multiple-choice

• Online Writing Special StudyOnline Writing Special Study– Single prompt with essay responseSingle prompt with essay response

2/15/2006

Preservation of StatePreservation of State

• Students expect responses to items to retain Students expect responses to items to retain state as they move from one item to another and state as they move from one item to another and if they return to an item, it will be as they had left if they return to an item, it will be as they had left it before.it before.

• Preservation of state is not always easy since Preservation of state is not always easy since online assessment systems must be able to online assessment systems must be able to recover from disconnection.recover from disconnection.

• At this time for the Online Test of Computer At this time for the Online Test of Computer Skills and the EOC Online Tests we are able to Skills and the EOC Online Tests we are able to manage state for all items except the item the manage state for all items except the item the student is currently working on.student is currently working on.

2/15/2006

Problems/OpportunitiesProblems/Opportunities

• Rate of technology changeRate of technology change

• Window of opportunity for testingWindow of opportunity for testing

• SchedulingScheduling

• Having enough…Having enough…– BandwidthBandwidth– ComputersComputers– SpaceSpace– TimeTime

2/15/2006

Managing ThingsManaging Things

• NCRegistrationNCRegistration– User AccountsUser Accounts– Student RegistrationsStudent Registrations– Scheduling SessionsScheduling Sessions– Receiving ReportsReceiving Reports

2/15/2006

"There is no reason anyone would want a "There is no reason anyone would want a computer in their home." computer in their home."

Ken OlsonKen Olson

President, chairman and President, chairman and founder of Digital Equipment founder of Digital Equipment Corporation (DEC), maker of Corporation (DEC), maker of business minicomputers, business minicomputers, arguing against the PC in arguing against the PC in

1977. 1977.

2/15/2006

Online Test of Computer SkillsOnline Test of Computer SkillsNotable Preliminary/Unofficial Numbers from Notable Preliminary/Unofficial Numbers from

Fall 2006Fall 2006• 205 students tested in one day at one school (for 05-06 205 students tested in one day at one school (for 05-06

this was 139)this was 139)• 52 schools tested 100 or more students in one day (for 52 schools tested 100 or more students in one day (for

05-06 this was 7)05-06 this was 7)• 5,068 maximum testing sessions used in a single day 5,068 maximum testing sessions used in a single day

(for 05-06 this was 3,063)(for 05-06 this was 3,063)• 23 days of testing in which volume of used sessions was 23 days of testing in which volume of used sessions was

3,000 or more3,000 or more• ~133,980 total students tested~133,980 total students tested• ~106,035 8~106,035 8thth grade students tested grade students tested• ~2,228 different test administrators with 52 logging in ~2,228 different test administrators with 52 logging in

300 or more students300 or more students• ~3,446 students had the Computer Reads Aloud option~3,446 students had the Computer Reads Aloud option

2/15/2006

Online Progress IndicatorsOnline Progress IndicatorsMaximum Number of Students Tested in a Day

0

1000

2000

3000

4000

5000

6000

2005-06 Fall 2006

Number of Schools Testing 100 or More Students in a Day

0

10

20

30

40

50

60

2005-06 Fall 2006

Maximum Number of Students Tested in a Day at a School

0

50

100

150

200

250

2005-06 Fall 2006

Average Minutes on Test for Students with One Login

0

15

30

45

60

75

90

105

120

2005-06 Fall 2006

2/15/2006

Platform ShiftingPlatform Shifting

Fall 2006

Mac OS X (10.3.3-9)1.544%

Windows Me (4.90)0.002%

Windows 2000 (5.0)9.178%

Mac OS X (10.4.1-8)5.734%

Windows 98 (4.10)5.495%

Windows XP (5.1)78.046%

Fall 2005

Windows XP (5.1) 65.22%

Windows Me (4.90) 0.01%

Windows 98 (4.10) 16.05%

Windows 2000 (5.0) 10.04%

Mac OS X (10.4.x) 1.88%

Mac OS X (10.3.x) 6.73%

Mac OS X (10.2.x) 0.07%

2/15/2006

Online Computer SkillsOnline Computer SkillsItems for Spring 2007Items for Spring 2007

• Testing Window 3/12/07 – 6/8/07 Testing Window 3/12/07 – 6/8/07 • NCRegistration accessible on 2/5/07 for NCRegistration accessible on 2/5/07 for

scheduling, etc.scheduling, etc.• Reports available to LEA Test Reports available to LEA Test

Coordinators through NCRegistrationCoordinators through NCRegistration• JAWS compatibility not tested/approved – JAWS compatibility not tested/approved –

also newer versions of JAWS released also newer versions of JAWS released since last tested – research/development since last tested – research/development ongoing processongoing process

2/15/2006

Even if you're on the right track you'll Even if you're on the right track you'll get run over if you just sit there.get run over if you just sit there.

Will RogersWill Rogers

2/15/2006

NCDesk Moving ForwardNCDesk Moving Forward

• Adjustments for accommodating delivery of Adjustments for accommodating delivery of Writing assessments onlineWriting assessments online– Assessment selection on login pageAssessment selection on login page– Programmatically remove spell check functionsProgrammatically remove spell check functions– Programmatically remove image insertion functionsProgrammatically remove image insertion functions– Other features deemed necessaryOther features deemed necessary

• Additional features for testing audio functionalityAdditional features for testing audio functionality• Additional features for viewing correct responses Additional features for viewing correct responses

for the Test Simulation possiblefor the Test Simulation possible

2/15/2006

EOC Online (Optional)EOC Online (Optional)

• Fall 2006Fall 2006– First time EOCs offered online as an optionFirst time EOCs offered online as an option– One school participated in Algebra IOne school participated in Algebra I

• No problems reportedNo problems reported• Students “enjoyed” itStudents “enjoyed” it

• Spring 2007Spring 2007– Registration & Scheduling now available via Registration & Scheduling now available via

NCRegistrationNCRegistration• Not included:Not included:

– Chemistry Field Test Chemistry Field Test – Physical Science Field Test Physical Science Field Test – Biology Field TestBiology Field Test

2/15/2006

EOCs Online EOCs Online Moving Forward (Next Year)Moving Forward (Next Year)

• Making the answer choices clickableMaking the answer choices clickable• Improvement of the calculatorImprovement of the calculator

– Moveable/draggable on screen (within test interface)Moveable/draggable on screen (within test interface)– Add graphing capabilityAdd graphing capability

• Improvement of online reference contentImprovement of online reference content– Moveable/draggable on screen (within test interface)Moveable/draggable on screen (within test interface)

• Attempt to improve item view state preservationAttempt to improve item view state preservation– PositionPosition– Zoom percentageZoom percentage

• Optional to continue until such time as it is no longer Optional to continue until such time as it is no longer feasible to maintain paper-and-pencil testing.feasible to maintain paper-and-pencil testing.

2/15/2006

NCEXTEND2NCEXTEND2 Online 2007-08 Online 2007-08

• Process under way to present all Process under way to present all NCEXTEND2NCEXTEND2 in an online format in an online format

• Format similar to EOC Online Format similar to EOC Online

• Format for Writing would be delivered Format for Writing would be delivered using NCDesk with a single editor item.using NCDesk with a single editor item.

2/15/2006

Computers are useless. They can only Computers are useless. They can only give you answers.give you answers.

Pablo PicassoPablo Picasso