Introducing the Computer-Based Test for the National ...

1. Research Background

3. Research Content

2. Research Goal

4. Policy Recommendations

Research Report 2020

Introducing theComputer-Based Test for the

National Assessmentof Educational Achievement:

A Developmental Studyfor the eNAEA

Head Researcher│Jaebong Lee

Research Report 2020

PIM 2021-8-3

III. Introducing the Computer-Based Test

for the National Assessment of Educational

Achievement: A Developmental Study for

the eNAEA

KICE Research Report 2020

Published Date August 27, 2021

Publisher Tae-Joong Gahng

Publishing AgencyKorea Institute for Curriculum and Evaluation

Address8, Gyohak-ro, Deoksan-eup, Jincheon-gun, Chungcheongbuk-do, Republic of Korea

Telephone +82-43-931-0114

Fax +82-43-931-0884

Home Page http://www.kice.re.kr

Printing Company KS SENSATION (+82-2-761-0031)

ISBN 979-11-91582-35-2 95370 979-11-91582-44-4 95370 (Set)

PIM 2021-8-3

※ Unauthorized copy of contents of this report is prohibited.

ISBN 979-11-91582-35-2 95370

ISBN 979-11-91582-44-4 95370 (Set)


IIIIntroducing the

Computer-Based Test for the National Assessment of

Educational Achievement:

A Developmental Study for the eNAEA

Head Researcher | Jaebong Lee

Co-researchers | Junshik Kim, Jiseon Park, Kyunghee Sung,

Kwangsang Lee, Sora Lee, Hyeyun Jung, Soyoung Choi,

Kamyoung Kim (Kyungpook National University),

Yumin Ahn (Kongju National University),

Minsu Ha (Kangwon National University)


4

1. Research Background

◈ Need to establish methods of assessment in line with the achievement standards

and competency-based assessment introduced in the 2015 Revised National

Curriculum and the international trend toward the introduction of computer-based

assessment

◈ Need to search for a new direction of the National Assessment of Educational

Achievement (NAEA) that can accommodate the educational innovations for

the future society

○ With the dawn of the Fourth Industrial Revolution and the transition toward

an intelligent information society, countries around the world are stepping

up their efforts to nurture and assess new competencies for the present

demands. Likewise, Korea has established the importance of competency in

the subject curriculum and general guidelines for the 2015 Revised National

Curriculum.

○ Large-scale and international assessment of educational achievement is

generally working toward computer-based assessment, both to provide the

effective evaluation of the competencies required for the future society and

as a method of overcoming the limitations of the traditional pencil-and-paper

assessment.

○ Likewise, NAEA must provide a holistic evaluation of its contents to guarantee

the nurturing of creative talent demanded by the society of the future and

look for new methods of assessment items in terms of methodology to guarantee

the appropriate measurement of student competency.

◈ Need to establish specific methods of computer-based assessment development

in order to support the policy of the Ministry of Eduction to fully implement the

electronic National Assessment of Educational Achievement (eNAEA) in 2022.

○ In March 2019, the Ministry of Education of Korea presented a plan for the

full administration of computer-based assessment in 2022 as a plan to

Introducing the Computer-Based Test for the National Assessment of Educational Achievement:A Developmental Study for the eNAEA

5

improve NAEA (Ministry of Education, 201p9).

○ Providing an effective response to this policy requires the establishment of

methods for implementing transitory strategies for computer-based

assessment. Concrete methods must also be established for the implementation

of the eNAEA, including new assessment and subject-specific items.

2. Research Goal

○ This study seeks to establish the methods of introducing the computer-based

test for the NAEA, which are in line with the recent technological

development and demands from social and political sectors, including the

entrance into an intelligent information society and the recent revision of

the National Curriculum.

○ This study proposes an effective test development procedure to prepare for

the upcoming 2022 eNAEA.

○ By exploring the strategies to facilitate a stable transition to a computer-

based assessment scheme, this study establishes the directions for eNAEA

test item development, devises valid item types, develops subject-specific

plans and sample items, and establishes methods of developing test items in

line with the requirements of computer-based assessment.

3. Research Content

◈ Literature review and direction-setting for the transition toward computer-based

assessment

○ Implications from the analysis of transition underwent through PISA, TIMSS,

NAEP, and NAPLAN

― These cases involved a gradual transition toward computer-based assessment,

and by doing so, realized the competency-based assessment, hitherto thought

to be difficult in pencil-and-paper assessment, by introducing the innovative


6

test items that allow test-takers to interact with the computer.

― In the case of NAEA, where the transition must be completed in a relatively

short period, there is a need to exercise the utmost care in preparing the

transition from the current paper-based test and development of new

innovative items and test-set designs to minimize the trial-and-error

process during the transition.

○ Implications from the literature review of the transition to computer-based

assessment

― Studies that discussed the opportunities and risks involved with the

transition of large-scale assessment to computer-based formats (Bridgeman,

2009; Bryant, 2017; de Klerk, 2012; Drasgow & Mattern, 2006; Reich &

Petter, 2009; Sireci & Zenisky, 2006; Strain-Seymour et al., 2009; van Lent,

2009) were analyzed in this study.

― Opportunities associated with the transition toward computer-based

assessment are improvement in the assessment through the introduction of

innovative items, improvement in student motivation and convenience, and

improvement of test administration efficiency through the electronification

of the system. Risks include issues in problems of proving the enhancement

of the construct validity, possibility of decreasing measurement efficiency,

as well as increase of test administration costs and security-related

management points.

― Transition strategies that are deemed suitable for the transition of NAEA

to a computer-based system: 1) strategies involving the innovative change

of the items themselves beyond the mere improvement of the system

efficiency, which can help the assessment system reduce the gap between

the assessment and real world, and induce educational innovations; and 2)

strategies that maximize the connection between the conventional

pencil-and-paper assessment and computer-based assessment and guarantee

its stable implementation.


7

○ Setting the direction for test development in eNAEA

― [Figure I-1] describes the transition trajectory from a conventional pencil-

and-paper-based system to the recent introduction of competency- and

computer-based assessment.

Ability-Based Assessment

Paper-and-Pencil Assessment

Traditional Knowledge-Based Assessment

Computer-Based Assessment

Direction of NAEA development

The 2009 Revised National Curriculum based paper-and-pencil assessment

The 2015 Revised National Curriculum based paper-and-pencil assessment

Research area for the present study

Method

Content

[Figure I-1] Direction of developments in NAEA following the introduction of

competency- and computer-based assessment

― Based on the analysis of international and overseas cases of the transition

to computer-based assessment, opportunities and risks associated with the

transition, and transition strategies, this study established eight aspects in

the direction-setting of the eNAEA test item development.

― Directions for eNAEA test item development: ① Develop items that are

appropriate to the goal of NAEA (examining the quality of the curriculum).

② Develop items that clearly represent the measurement construct. ③

Apply technical features only when the purpose of application is clear. ④

Develop items that encourage interactivity. ⑤ Apply technical features in

a step-wise fashion to accommodate students’ technical ability. ⑥ Develop

subject-specific items that are typical and applicable to the school


8

context. ⑦ Develop items that maximize the utility of automatic scoring.

⑧ Develop items that secures stable test administration.

― These eight directions serve as basic principles of item format development

or system design and can be used as the criteria in the important

decision-making processes.

◈ Investigating and setting eNAEA item types

○ Types of test items utilizing technical features were set, in line with the

characteristics with NAEA under the 2015 Revised National Curriculum that

emphasizes competency.

― A list of major technical features applied in eNAEA can be found in <Table

I-1> and <Table I-2>.

<Table I-1> Technical features for response actions

Technical Feature Example a DescriptionSubject-Specific Utility

K SS M S E

Radio Button∙ Multiple choice question with one answer out

of five possible choices▲ ▲ ▲ ▲ ▲

Check Box∙ Multiple choice question with two or more

answers out of multiple (three to five) options▲ ▲ ▲ ▲ ▲

Matching∙ Matching the items in line with content

relations, information or other criteria▲ ▲ ▲ ▲ ▲

Pull-Down

∙ Inserting information by choosing one of the

answers from the pull-down menu

∙ Other names: Inline choice, drop-down

menu

▲ ▲ ▲ ▲ ▲

Hot Spot

∙ Clicking a specific area in a figure or text

(number, word, phrase, sentence, etc.)

∙ Other names: Figural response for nontextual

responses

▲ ▲ ▲ ▲ ▲

Drag and Drop

∙ Choosing an image or a word by clicking and

dragging to the answer field

∙ Includes the task of re-ordering the series of

numbers, words, phrases, sentences, etc.

▲ ▲ ▲ ▲ ▲


9

<Table I-2> Technical features for interactional activities

Technical Feature Example a DescriptionSubject-Specific Utility

K SS M S E

Graphing and

Drawing

∙ Includes connecting two points on a graph

with a line

∙ Includes free drawing functions on a separate

field

▽ ▲ ▲ ▲ ▽

Short Answer∙ Inputting characters or numbers into the text

box using the keyboard▲ ▲ ▲ ▲ ▲

Writing

Response

∙ Answers that are longer than a short answer

∙ Provided with a separate editing function

(cutting, pasting, underline, italics, etc.) for

free text response

▲ ▲ ▲ ▲ ▲

Math Symbols

∙ Mostly used in the math tests

∙ Can be used to enter chemical reaction

equations in the science tests

∙ Requires a separate math symbol input board

▽ ▽ ▲ ▲ ▽

Table

Completion

∙ Inputting numbers, symbols, characters, and

other answers into the answer field in the

table using a keyboard

▽ ▲ ▽ ▲ ▽

a: Examples provided include modified versions of the relevant examples from previous tests.

▲: Very likely to be used. ▽: Less likely to be used

K: Korean language, SS: Social Studies, M: Mathematics, S: Sciences, E: English

Technical

Feature

Level of

InteractivityDescription

Subject-Specific Utility

K SS M S E

Simulation High

∙ Identifying the relationship between variables by

manipulating tools or variables or predicting the

result by examining the function and significance

of the tool

▽ ▲ ▲ ▲ ▽

Tool

Manipulation

Moderate

– High

∙ Measuring an object presented in the test material

by manipulating a tool (protractor, ruler, calculator,

slider, etc.)

▽ ▽ ▲ ▲ ▽

Information

SearchHigh

∙ Searching for information through dictionaries or

the internet▲ ▲ ▽ ▽ ▲

Virtual

DialogueHigh

∙ Communicating with the computer based on a

dialogue scenario algorithm▲ ▽ ▽ ▽ ▲

Hypertext/

HyperlinkModerate

∙ Interacting with nonlinear, noncontinuous information

made up of links between documents, words, and

images

▲ ▲ ▲ ▲ ▲

Tab Low∙ Interacting with various information in different

formats or of different characteristics ▲ ▲ ▲ ▲ ▲


10

○ Main item types for eNAEA were selected in the two aspects of test-taker

response actions and interactional activities, with consideration of the

efficiency and validity of the assessment tool.

― Considering that computer-based assessment systems allow for the utilization

of different technical features in student interactional activities and answer

areas (refer to [Figure I-2]), item types were set according to the two aspects

of test-taker response (refer to <Table I-3>) and interactional activities (refer

to [Table I-4]).

Instructions

Domain of the Interactional Activities

Domain of the Response Actions

[Figure I-2] Example of eNAEA item structure

Technical

Feature

Level of

InteractivityDescription

Subject-Specific Utility

K SS M S E

Pop-Up Low

∙ Interacting with information by placing the cursor

over events, which generate temporary windows

that disappear after the pointer is moved

▲ ▲ ▽ ▽ ▲

Multimedia Low∙ Presenting or searching data in the form of

animation, audio, or video▲ ▲ ▲ ▲ ▲

▲: Very likely to be used. ▽: Less likely to be used.

K: Korean language, SS: Social Studies, M: Mathematics, S: Sciences, E: English


11

<Table I-3> eNAEA item types in the aspect of response actions

Category eNAEA Item Type Technical Feature Notes

Selected

Response

1Multiple

Choice

Radio Button,

Check Box

∙ Includes the five-response multiple choice

items from the conventional pencil-and-paper

assessment

∙ Includes other multiple choice items

2Expanded

Choice

Hot Spot,

Pull-Down

∙ Can largely replace conventional multiple choice

item

∙ Removes the unnecessary textualization process

in the formulation of response

∙ Includes a pull-down feature, where the list of

choices is provided in the material itself

∙ Includes multiple choices in the “hot spot”

function

3Connecting

Tasks

Matching,

Drag and Drop

∙ Can largely replace conventional multiple choice

items

∙ Removes unnecessary textualization process in

the formulation of response

∙ Includes items involving classification according

to specific criteria

∙ Includes multiple choices in the matching

function

4 Ordering Drag and Drop

∙ Organizing the order of number, words, phrases,

sentences, or images, via clicking (selecting) or

using the drag and drop function

Constructed

Response

5 Short Answer

Table Completion,

Short Answer

Input, Math

Symbols Input

∙ When simple keyboard input is use, identical to

the conventional constructed response items

from the pencil-and-paper assessment

6Extended

Writing

Extended Writing

Input, Math

Symbols Input

∙ When simple keyboard input is use, identical to

the conventional constructed response items

from the pencil-and-paper assessment

∙ Includes writing using document editing functions

7Correction

Tasks

Hot Spot + Short

Answer/Extended

Writing/Math

Symbols Input

∙ Choosing and correcting incorrect contents

8Completing a

Graph/Figure

Graphing and

Drawing,

Drag and Drop

∙ Includes drawing or moving graphs

∙ Includes free drawing

∙ Includes the completion of incomplete items

(e.g. chemical structure models of sciences test


12

<Table I-4> eNAEA item types in the aspect of interactional activities

eNAEA Item Type Technical Feature Notes

1Simple

Presentation

∙ Similar to the conventional pencil-and-paper item, static

screen, 2D plane

2Utilizing

Information

Information Pop-Up,

Information Search,

Tab,

Hypertext/Hyperlink

∙ Overcomes limitations in space allocation and utility in

conventional pencil-and-paper assessment

∙ When additional description or word information is required

∙ Information search does not refer to open search, but rather

to limited search using closed archives

3Media

Inclusion

Animation, Audio,

Video∙ Presents problem situation using multimedia

4

Tool

Manipulation

/Simulations

Protractor, Ruler,

Calculator, Slider,

Timelapse, Etc.

∙ Using simulated tool and variable manipulation to identify the

relationship between variables (e.g. drawing a diagram using

a protractor and ruler, calculation simulation, virtual experiment)

5 Dialogue Virtual Dialogue

∙ Creates the algorithm of virtual dialogue scenarios that

resemble communications in the real world. (e.g. virtual

messenger)

― [Figure I-3] is an example of an “extended choice” item type using a “hot

spot,” while [Figure I-4] provides an example of “utilizing information,”

item type using a “tab”.

Hot spot – Area selection

*Modified from Item 2 of the 2019 NAEA, middle school Social Studies

[Figure I-3] Example of a “simple presentation” – “extended choice” in Social Studies


13

Tab

[Figure I-4] Example of “utilizing information” - “extended writing” in Korean

◈ Development of eNAEA item type application plan and items

○ The study set the test-taker’s response actions and interactional activities as

the two axes and then examined and finalized the item types for eNAEA and

developed an application plan for the test items accounting for the

subject-specific characteristics (refer to <Table I-5>). The basis and the

items of consideration for this plan are outlined below.

<Table I-5> Application plan per eNAEA item type

Interactional

Activities

Test-Taker

Response

1 2 3 4 5

Simple

Presentation

Utilizing

InformationMedia Inclusion

Tool

Manipulation/

Simulations a

Dialogue a

1 Multiple Choice

2 Expanded Choice

3 Connecting Tasks

4 Ordering

5 Short Answer

6 Extended writing


14

○ The current assessment framework of each NAEA subject area was reviewed

for their possible inclusion in the computer-based assessment, with a

special focus on the improvement of the competency-based assessment in

the transition to a computer-based assessment.

― “Competencies” set in the test frameworks of Korean, Social Science, and

Sciences subjects were confirmed to be applicable to the computer-based

assessment through an expert committee.

― In Mathematics, “Communication” was newly added to the “Competencies”

set of the test framework, which initially included “calculation and

understanding,” “inference,” “problem-solving,” and “information process.”

― In English, the “Competencies” set of the test framework, which initially

consisted of “English Communication Competency,” was extended by adding

“Community Competency.”

○ The item development process for the eNAEA pilot test (refer to [Figure I-5])

was used to develop the sample items for each category, followed by

detailed discussions on the subject-specific achievement standards, subject

competencies, and implications for item development.

Interactional

Activities

Test-Taker

Response

1 2 3 4 5

Simple

Presentation

Utilizing

InformationMedia Inclusion

Tool

Manipulation/

Simulations a

Dialogue a

7 Correction Tasks

8Completing a

Graph/Figure a

* Short-term (by 2022), Midterm (within 2-3 years after implementation)

** Having two periods in a single cell denotes that there are differences in the introduction and

application within each type.

*** This is based on an average of application periods for the five subjects of eNAEA. There may be

exceptions.

a The item types that can be more affected by the level of technical development.


15

3. Analyze the applicability of

technical features to the

2020 NAEA items

2. Develop analytical

framework for eNAEA

items

1. Analyze existing NAEA

items, achievement

standards, and

competency

5. Establish details in the

methods of transition for

existing items and

development process for

new ones

4. Select items from 2020

NAEA to be technically

modified for the pilot

8. Confirm and upload

the item

7. Analyze usability 6. Design methods of

realizing technical

features and assessment

items

Total

Subject-specific / Total Subject-specific / Total Subject-specific / Total

Subject-specific Subject-specific

Total

Subject-specific

[Figure I-5] Item development process for the eNAEA pilot test

◈ eNAEA item development process

○ The study outlines a new test item development process for the eNAEA

based on the analysis of the items of improvement required from the

conventional pencil-and-paper system.

― Compared to the paper-and-pencil version of NAEA, eNAEA requires

additional stages in the test development process: storing items to the item

bank and extracting item from the item bank for test delivery. The study

outlined a new test item development process for the eNAEA, which is

customized to the digital characteristics of the test system, as well as to

the item types utilizing some degree of technology and multi-media

― After analyzing the relative advantages and disadvantages of possible item

development methods, such as “camp-types” and “competition-types,” the

study concluded that both camp-type and competition-type item

development methods should be employed during the initial short-term

stages of eNAEA implementation until the new test becomes stabilized,

after which the ratio of the two methods could be revisited.


16

4. Policy Recommendations

○ This study presents its policy recommendations as in [Figure I-6], with the

ultimate goal of utilizing the eNAEA system in a proactive manner to provide

a student- and school-centered student achievement assessment service that

can support innovation in school education and student assessment and

ensure the continuous development of the eNAEA system.

○ Policy suggestions are made from the two points of “supporting the innovation

of school education and student assessment by employing the eNAEA

system” and “supporting the improvement of the eNAEA system.”

― The first point of “supporting the innovation of school education and student

assessment by employing the eNAEA system” includes recommendation to

“provide a school- and student-centered innovative student achievement

assessment service by using the eNAEA system,” “develop and provide a

school-based digital assessment platform (e-assessment platform),” and

“strengthen teachers’ assessment skills (e-assessment expertise) according

to the change of assessment environments.”

― Policy recommendations in the second point of “supporting the improvement

of the eNAEA system” includes the need to “enhance the eNAEA test

development and administration system to ensure test validity and user

experience,” “build and provide a proper infrastructure for the stable

administration of eNAEA,” and “support foundational and in-depth

research to improve the test validity of eNAEA and strengthen the

measurement of student ability and performance.”


17

Supporting the innovation of school education and student growth through innovations in assessment

Provide a regular-base diagnostic evaluation service of academic achievement by

integrating the NAEA database and eNAEA system.

Expand the availability of the eNAEA diagnostic evaluation service to strengthen support

toward achievement assessment in the field.

Improve upon the diversity and immediacy of eNAEA assessment results to provide

personalized information to learners.

Carry out a need analysis for the development of the e-assessment platform.

Develop and distribute the e-assessment platform to schools.

Develop teacher training and certification programs to strengthen teacher expertise in

e-assessment.

Utilize teacher learning communities to advertise item types and encourage their use.

Support the enhancement of eNAEA test development tools to ensure test validity.

Support user interface enhancement to ensure user accessibility and experience.

Provide improvement in test administration environment of eNAEA to a school unit.

Provide administrative and financial support to establish tablet-based assessment

systems in line with changes in computer use.

Establish supportive measures toward the needs of special education students in

computer-based assessment.

Conduct research on the effects of the transition to computer-based assessment on

student achievement.

Conduct research on how to examine and use performance data of eNAEA to improve

test validity.

Provide a school- and student-centered innovative student achievement assessment service using the eNAEA system

Develop and provide a school-based digital assessment platform (e-assessment platform)

Strengthen teachers’ assessment skills (e-assessment expertise) according to the change of assessment environments

Enhance the eNAEA test development and administration system to ensure test validity and user experience

Build and provide a proper infrastructure for the stable administration of eNAEA

Support foundational and in-depth research to improve the test validity of eNAEA and strengthen the measurement of student ability and performance

Recommendation

Recommendation

Recommendation

Recommendation

Recommendation

Recommendation

Supporting the innovation of school education and student assessment by employing the eNAEA system

Supporting the improvement of the eNAEA system

Strategy

1

Strategy

2

Strategy

3

Strategy

1

Strategy

2

Strategy

1

Strategy

2

Strategy

1

Strategy

2

Strategy

1

Strategy

2

Strategy

1

Strategy

2

Strategy

3

[Figure I-6] Policy recommendations and strategies


18

References

Bridgeman, B. (2009). Experiences from large-scale computer-based testing in

the USA. In F. Scheuermann, & J. Björnsson (Eds.), The Transition to

Computer-Based Assessment (pp. 39-44). Luxembourg: Office for

Official Publications of the European Communities.

Bryant, W. (2017). Developing a strategy for using technology-enhanced items

in large-scale standardized tests. Practical Assessment, Research, and

Evaluation, 22(1), 1-10.

de Klerk, S. (2012). An overview of innovative computer-based testing. In T.

J. H. M. Eggen, & B. P. Veldkamp (Eds.), Psychometrics in Practice at

RCEC (pp. 137-150). Retrieved May 27, 2020, from https://ris.

utwente.nl/ws/portalfiles/portal/5579570/Chapter_12.pdf.

Drasgow, F., & Mattern, K. (2006). New tests and new items: Opportunities and

issues. In D. Bartram, & R. K. Hambleton (Eds.), Computer-Based Testing

and the Internet: Issues and Advances (pp. 59-75). England: John Wiley

& Sons Ltd.

Ministry of Education (March 28, 2019). No child gets left behind in basic

education. Ministry of Education Press Release.

Reich, K., & Petter, C. (2009). eInclusion, eAccessibility and design for all issues

in the context of European computer-based assessment. In F. Scheuermann,

& J. Björnsson (Eds.), The Transition to Computer-Based Assessment (pp.

68-74). Luxembourg: Office for Official Publications of the European

Communities.

Sireci, S. G., & Zenisky, A. L .(2006). Innovative item formats in computer-

based testing: In pursuit of improved construct representation. In S. M.

Downing, & T. M. Haladyna (Eds.), Handbook of Test Development (pp.

329–347). Mahwah, NJ: Lawrence Erlbaum Associates Publishers.

Strain-Seymour, E., Way, W. D., & Dolan, R. P. (2009). Strategies and Processes for

Developing Innovative Items in Large-Scale Assessments. Pearson Education,


19

Inc. Retrieved March 29, 2020, from https://images.pearsonassessments.

com/images/tmrs/StrategiesandProcesses forDevelopingInnovativeItems.pdf.

van Lent, G. (2009). Risks and benefits of CBT versus PBT in high-stakes testing:

Introducing key concerns and decision making aspects for educational

authorities. In F. Scheuermann, & J. Björnsson (Eds.), The Transition to

Computer-Based Assessment (pp. 76-84). Luxembourg: Office for Official

Publications of the European Communities.

Date post:	21-Mar-2022
Category:	Documents
Upload:	others
View:	2 times
Download:	0 times

Introducing the Computer-Based Test for the National ...

Documents