Research: Science and Education

Item Design Considerations for Computer-Based Testing of Student Learning in Chemistry

Craig W. Bowen Deportment of Chemistry and Biochemistry, University of Southern Mississippi, Box 5043, Hattiesburg, MS 39406-5043

Across the nation there are many efforts going on to reform the general chemisrry curriculum. A great deal of discussion has occurred concerning general chemisrry contem and pedagogy (see selections in New Directions for General Chemistry, ref J). While litde about assessing learning has accompanied these discussions, a workshop sponsored by the National Science Foundation included a component on assess­ment of learning. One recommendation in the execurive summary addressed chis issue by scaring (2):

The methods we use for assessing our students and our teaching must change so that they no longer focus on the lowest levels of learning and so that they provide us with the insight into our methods and our tools that we need to drive change.

The purpose of this paper is ro consider how computers might help to broaden assessment pracrices by examining the types of test items chat might be used in com purer envi­ronments-both for originally developed items and by "repurposing" exisring media. Ir examines ideas from cogni­tive psychology and research on problem solving in chemis­try that can be used co guide irem design, and chen briefly reviews how compurers have been used for assessing student learning in the past. T he paper should be ofimeresr to people developing computer-based insrrucrional materials in chem­istry because ir can provide guidance on item writing for com­puter environmentS. lc should also be of interesr ro chemisrry instructors because it can help rhem ro chink abour how to evaluare com purer-based materials for facilitating and assessing studem learning. Because its focus is on designing com purer­based test irems (rather than test administrarion), the paper will leave discussion of security issues associated wirh com­puter-based resring to ocher papers.

Ideas from Cognitive Psychology and Problem Solving Research in Chemistry

It is recognized that chemisrry is a multifaceted science requiring complex thinking. Cognirive psychology offers a way for considering how thinking takes place (3-4). Repre­sentarions are models about the world that people create from their existing knowledge while solving problems. D ifferent kinds of representations (e.g., external/internal or verb~picrorial) have different advantages and disadvantages for problem solving. Problem solving in chemistry involves represeming phenomena in ar least three ways: macroscopically, symbolically, and at the particulate level (5-8). These representational approaches are summarized as follows.

Macroscopic Representations. Models of the world that are based on knowledge and operations involving an un­derstanding of observable chemical phenomena. Symbolic Representations. Models of the world that are based on knowledge and operations involving descrip­tions or explanations of chemical phenomena that have been translated inro a different symbolic form (e.g., math­ematical or verbal).

Particulate Representations. Models of the world that are based on knowledge and operations grounded in imagining what atoms and molecules do during various chemical and physical changes. As an example, the macroscopic level involves what is

observed in a laboratory situation or through a demonstration. The mental model one builds from noticing that a white milky solid forms when aqueous solutions of sodium chloride and silver nitrate are mixed is a macroscopic representation. The symbolic level is most common ro chemists because it in­volves using symbols (of either a chemical or mathematical narure} ro represent chemical phenomena. For example, the equation below symbolically represents the macroscopic reaction of hydrogen and oxygen reacting to form water.

0 2 + 2H2 ~ 2H20 + heat

The ideal gas law, PV = nRT is another symbolic represema­tion mathematically relating various properties of ideal gases.

,Chemists also think about what atoms and molecules do when undergoing chemical or physical changes. When water boils, for example, chemises imagine molecules moving further apart instead of breaking bonds within a molecule as in Figure I.

These ways of represeming chemical phenomena are important because they allow chemists to solve many rypes of chemical problems. While chemists realize and understand the imP.ortance of these ways of representing phenomena, many studems do not. Herron, in his address to the Royal Society of Chemistry, expressed the importance of represen­tations in problem solving (Herron, J. D. Students' Under­standing of Chemistry: An Issue in Chemical Education; paper presented at the Nyholm Symposium, Royal Society of Chemisrry, London, 1983}:

What the experienced chemist writes on paper and what the novice writes on paper may appear to be similar, bur examination of problem-solving protocols suggests that the thought processes may be very different. Whereas the expert is using symbols ro represent physical events that he imagines ro be raking place in accordance with general laws of nature, the student is using symbols to represent symbols which he then manipulates according to memo­rized rules which have no connection with physical reality.

Given char these ways of represeming chemical phenom-ena-macroscopically, symbolically, or ar rhe parriculare level-are used by chemists while solving chemical problems,

Figure 1. Representation of what happens during phase chonge from liquid water to steam.

1172 Journal of Chemical Education • Vol. 7 5 No. 9 September 1998 • JChemEd.chem. wisc.edu

ic makes sense co use them as a framework for measuring chemistry learning. Bowen and Bunce recendy reported che development of a new examination published by the ACS Examinations Institute that involves measuring student un­derstanding of macroscopic and particulate knowledge of chemical concepts (9). An important point of this work is that the three ways of representing chemical phenomena can be used to design assessment activities. Paper-and-pencil tests can be developed to measure some of these aspecrs of learning, but other computer-based assessment might be better suited for determining student understanding of chemistry across these representational approaches.

Past Uses of Computers in Assessing Student Learning

Computers have been utilized in testing student learning in chemistry for at lease 30 years. They have been used primar­ily for administrative reasons (e.g:, producing customized tests for makeup exams) rather than for the unique type of items they might include. One of the first reported uses involved computer scoring of student answer sheers or lab results in large classes (I 0, 11). Computers have since found many more roles in the assessment process. For example, Kumar and Helgeson ( 12) explained that computers are now used for the conscruccion, application, and scoring of multiple-choice exams, the development and administration of constructed response exams, and even computer-adaptive testing systems, which are very useful in the assessment area where solution pathway analysis is important. The recent increase in Internet usage has even opened the door for online testing practices (13, 14). These articles and others primarily indicate che computer as a useful management tool (for construCting or administering tests or keeping track of student grades). The types of quiz or test items that are often presented via computer could just as easily appear on paper. However, measurement of student chemistry learning can be much broader today because of che video and animation power available on computers.

Types of Items in Computer-Based Testing

Like paper-and-pencil resrs, computers might be used to

measure student understanding of chemistry in at least two very different ways.

On one hand, computer-based testing can present items co students that are very open-ended. For example, work be­ing done at the ACS Examinations Institute is focusing on de­veloping icems that require high school scudenrs to integrate several areas of chemistry in order to answer scenario-based problems (15) . One scenario item being developed is the Con-sumer Watchdog. A-

Research: Science & Education

The srudenr assumes the role of a laborarory technician in an organization that concerns irself with the chemical composition of consumer products. In this instance, the organization is concerned about the fact that, while soft drink labels specify the total amounr of sugar in the bev­erage, they do not specify which sugars are present. In a series of inreractions with the laboratory director (which keep the studenr on track), the student uses a spectro­photometric technique (the Folin-Wu method, which is based on Beer's law) to find the relative amounts offruc-tose and glucose in the beverages. The student prepares a report of the invescigacion for Watchdog Reports.

Multiple-choice type items represent a second approach to computer-based testing. Our research group is working co develop a test-item database of multiple-choice items that measure student understanding across the multiple represen­tational levels described earlier (16; Bowen, C. W. Conse­qumw of Cognitive Science for Assessing Student Learning in Chemistry, paper presented at the Gordon Research Conference on Innovations in Teaching College Chemistry; Plymouth, NH, July 1996.). Several sample items across the macroscopic, particulate, symbolic, and integrated levels are given below.

Macroscopic Items. Items focus on measuring student understanding of observable chemical phenomena as shown in Figure 2. In these cases students are presenred digitized video of various chemical situations. Questions are asked to measure their ability to interpret macroscopic phenomena. These items would be more difficult to pose in a paper-and­pencil environment-panicularly a dynamic item such as the polar and nonpolar liquids in sample item 2.

Particulate Items. Items focus on measuring student un­derstanding of what atoms and molecules are doing during various chemical and physical changes. With sample items in Figure 3 we are trying to measure srudencs' understanding of what is occurring at the nano-level. Because of the dynamic nature of chemistry, animations in a computer environment are more useful for measuring student learning in this area than are paper-and-pencil approaches. These images may be static in nature (e.g., the crystal structure question), or need animation (e.g., raising the temperature of the gas).

Symbolic Items. Items focus on measuring student understanding of descriptions or explanations of chemical phenomena that have been translated into a different symbolic form (e.g., mathematical or verbal) shown in Figure 4. These types of items are typically found on paper-and-pencil general chemistry tesrs (if multiple-choice items are used). They do not call for heavy video or animation capabilities of computers. However, they should be included because we do expect students to think about chemistry and solve problems across all three levels.

I , .,..._......, ..... ..,.b a W'bi:k,~ ..... '-olf• ........ --w.o .... ..... _ .. .....,.,.......,..., 2. Afll-.W"dwp4-'d:IM~...., ......... .,...,.

0-0 o,,.,.. O K-wo.y.,. o w ..... 0 0sytMWAit

.,.. ...... ~. ~ilpi··-...., ... ,..,.

0 0 0 0

Figure 2 . Three sample macroscopic items. (Available from author.)

JChemEd.chem.wisc.edu • Vol. 75 No. 9 September 1998 • Journal of Chemical Education 1173

Research: Science and Education

t. WNch ~4 ~ iht Mtt tttt-nla* .. a «nbtlft COCIWNCIJ up uocntuMid~Mf~M.....,..?

l. O.tow I• • ,..,., wbicb ~ a picture of .om. •'*IMOe II a •i• iM\anl ia lim.. WN.th ehoiot *taiN-t lht pt»M .. __ , 3·. Tbtatimuiontoiht~~moltooutM.. ~ •••

apmo~lllplnieuiV~IUN. ltf» ~-.MNM4, wblcllolWdloioMt.low . .

o o..

o...-• 0 ._. ....

o o,. .. m-~id

figure 3. Three sample particulate items. (Available from author.)

b.t~-Chtf't<IUilw~oiU..,....,. •• ..... w,.......,._., .,.

. . . .. .. : . ·. : :· .. .. .. .

0 0 0

2 , Which tqt.IUlooo "'"""*a~ WI would abowa ~ibk ...... , O N.O O H,O O N> O UBr o •a

0 N'IO ( .. ) • KN03 (.., _,. H0 • N~

0 l<oO ( .. ) • Ao)O)J (00) -> A~ • """'J O HIO ( .. } • tf.:SO,. (.,.) _ ,. ~4 •tta

0 N>Q O tt,O 0 !OoOH 0 U8< 0 <0 0 !toO ("') • I<H,OH (00) - • NoOK • Ill~

figure 4 . Three sample symbolic items. (Available from author.)

I . ()bMnotwhtA...,.,...IIOf:ht,~lalbt~IIOd» rf.a.btutht~-lttbown. Attt..tnd.,lhtPf'OO"I,tlM• k<:adoft.- tht,.... • ..,...,.,. fOt\ht •ubtwlot .. ~?

l , eo.;., ...... k.,..... ... lbt~ .,...._. , Whk:lt,~Mit~ •tal It -.,.ftl~ In lbt ~

l. ~wtlla..,..,.illt:ht~ol~ola ... ~tobrifhl. Tm:COiftol ...... wNoth poctioaciWpt.M.~1

Figure 5. Three sample integrated items. (Available from author.)

Integrated Items. The items in Figure 5 focus on mea­suring student understanding of descriptions or explanations of chemical phenomena that link at least two forms of rep­resentation. These types of questions are not a different level, but they do require students to make connections across levels. It is important to try to measure this because one aspect of successful problem solving in chemistry involves being able to switch representations (e.g., instead of thinking symboli­cally, when a problem solver gets sruck on a problem it might be useful to think about what is happening on th~ano-scale).

"Repurposing" Existing Media

Developing paper-and-pencil test items is not as labor­and resource-intensive as developing computer-based test items along the lines described above. One of the most labor­intensive aspects of item development involves creating anima­nons or shooting and digitizing quality video of macroscopic­level chemical events. One way to reduce development costs is to use existing media. A number of videodiscs are avail­able from journal of Chemical Education Software (e.g., ChemDemos, ChemDemosii, and the Periodic Table Videodisc) that contain excellent animations and video footage that can

0

• 0

• • • • • • ~

!' •

•

. -· .,.. ~~ ·

be used in computer-based test items. Many of these video­discs (as well as others available from most textbook pub­lishers) are being ported onto COs. This will make repurposing video footage even easier.

Instead of using the video to demonstrate various chemical phenomena, it can be shown and then students can be asked questions about what is occurring. Repurposing video footage for assessing learning has been described elsewhere in this journal and can reduce development costs of computer-based test irems (J 1).

Summary

Chemistry is a science rich in different ways of thinking about the world. Cognitive psychology and research on problem solving in chemistry suggests that understanding in chemistry involves being able to think across macroscopic, symbolic, and particulate levels. Although this can be used as a framework for measuring student learning in various media, it is particularly useful for computer-based environments. Computers offer an environment in which macroscopic- and particulate-level understanding is readily measured because of the video and animation processes that can be used. Developers

1174 Journal of Chemical Education • Vol. 75 No. 9 September 1998 • JChemEd.chem.wisc.edu

should utilize the power offered by computers by asking questions across all three representational levels. Similarly, instructors responsible for selecting instructional software should examine whether the software focuses on enhancing student chemistry learning at symbolic, parriculate and mac­roscopic levels.

Literature Cited

I. N= Di~ctions for Gmn-al Chemistry; Uoyd, B. W, Ed.; Division of C hemical Education, American C hemical Society: Lancaster, PA, 1994.

2. Bodner, G., pand chair; Arena, S.; Bhat, C.; Kozlowski, A.; Kozma, R.; Lagowski, J. J.; Pryde, L.; Spencer, B.; WiUiams, T.; Zumdahl, S. Assessing Instructional Innovation; Improving the Preparation of Chemistry Teachers; Assessing Student Learning. In Innovation and Change in the Chemiltry Ctmiculum; NSF 94- 19; National Science Foundation: Washington, DC, 1994; pp 10-12.

3. Hayes, J. R. The Complete Probkm Solver, 2nd ed.; Lawrence Erlbaum: Hillsdale, NJ, 1989; pp 3-34.

4. Paivio, A. Memal Representations: A Dual Coding Approach; Ox­ford University Press: New York, 1986; pp 53-83.

5. Herron, J. D. The Chemistry Classroom: Formulas for Succmfol

In the Classroom

Teaching; American C hemical Sociery: Washington, DC, 1996. 6. Gabel, D. L.; Bunce, D. M. In Handbook of Research on Science

Teaching and Leaming; Gabel, D. L., Ed.; Macmillan: New York, 1994; pp 301-326.

7. Smith, K. J.; Men, P. A.j. Chem. Educ. 1996, 73, 233- 235. 8. Russell,}. W; Kozma, R. B.; Jones, T.; Wykoff,}.; Marx, N.; Davis,

J.J. Chem. Educ. 1997, 74, 330-334. 9. Bowen, C. W.; Bunce, D. M. Chemical Educator 1997, 2(2);

httplljournals.springer-ny.comlchedr. 10. Smith, S. R.; Schor, R.; Donohue, P. C.j. Chem. Educ.1965,

42, 224. II. Hinckley, C. C.; Lagowski, J. J. j. CIJtm. Educ. 1966, 43, 575. 12. Kumar, D. D.; Helgeson, S. L.j. Sci. Ed. Techno£ 1995,4,29-36. 13. Tissue, B. M.; Earp, R. L.; Yip, C. W; Anderson, M. L.j. Chem.

Educ. 1996, 73, 446. 14. Ager, T.J. Comput.-Baud lmtruct. 1993, 20(2), 52- 57. 15. Eubanks, D.; Eubanks, L.; et al. Computer-Baud Assessment Ob­

jective Tasks; http:lltigerched.ckmson.edulcomboarlcomboatinfo.html (accessed March 1998).

16. Bowen, C. W. A Framework For Computer-Based Assessment of Stu­dent Learning in Chemistry; paper presented at the National Meet­ing of the American Chemical Society; C hicago, IL, August 1995; CHED Abstr. 233.

17. Bowen, C. W.; Phelps, A.].}. Chem. Educ. 1997, 74, 7 15-719.

edited by Resources for Student Assessment ------------------John Alexander

University of Cincinnoti Cincinnoti, OH 45221

#Conceptual Questions" on LeChotelier's Principle

Benjamin P. Huddle Chemistry Department, Roanoke College, Salem, VA 24153

The questions bel~w are designed to assess the student's abilicy to conceptualize chemical equilibrium and to predict the effect of changes made to a system at equilibrium, using LeChatelier's principle, without doing any equilibrium con­stant calculations.

The Problem

The exothermic reaction o(g) ;=!: • (g) was allowed to come to equilibrium, as represented in the box below:

equilibrium system

1. Some • was added to the system at equilibrium. Which box (A-E) best represents the new position of equi­librium? Explain your answer.

2. The temperature of the system at equilibrium was in­creased. Which box (A-E) best represents the new position of equilibrium? Explain your answer.

3 . The pressure of the system at equilibrium was in­creased. Which box (A-E) best represents the new position of equilibrium? Explain your answer.

Acceptable Solutions

1. B is the only correct box. The original equilibrium system consisted of 6 e's and 4 o 's. If some e's are added to this system, some of the e 's will react ("equilibrium will shift to the left") and produce some o 's, until the ratio of e 's co o's is once again the equilibrium ratio. Both boxes B and C have this ratio, but box B has additional e 's and o 's, as required by the problem.

2. E is the only correct box. The effect of increasing tem­perature on an exothermic reaction is to decrease the value of the equilibrium constant ("equilibrium will shift to the left"), decreasing the amount of product and increasing the amount of reactant. In box E the ratio of e's ro o's has decreased from 6:4 (l 0 total) to 5:5 (1 0 rotal). In all other boxes the ratio is equal to or greater than the original ratio.

3. C is the only correct box. Pressure should have no effect on the position of equilibrium, since there are the same number of gas molecules on both sides of the reaction. The original number of e's and o 's should still be present.

JChemEd.chem.wisc.edu • Vol. 75 No. 9 September 1998 • Journal of Chemical Education 1175