Concept Inventories101

Biology Scholars Program

SoTL Institute

July, 2011

William CliffDepartment of Biology

Niagara University

Questions to be Addressed

• What is a concept inventory?

• How have concept inventories been

used in assessment?

• What are some field-tested

inventories available in biology?

What is a concept inventory?

from Wordle.net

What is a concept inventory? • Set of field-tested conceptual

diagnostic questions (CDQs)

• Validated−Importance−Authenticity−Interpretation

• Reliable

What is a conceptual diagnostic question?

• Based on key concepts in a field (as

identified by instructors/experts)

• Formated as a selected response item−Correct answer−Distractors derived from research on

student misconceptions

• Written in plain, ordinary language

http://www.wcer.wisc.edu/archive/cl1/flag/cat/diagnostic/diagnostic1.htm

As seen from your location, when is the Sun directly overhead at NOON (so that no shadows are cast)? A. Every day. B. On the day of the summer solstice. C. On the day of the winter solstice. D. At both of the equinoxes (spring and fall). E. Never from the latitude of your location. Sample item from the Astronomy Diagnostic Test (ADT) version 1 (Zeilik et al., 1998). The correct response is “E”.

Sample Conceptual Diagnostic Question

http://www.wcer.wisc.edu/archive/cl1/flag/cat/diagnostic/diagnostic1.htm

As seen from your location, when is the Sun directly overhead at NOON (so that no shadows are cast)? A. Every day. B. On the day of the summer solstice. C. On the day of the winter solstice. D. At both of the equinoxes (spring and fall). E. Never from the latitude of your location. Sample item from the Astronomy Diagnostic Test (ADT) version 1 (Zeilik et al., 1998). The correct response is “E”.

Sample Conceptual Diagnostic Question

BRIEFLY EXPLAIN/DEFEND YOUR RESPONSE

http://www.wcer.wisc.edu/archive/cl1/flag/cat/diagnostic/diagnostic1.htm

As seen from your location, when is the Sun directly overhead at NOON (so that no shadows are cast)? A. Every day. B. On the day of the summer solstice. C. On the day of the winter solstice. D. At both of the equinoxes (spring and fall). E. Never from the latitude of your location. Sample item from the Astronomy Diagnostic Test (ADT) version 1 (Zeilik et al., 1998). The correct response is “E”.

Sample Conceptual Diagnostic Question

CDQ: Jared, the Subway® man, lost a lot of weight eating a low-calorie diet.

Where did all the fat/mass go?

A. The mass was released as CO2 and H2O.

B. The mass was converted to energy and used up.

C. The mass was converted to ATP molecules.

D. The mass was broken down to amino acids and eliminated from the body.

D’Avanzo, C. Bioscience 58:1079-1085, 2008.

CDQ: When a person loses weight, what happens to the fat in the person’s body? Circle True (T) or False (F) for each option.T F Some the fat is broken down and leaves the person’s body as water and gas.

T F Some the fat is converted into energy.

T F Some of the fat is used up and disappears.

T F Some of the fat is broken down and leaves the person’s body as feces and urine.

http://www.biodqc.org/node/515

Conceptual Diagnostic Questions

CDQ: The selection of antibiotic-resistant, transformed bacteria is based upon a change in the:

A. phenotype of the bacteria

B. genotype of the bacteria

C. phenotype and genotype of the bacteria

D. genotype and physiology of the bacteria

E. genotype and morphology of the bacteria

Marbach-Ad G. et al. J Microb. Biol. Edu. 16:43-50, 2009.

CDQ: Consider a short polar region and a short non-polar region in a long polypeptide chain. When dissolved in water, the polypeptide will most likely fold to form a protein in which:

A. the non-polar region is exposed on its surface and the polar region is interior.

B. the polar region is exposed on its surface and the non-polar region is interior

C. both the non-polar and the polar region are exposed on its surface.

D. both the non-polar and the polar region are interior.

Shi, J. et al. CBE-Life Sciences Education 9:453-461, 2010.

Conceptual Diagnostic Questions

Host Pathogen Interaction Concept Inventory Question 1

Question and Answers General Microbiology (n=68)

Bacterial Genetics (n=25)

Selection of an antibiotic resistant organism is based upon a change in the:

A. phenotype 1 7 B. genotype 38 4 C. both 25 13 D. neither 0 1 e. either 4 0

Marbach-Ad G. et al. J Microb. Biol. Edu. 16:43-50, 2009.

TABLE 3A

Ans. “b”. genotype “When an organism becomes

resistant to antibiotics (when it acquires an antibiotic-

resistant gene that has been inserted as a marker),

the organism’s genotype has been changed.”

Ans. “b”. genotype “This must be a change in the

genotype because having antibiotic resistance will

not necessarily change the look of an organism

(phenotype). It will merely allow it to survive in

situations where the antibiotic is present”

Host Pathogen Interaction Concept Inventory Question 1“Defend Your Response”

Marbach-Ad G. et al. J Microb. Biol. Edu. 16:43-50, 2009.

1. The selection of antibiotic-resistant, transformed bacteria is based upon a change in the:

A. phenotype of the bacteria

B. genotype of the bacteria

C. phenotype and genotype of the bacteria

D. genotype and physiology of the bacteria

E. genotype and morphology of the bacteria

Percentages of correct answers for question 1 on pre- and post-course surveys

Pre or Post

General Micro 2006

Pathogen Micro

General Micro 2007

Bacterial Genetics

Immunol Epidemiol

Pre 25 32 12 35 56 36 Post 30 24 26 44 67 42

Marbach-Ad G. et al. J Microb. Biol. Edu. 16:43-50, 2009.

Host Pathogen Interaction Concept Inventory Question 1(revised)

TABLE 5

How have concept inventories been used in assessment?

• To reveal prior knowledge and misconceptions

that students bring to the class room

• To measure conceptual gains associated with

different methods of instruction

• To enhance professional development of

faculty and curricular reform

Suggestions for Use• Adopt already-developed, field-tested instruments

− must follow the guidelines for its use for the results to be valid and reliable.

− give the assessment as a pre- and post-test.

− give all items in the order presented on the instrument.

• Adopt already-developed test items (CDQs)− cannot directly compare results to those normed from

the complete instrument.− may better match course goals.

• Develop your own test items (CDQs)− match questions closely to course goals. − investigate the research literature before beginning.− build up a bank of well-constructed/researched items.

Zeilik, M. Conceptual Diagnostic Tests. Field-Tested Learning Assessment Guide. www.wcer.wisc.edu/archive/cl1/flag/cat/cat.html

Limitations

• Limited number of biology concept

inventories are currently available.

• Inventories may not match assessment

goals.

• Vocabulary – e.g. use of jargon

• Format – close ended, multiple choice

Some field-tested inventories available in biology

• Introductory Biology (Klymkowsky and Garvin-Doxas, 2008;

Klymkowsky et al. 2010)

• Natural Selection (Anderson et al. 2002)

• Genetics − Non Majors (Bowling et al. 2008)

− Majors (Smith et al. 2008)

• Host-Pathogen Interactions (Marbach-Ad et al. 2009)

• Molecular and Cellular Biology (Shi et al. 2010)

My Work with Conceptual Diagnostic Testing

• Case Based Learning-mediated Misconception

Repair in Respiratory Physiology

• Chemistry Misconceptions Associated with Body

Calcium Homeostasis

• Teleological Thinking in Physiology

• Misconceptions in Renal Physiology

• Core Principles of Undergraduate Physiology− Developing a Physiology Concept Inventory

Physiology of Oxygen Transport by the Blood

Conceptual Diagnostic Question ASa / PO2 Misconception

Michael JA, et al.. Am. J. Physiol. 277 (Adv. Physiol. Educ. 22): S127-S135, 1999.

1st TIER

2nd TIER

Conceptual Diagnostic Question BSa / PO2 Misconception

Prevalence of Respiratory Misconceptions

Misconception

Sa/PO2 O2/CO2 VT/f Met/Vent

% P

opul

atio

n w

ith M

isco

ncep

tion

0

20

40

60

80

100PRETESTPOSTTEST

*

* P<0.05

CBL Remediates Student Misconceptions

Cliff, W.H. Adv. Physiol. Educ. 30: 215-223, 2006

CBL Improves Conceptual Understanding

Cliff, W.H. Adv. Physiol. Educ. 30: 215-223, 2006.

Cliff, W.H. Adv. Physiol. Educ. 30: 215-223, 2006

Misconception Repair By CBLIs Progressive

Calcium Balance in the Body

A beaker contains calcium - Ca+2, phosphate - PO4 –3and calcium phosphate -

Ca3(PO4)2. These substances are at equilibrium in a fixed volume of water.

Ca+2 + PO4 –3 ⇋ Ca3(PO4)2 (solid)

1. Suppose you add CaCl2 to the beaker. All the CaCl2 dissolves in the water and forms Ca2+ and Cl―. After adding the CaCl2, you would predict that the concentration of phosphate will:

A. increase B. decrease C. not change

Explain your answer: 2. Suppose you were to add more Ca3(PO4)2 to the beaker. The concentration of calcium will:

A. increase B. decrease C. not change

Explain your answer:

Conceptual Diagnostic Test about the Chemical Equilibrium of Calcium Salts

Cliff W. Adv. Physiol. Educ. 33:323-328, 2009

Cliff W. Adv. Physiol. Educ. 33:323-328, 2009

CDQ1: Suppose you add CaCl2 to the beaker. All the CaCl2 dissolves in the water and forms Ca2+ and Cl―. After adding the CaCl2, you would predict that the concentration of phosphate will:

A. increase B. decrease (* CORRECT) C. not change

TABLE 1. Multiple Choice Answers to CDQ1

Ans A B* C total % # % # % # #### All 6% 4 52% 34 42% 28 66

Results

CDQ 1Answer C Students failed to apply the common ion effect to the solubility equilibrium established between calcium, phosphate and calcium phosphate. As a result, they did not use Le Châtelier’s principle to correctly predict the change in phosphate concentration that will occur as a result of the addition of calcium to the solution. Implications for Teaching: Students may not be able to correctly apply the common ion effect to determine changes in the equilibrium concentrations involved in physiochemical reactions. Relevant Misconceptions in Physiology: Hypercalcemia would not alter plasma phosphate concentrations. Addition of sodium bicarbonate to the body fluids would not change the concentration of hydrogen ions.

Conclusions

Cliff W. Adv. Physiol. Educ. 33:323-328, 2009

CDQ2: Suppose you were to add more Ca3(PO4)2 to the beaker. The concentration of calcium will:

A. increase B. decrease C. not change (* CORRECT)

TABLE 2. Multiple Choice Answers to CDQ2

Ans A B C* total % # % # % # #### All 86% 57 3% 2 11% 7 66

Results

CDQ2Answer A

Students failed to use the principle that the concentration of a pure solid is constant for a heterogeneous equilibrium. As a result, some students mistakenly applied Le Châtelier’s principle to the situation and predicted that the addition of solid calcium phosphate would cause the reaction to shift to the left and the concentration of calcium to increase. Other students based their answer on notions of chemical dissociation or addition without explicitly considering the chemical equilibrium. Implications for Teaching: Students may mistakenly apply Le Châtelier’s principle to physiological circumstances where it is not appropriate - e.g. the contribution of bone to calcium homeostasis. Relevant Misconception in Physiology: Increased bone mass should cause an increase in the concentration of calcium ions in body fluids.

Conclusions

Teleological Thinking in Physiology

Table 1. Frequency Distribution of Student Responses on 10 Item Conceptual Diagnostic Test

Figure 1. % Teleological Answers compared to Course Grade (n=72)

Cliff, W.H. FASEB J. 23: 632.12, 2009.

Results and Conclusions1. Overall, 39 % of the answers chosen on the ten-item test were

teleological. Percentages ranged from 6% to 76% for individual items

(Table 1). Considerable variance exists in student preference for teleological explanations for the cause of different physiological phenomena.

2. There was no correlation between the total number of teleological

answers chosen by individual students and final course grade (Figure

1).The preference for teleological answers on the conceptual diagnostic test did not appear to interfere with student ability to think or reason mechanistically in other contexts.

3. Structured interviews indicated that some students internally

translate the teleological wording of test answers into mechanistic

processes. Internal restructuring indicates that students may accept teleological formulations as answers without necessarily embracing teleological reasoning as a means of explanation.

Instructors need not hesitate to use teleological formulation in the class

room for its heuristic value, as long as students are able to finish with

satisfactory mechanistic explanations for physiological phenomena.

RDQ4: The plasma concentration of glucose increases above normal and the rate (ml/min) of glomerular filtration remains constant. The concentration of glucose found in the filtrate within Bowman's space will: A. increase B. decrease C. remain constant

Please explain your answer:

Bowman’s Space Glomerulus

[Glu]

Bowman’s Space Glomerulus

[Glu] [ ? ]

ml/min

ml/min

[Glu] [Glu]

Qualitative Conceptual Diagnostic Question on Glomerular Filtration

Cliff, W., M.P. Wenderoth, and J. Michael. FASEB J. 17:A816, 2003.

Qualitative Conceptual Diagnostic Question on Glomerular Filtration

Cliff, W., M.P. Wenderoth, and J. Michael. FASEB J. 17:A816, 2003.

RDQ4: The plasma concentration of glucose increases above normal and the rate (ml/min) of glomerular filtration remains constant. The concentration of glucose found in the filtrate within Bowman's space will:

A. increase (* CORRECT) B. decrease C. remain constant

TABLE 1. Multiple Choice Answers to RDQ4

U – undefined on survey Ans A* B C U total % # % # % # % # #### All 65.6 82 7.2 9 24 30 3.2 4 125 Responses 68% 7% 25%

Summary 68% of respondents predicted filtrate glucose concentration would increase

following increased plasma concentration and no change in GFR.25% predicted no change.7% predicted decrease. 

Many of the students who predicted no change indicated that they believed that the GFR determined the glucose concentration in the filtrate.   Some students who answered the prediction correctly indicated that they thought both plasma concentration and GFR influenced the filtrate glucose concentration. For example: “Greater filtration means greater [ ] in filtrate.” “more glucose will be filtered so its concentration will increase” “one increases above normal and the other remains constant, so we should see an increase” “if there is a higher glucose concentration per mL, and the mL filtered is the same, there will be more glucose coming in per mL anyway.” These answers suggest that a substantial number of students have the misconception that the concentration of glucose in the filtrate is equivalent to the filtered load.

Quantitative Conceptual Diagnostic Question on Glomerular Filtration

Cliff, W., M.P. Wenderoth, and J. Michael. FASEB J. 17:A816, 2003.

RDQ4: After ingesting a meal, the concentration of glucose in the blood increases from 1 mg/ml to 2 mg/ml. The glomerular filtration rate (ml/min) of the kidneys does not change. Under these circumstances, the concentration of glucose found in the filtrate within the Bowman’s space will be:

A. 1 mg/ml B. between 1 and 2 mg/ml (greater than 1 mg/ml but less than 2 mg/ml) C. 2 mg/ml (* CORRECT) D. greater than 2 mg/ml

TABLE 2 Multiple Choice Answers to RDQ4 Ans A B C* D total % # % # % # % # #### All 17.1 25 31.5 46 41.8 61 9.59 14 146

Summary 42 % correctly predicted the filtrate concentration will equal the final plasma concentration.31% predicted an intermediate concentration between the initial and final plasma concentrations.17 % predicted no change in concentration.10% predicted a higher concentration. 

Some of the students who predicted an intermediate glucose concentration reasoned that a combination of increased plasma concentration with no change in the GFR leads to an intermediate filtrate concentration. 

Some students who predicted correctly indicated that both plasma concentration and GFR influenced the filtrate glucose concentration.  “The GFR does not change and so blood concentration won't change as it flows out of the capillary into Bowman's capsule.” “The concentrations will be the same if the rates are the same” As with the qualitative survey, these students express the misconception that the filtrate concentration is equivalent to the filtered load.  

.

Summary cont. .  

A substantial fraction of students who answered incorrectly (41%) would have given a correct answer if asked to predict the qualitative change in glucose concentration. Only in response to a quantitative query did they gave evidence they held a misconception.

CONCLUSIONS After instruction in renal physiology, a sizeable number of students demonstrate they hold the misconception that the concentration of glucose in the filtrate is equivalent to its filtered load.  This observation reinforces the conclusion from investigations of general models in physiology that student misconceptions about “concentration” and “amount” can have a significant impact on their understanding of renal physiology.   

IMPLICATIONS FORTEACHING AND LEARNING

 

 It is important to help students recognize the distinction between “concentration” and “amount” in order for them to avoid developing misconceptions about solute and water transport in the kidney and other areas of physiology. A combination of qualitative and quantitative diagnostic probes can be useful to uncover different facets of student understanding of physiological concepts.

Defining and Assessing the Core Principles for Undergraduate Physiology NSF TUES Grant (2011-2014)

PI: Jenny McFarland (Edmonds Community College)

CO-PIs: William Cliff (Niagara University), Joel Michael (Rush Medical

College), Harold Modell (Physiology Education Research

Consortium), Ann Wright (Canisius College)

RECRUITING FACULTY PARTICIPANTS to HELP DEVELOP PHYSIOLOGY CONCEPT INVENTORY!

More Information

• D’Avanzo, C. Biology Concept Inventories: Overview, Status, and

Next Steps. Bioscience 58:1079-1085, 2008.

• Knight, J. Biology Concept Assessment Tools: Design and Use.

Microbiol. Australia March, 5-8, 2010.

• Libarkin, J. Concept Inventories in Higher Education Science.

National Research Council Promising Practices in Undergraduate

STEM Education Workshop 2, Washington, DC, 13-14 October 2008.

http://www7.nationalacademies.org/bose/Libarkin_CommissionedPap

er.pdf

• Smith, J and Tanner, K. The Problem of Revealing How Students

Think: Concept Inventories and Beyond. CBE-Life Sciences Education

9:1-5, 2010.

• Zeilik, M. Conceptual Diagnostic Tests. Field-Tested Learning

Assessment Guide. www.wcer.wisc.edu/archive/cl1/flag/cat/cat.html