Ada Emmett and Judith EmdeUniversity of Kansas Libraries

Assessing Chemical Information Literacy Skills using the ACRL Standards

ALA 2006 Annual Meeting, New OrleansSTS Research Forum

Sunday June 25th

2

Overview

Chemistry bibliography course

Assessment tool development to measure learning outcomes

Assessment results and observations

ACRL standards: advantages

3

Bibliography of Chemistry : CHEM 720

One hour credit offered for graduate students

Major chemistry and biomedical research tools

Grading: satisfactory/unsatisfactory

4

Questions

What should students learn?

Does teaching produce the desired learning outcomes?

How can we assess student learning?

5

Used to develop:

Learning outcomes

Teaching activities

Assessment tool

An “information literate” student

ACRL’s Information Literacy Competency Standards for Higher Education

6

ACRL’s Information Literacy Competency Standards for Higher Education

Alternatives to ACRL literacy standardsACSSTS

7

ACRL’s Information Literacy Competency Standards for Higher Education

1. Determine the extent of information needed 2. Access the needed information effectively and

efficiently 3. Evaluate information and its sources critically and

incorporate selected information into his or her knowledge base and value system.

4. Use information effectively to accomplish a specific purpose

5. Understand the economic, legal, and social issues surrounding the use of information, and access and use information ethically and legally

http://www.ala.org/ala/acrl/acrlstandards/informationliteracycompetency.htm

8

Year One: 2004

MethodologyDeveloped assessment tool Conducted pre- and post-course

assessment interviewsUsed control groupQuantified data

9

Year One: 2004

Results

Averagescore

Pre-Test Score

Post-TestScore

Possible Percent Improvement

PointsImproved

Studentsin Class

N=26

45 65 72 44% 20

“Control” N=4

48 56 72 17% 8

10

Year One: 2004

45 48

6556

7272

0

10

20

30

40

50

60

70

80

Students in Class Control

Pre-Pre-

Possible Possible

Post- Post-

11

Year Two: 2005

Assessment tool: redesign and refinement

SPSS Data Entry Builder to enter data One-on-one interviewsPre- and post-test given

students 16small control group 5

12

Year Two: 2005

Results

Averagescore

Pre-Test Score

Post- TestScore

Possible Percent Improvement

PointsImproved

Studentsin Class

N=16

61 99 120 62% 38

“Control” N=5

73 85 120 16% 12

13

Year Two: 2005

ResultsAverage 2005 Scores in Pre- and Post- and Total

Possible Points by Group

6173

9985

120 120

0

20

40

60

80

100

120

140

Students In Class Control

Pre- Post- Possible

14

Year Three: 2006

Used “backward design”– starting with ACRL learning outcomes

Develop assessment questionsDesign course lectures/assignmentsCreate tool – web-based questionnaire

Run as pre- and post-testNo controls

15

Year Three: 2006

Results

Points Improved: 27Percent Improved: 57%

Average scores in Pre- Post- and Total Possible Points

47.5

74.5 80.0

-

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

1

Pre-

Post-Possible

16

Comparison by Assessment Questions

Comparison of Pre- and Post-survey Average Scores by Question

0.00

1.00

2.00

3.00

4.00

5.00

3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28

Question Number

Ave

rag

e P

oin

ts

Per

Qu

esti

on

Total Possible Per Item Scores Before Instruction Scores After Instruction

17

2004-2006 Improvement

2004: 44% improvement (“control” 16%) 2005: 62% improvement (“control” 17%) 2006: 57% improvement

44%

57%62%

0%

10%

20%

30%

40%

50%

60%

70%

2004 2005 2006

18

Assessment questions grouped by standards

Standard #2:The information literate student accesses needed information

effectively and efficiently.

Performance indicator #5:The information literate student extracts, records, and manages

the information and its sources. Outcome “e”:Uses various technologies to manage the information selected

and organized.

Assessment questions: Select among the following tool(s) that manage references or

citations and therefore assist in writing research papers.

Describe two features of a software program that assist in managing citations.

19

Assessment questions grouped by standards

Average Scores on Survey Questions as Associated with Information Literacy Standards

0.00

1.00

2.00

3.00

4.00

5.00

6.00

1.2 1.3 2.1 2.2 2.4 2.5 3.2 3.7 5.1 5.2

Information Literacy Standards and % of Improvement from Pre- to Post-

Po

ints

Per

Qu

esti

on

Pre Score Post Score Points Possible

28.2% 15.9% 44.8% 43.8% 44.8% 36.1% 26.9% 44.8% 81.3% 6.3%

Average Scores on Survey Questions as Associated with Information Literacy Standards

0.00

1.00

2.00

3.00

4.00

5.00

6.00

1.2 1.3 2.1 2.2 2.4 2.5 3.2 3.7 5.1 5.2

Information Literacy Standards and % of Improvement from Pre- to Post-

Po

ints

Per

Qu

esti

on

Pre Score Post Score Points Possible

28.2% 15.9% 44.8% 43.8% 44.8% 36.1% 26.9% 44.8% 81.3% 6.3%

20

Students’ perceptions

Overall Average Student Scores from 2006 Survey

Score Perception

Score

Perception

-

10

20

30

40

50

60

70

80

Pre-Instruction Post-Instruction

Tota

l Ave

rage

Sco

re

(80

Pos

sibl

e)

21

Observations

Mapping standards is subjectiveConsultation on mapping questions to standards

Some standards such as 3 and 4 are difficult to assess

Strong subject expertise needed (faculty/instructor collaboration)

Mechanism of delivering assessment tool needs improvement

Ongoing review of assessment tools’ strengths/weaknesses

22

Observations

Assessing all learning outcomes difficultKeep it simple to several specific outcomes.

Did the class and the test incorporate most important learning outcomes for that student group?

Faculty/instructor collaboration

23

ACRL standards: advantages

Provides mechanism to assess information literacy skills via learning outcomes (using “backward design”)

Assists in the development of course content through backward design

24

Discussion, Questions and Comments

25

Contact information

Ada Emmett

aemmett@ku.edu

785-864-8831

Judith Emde

jemde@ku.edu

785-864-4931

Supplemental information at http://www.people.ku.edu/~jemde/

26

27

Mapping lectures / exercises to standards

Potential learning outcome of lecture and exercise mapped to standards at beginning of semester

Comparison of lectures/exercises to results of post assessment by grouped standards.

28

Mapping lectures / exercises to standards

Correlation Between Number of Lectures/Exercises by Standard and Associated Student Score Improvement

-

5

10

15

1.2 1.3 2.1 2.2 2.4 2.5 3.2 3.7 5.1 5.2

Information Literacy Standards

Total Lectures and/or Exercises associated with each

0%

100%

Pre to Post % Improve-

ment by Standard

# L-E's by Std % Improvement