Assessing our Assessment: Failures and Successes

at UT-Austin

AJ Johnson, Michele Ostrow, Meghan Sitar

Library Instruction ServicesUniversity of Texas Libraries

FAILURE 1Didn’t ask for help at the beginning

http://www.flickr.com/photos/inju/246717376/

FAILURE 2Didn’t recognize the difference between faculty and librarian

culture

http://fc04.deviantart.com/fs16/f/2007/160/7/f/WilliamSnape2_by_WilliamSnape.jpg

http://www.hermionegrangerswand.com/

FAILURE 3Didn’t anticipate low response

rates

http://www.houseofzing.com/book/images/ignore.jpg

ww

w.h

ou

seofzin

g.co

m/b

oo

k/imag

es/ign

ore.jp

g

FAILURE 4Didn’t consider how faculty view

librarians

htt

p://

ww

w.f

lickr

.com

/pho

tos/

bibl

iona

/238

9790

54/

htt

p://

ww

w.f

lickr

.com

/pho

tos/

kais

hin/

1660

2380

46/

FAILURE 5 Didn’t hire an army of instruction

librarians and statisticians

"United States Army, The." Online Photograph. Encyclopædia Britannica Online. 22 May 2008  <http://www.britannica.com/eb/art-82331>.

FAILURE 6 Didn’t pre-test

FAILURE 7Still didn’t pre-test

FAILURE 7Still didn’t pre-test

FAILURE 8 Designed our post-test around

faculty characteristics

FAILURE 8 Designed our post-test around

faculty characteristics

FAILURE 9Didn’t match pre-tests to post-

tests

FAILURE 9Didn’t match pre-tests to post-

tests

FAILURE 10 Project design didn’t reflect flexibility of our instruction

programs

FAILURE 10 Project design didn’t reflect flexibility of our instruction

programs

FAILURE 11 Didn’t define a control group

“No controle (in control)” by renatotarga on Flickr, http://www.flickr.com/photos/rtarga/223776150/

FAILURE 12 Thought writing questions would

be the easy part

FAILURE 13Didn’t base questions on learning

outcomes

FAILURE 13Didn’t base questions on learning

outcomesOLD: Question - You found an article in a database but it isn’t full-text. How do you find the full text of the article somewhere else? (Circle all that apply)1.Search the Library Catalog. Use the article title as your search terms.2.Search the Library Catalog. Use the title of the journal, magazine or newspaper as your search terms.3.Follow the “FIND IT UT” link

FAILURE 13Didn’t base questions on learning

outcomesNEW: Learning Outcome - Students will be able to determine

where scholarly material will be found. Question - You are doing research on border security along the

U.S. Mexico border. Choose the BEST place to find a research study by a political science professor.Library article databasewww.google.comPublished encyclopediaDon’t know

FAILURE 14 Forced to use multiple choice tool to measure critical thinking skills

FAILURE 14 Forced to use multiple choice tool to measure critical thinking skills

Old:When evaluating web sites to see if you should use them in your research, which factors should you consider? (Choose all that apply)1.Site was on the first page of results in Google2.Author expertise3.Page design4.Bias5.Don’t know

FAILURE 14 Forced to use multiple choice tool to measure critical thinking skills

Newer:You find a web page about media mergers and know nothing about the author. What would be your best clue as to whether the page was a reliable source to use for your paper?1.date the page was last updated2.the URL (http://www...)3.the quality of the page’s graphic design4.the advertisements on the page5.don’t know

FAILURE 14 Forced to use multiple choice tool to measure critical thinking skills

Newest:You are doing research on the effect of illegal immigration on border violence. You find a website from an organization called The Illegal Immigration Research Coalition, which includes statistics on incidents of border violence. List at least three criteria you would use to evaluate this site for bias and credibility.

FAILURE 15Created questions with more than

one right answer1. Google searches will find information from library databases

True False Don’t know 2. Choose the BEST place to find a recent report from a media

watchdog organizationlibrary article databasewww.google.compublished encyclopediadon't know

FAILURE 16Created trick questions

You are doing Internet research about border security. Choose the domain of a website most likely to contain a U.S. presidential candidate's stance on immigration issues

.org

.edu

.gov

.comdon't know

FAILURE 17Didn’t define success

before we started

http://www.easywebcreator.com/applications/WebBlogManager/inc_webblogmanager.asp?CategoryID=227&ItemID=1061&action=refer

FAILURE 18Didn’t identify all the variables

http://www.flickr.com/photos/jdawg/172131301/

FAILURE 19Can’t really use the data

FAILURE 19Can’t really use the data

FAILURE 19 Can’t really use the data

Presentation and Questions Created by Participants available

at:

http://www.lib.utexas.edu/services/instruction/ostrow.html