THE ‘UNDERSTANDING LIBRARY IMPACTS’ PROTOCOL:DEMONSTRATING ACADEMIC LIBRARY CONTRIBUTIONS TOSTUDENT LEARNING OUTCOMES IN THE AGE OF ACCOUNTABILITY
Derek Rodriguez Doctoral Candidate, School of Information and Library Science
The University of North Carolina at Chapel HillProgram Officer, Triangle Research Libraries Network
9th Northumbria International Conference on Performance Measurement in Libraries and Information Services, August 23, 2011, York, England
Undergraduate education in the age of accountability
Concerns
• Access and cost• Attainment• Competencies of
graduates
Metrics
• Retention• Graduation rates• Student learning
outcomes
Student learning outcomes• Locally defined learning outcomes
• General education and the academic major
• Broad abilities and cross-discipline skills• e.g. Critical thinking, inquiry and analysis, information
literacy, quantitative literacy, written communication• VALUE rubrics (Rhodes, 2010)
• Discipline specific knowledge and skills• By academic major, e.g. History, Chemistry, Nursing,
etc.• Tuning projects (Gonzalez & Wagenaar, 2005; ICHE,
2010)- Rhodes, T., ed. 2010. Assessing Outcomes and Improving Achievement: Tips and Tools for Using Rubrics. Washington, DC: Association of American Colleges and Universities.- González, J. and Wagenaar, R. (eds.) (2005). Tuning Educational Structures in Europe II. Bilbao, ES: University of Deusto.- Indiana Commission for Higher Education. Tuning USA Final Report: The 2009 Indiana Pilot, 2010.
The challenge for libraries
Library use
Student effort
Student Learning Assessment
Program reviews
External frameworks
?
- Connecting library use to student learning outcomes
VALUE & Tuning
‘Understanding Library Impacts’- Explores library use as a component of student effort
Library use
LearningActivities
Student Learning Assessment
Program reviews
Student effort
External frameworks
VALUE & Tuning
A focus for library assessment
Capstoneexperience
General education
AcademicmajorEx
pect
atio
ns Student effort
Design for pilot study (spring 2011)
• Population: undergraduate history majors enrolled at 2 sites, a liberal arts college and a liberal arts university in the U.S. (N=74)
• Project: 20 – 40 page research paper using evidence from primary sources
• Step 1: Constructed a learning activities crosswalk using syllabi and rubrics for research papers
• Step 2: Students completed a web-based critical incident survey after turning in papers
• Step 3: Reports were distributed to study sites via secure, database-driven web-sites
Understanding Library Impacts- Explores library use as a component of student effort
Library use
LearningActivities
Student Learning Assessment
Program reviews
Critical incident survey
Learning activities
crosswalk External frameworks
‘Capstone’project
VALUE & Tuning
Activities
Getting oriented
Choosing a topic
Developing a thesis
Gathering evidence
Finding other sources
Creating a bibliography
Writing
Discipline-specific skills- Locates secondary and primary sources- Distinguishes among types of sources
Evidence and analysis- Evaluates and interpret primary sources- Uses secondary sources to provide context- Demonstrates an understanding of the methods
of history
Thesis and argument- Develops original thesis statement- Advances argument in support of thesis using
evidence from primary sources
Writing and Citing- Communicates argument in a coherent, well-
written paper- Follows disciplinary style and citation standards
Capstone-related outcomes
Learning activities crosswalk
Critical incident survey- Developed and refined in qualitative studies (2006, 2007)
Library use
- Electronicresources
- Traditional resources
- Services
- Facilities / Equipment
Top-ranked e-resource
Top-ranked traditional resource
Top-ranked service
Top-ranked facility
Learning activities
Helps and problems
Open ended
questions and probes
Demographics and affect
Academic challenge
Local questions
Deliverables and outcomes
manuscripts or archival materialsreference books
library instructiondigitized primary sources
electronic booksnon-library web-site
interlibrary loancomputers
indexes or databasesreference
internet search enginesstudy space
printerselectronic journals
bookslibrary catalog
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
41%
63%
68%
73%
78%
100%
Most common library uses, n=41 (55%)
gettin
g orie
nted
choo
sing a
topic
deve
loping
a the
sis
gathe
ring e
viden
ce
findin
g othe
r sou
rces
creati
ng a
biblio
graph
y
writing
the p
aper
0%
20%
40%
60%
80%
100%
51%
93%
83%
56%
‘Top ranked’ use types by activity (n=41)
Electronic resourcesTraditional resourcesServicesFacilities/Equipment
% o
f res
pond
ents
Outcomes related to the activity ‘Gathering evidence’
- Locates secondary and primary sources
- Distinguishes among types of sources - Evaluates and interprets primary sources
- Advances argument in support of thesis using evidence from primary sources
digitized primary sourceselectronic books
indexes or databases electronic journals
research guidenon-library web sites
library catalogbooks
manuscripts or archival materials
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
‘Top ranked’ resource use while 'gathering evidence'
Site ASite B
% of respondents
Outcomes related to the activity ‘Writing’
- Uses secondary sources to provide context
- Advances argument in support of thesis using evidence from primary sources
computers
group study room
printers
study carrel / workspace
library instruction
research consultation
interlibrary loan
0% 5% 10% 15% 20% 25% 30% 35% 40% 45% 50%
‘Top-ranked’ services and facilities used during 'writing'
Site ASite B
% of respondents
- Communicates argument in a coherent, well-written paper
- Follows disciplinary style and citation standards
Factors of use (helps and problems)
• Derived from qualitative studies and LIS literature• Gathered for ‘ranked uses’ only• Categorized for analysis
• Example category: ‘Help finding information’• “I learned about information sources for my project” (help)• “I learned new skills” (help)• “helped me when I got stuck” (help)• “the assistance I received wasn't helpful” (problem)• “it was difficult to find someone to help me” (problem)
Gaining independence
Issue with assignment
Anxiety
Access to tools
Affect of staff
Help finding information
Ease of use
Availability of space
Convenience
Access to information
0% 10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Proportion of students reporting factors of use during capstone project, by theme, n=41
HelpsProblems
% of respondents
Them
e
“Overwhelmed by various themes and layers of my research” (A-2)“Putting all of my information together into a cohesive paper” (A-5)“Time management” (A-8)“Several of the books I needed were being rebound…” (B-3)“Uncertainty about where to start” (B-10)“I was worried about finding primary source material” (B-15)“At a somewhat early point in the project, I realized that the scope of my project was much too wide …” (B-14)
47%
6%
29%
18%AssignmentEvaluating in-formationFinding in-formationTime manage-ment
36%
14%
45%
5%
Challenges faced during the project
Site A, n=17 Site B, n=22
Most important library uses, by type of use, n=37
“… those databases, JSTOR specifically, really helped me.” (B-4)“Library space to work on my thesis both writing and research” (B-2)“being able to get books from off campus” (B-3)
10%3%
14%
3%17%
52%
StruggleChange topicGet bySeek helpPurchase materialsVisit other libraries
35%
19%
8%
38%E-Resources
Traditional resources
Facility
Service
Alternate activity, if first choice hadn’t been available, n=29
Understanding Library Impacts protocol- Contributions and implications
Library use
LearningActivities
Student Learning Assessment
Program reviews
External frameworks
- Credible connections between library use and expectations for student learning- Generates rich data useful for advocacy and improvement - Places library in important campus conversations about student learning
‘Capstone’project
VALUE & Tuning
Next stepsRefining instruments; conducting reliability and validity testingExploring “patterns and predictions”Continue evaluation with history majors at 4 more sites (fall 2011)
Future workEvaluate protocol with other disciplines and settingsExplore integration with assessment systems and analytics projects
Questions?
Contact: [email protected]
Crosswalk: Broad abilities and discipline-specific outcomes
• VALUE Rubrics (Rhodes, 2010)• Critical thinking• Inquiry and analysis• Information literacy• Written Communication
• Tuning Outcomes for History (ICHE, 2010)• Historical knowledge• Thinking and analytical skills• Communication skills• Personal motivation
Rhodes, T., ed. 2010. Assessing Outcomes and Improving Achievement: Tips and Tools for Using Rubrics. Washington, DC: Association of American Colleges and Universities.Indiana Commission for Higher Education. Tuning USA Final Report: The 2009 Indiana Pilot, 2010.
Related capstone outcomes
- Locates secondary and primary sources
- Distinguishes among types of sources - Evaluates and interprets primary sources
- Advances argument in support of thesis using evidence from primary sources
digitized primary sourceselectronic books
indexes or databases electronic journals
research guidenon-library web sites
library catalogbooks
manuscripts or archival materials
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
Top ranked resource use while 'gathering ev-idence'
Site ASite B
% of students
Outcomes related to ‘gathering evidence’
Activity
Gathering evidence
Tuning Outcome
Formulate and test plausible historical hypotheses and
marshal an argument.
Project outcome
Advances argument in
support of thesis using evidence from primary
sources
VALUE Inquiry and Analysis
Analysis - Organizes and synthesizes
evidence to reveal insightful patterns,
differences, or similarities.
Site ASite B