Date post: | 15-Apr-2017 |
Category: |
Education |
Upload: | philippine-association-of-academicresearch-librarians |
View: | 879 times |
Download: | 2 times |
V M G
Do you know your
organization's VMG?
Do you know your library’s
VMG?
Do you know the role of your
library in your organization? In
the society?
Do you know your library’s
purpose?
DEMONSTRATING THE LIBRARY’S
IMPACT THROUGH ASSESSMENT AND
EVALUATIONChristine M. Abrigo
De La Salle University Libraries
Managing Today’s Learning Commons: Re-skilling Seminar for information professionals
Philippine Association of Academic/Research Librarians, Inc.-20-22 September 2016, Baguio City
Explain the importance of purpose and evidence
Explain why evaluate Explain the measures for evaluation Explain about impact and impact
evaluation (looking beyond the numbers)
Workshop
OBJ
ECTI
VES
OF
THIS
LEC
TURE
Importance of Purpose and
Evidence
Purpose the reason why something is done or
used (Merriam-Webster, 2015)
something you do to achieve something the intention answers the questions: “Why”, “What for”
and “Who for”? Aim the main and overarching stated
purposes of the library service (Markless & Streatfield, 2013)H
AVIN
G A
PU
RPO
SE
Why is having a purpose important?
Library’s purpose = borne out of your organization’s VMG
Creating a purpose sets the direction to what you want to achieve and how you will achieve it
HAV
ING
A P
URP
OSE
So what do we have in mind?
USERS We want to satisfy their needs.
PROGRAMS/PROJECTS We want to improve our services.
RESOURCES We want to provide the best/top of
the line.
HAV
ING
A P
URP
OSE
So what are we inclined to do?
We come up with new services, programs, projects, build resources (aside from the traditional operational services). Why?
We come up with activities under these programs and services
HAV
ING
A P
URP
OSE
BOTTOMLINE:
Are these working based on your purpose?
Are these effective? Are we creating impact?
HAV
ING
A P
URP
OSE
We need evidence… to show that change has occurred to show effectiveness (that the
programs, services, projects that we do is making a difference)
to prove that what we are doing is meaningful
to demonstrate our value to our parent organization, learners and communityTHE
NEE
D F
OR
EVID
ENCE
Why evaluate?
Assessment Evaluationthe systematic collection of data to monitor the success of a program or course in achieving intended outcomes
the process of determining worth, merit, or value of something; assessment of “goodness”
judging people’s knowledge or skills; getting a baseline of what your stakeholders know and what they need
a judgment about whether the program has met its intended outcomes
Mostly descriptive and quantitative
Mostly qualitative
Why evaluate? Because… it tells you how well the library is performing a library rarely does it (comes up with a
service/program, implements, and leaves it be)
a library focuses on inputs and outputs (no impact)
a library would likely be inclined to calculate efficiency (i.e., making the most out of resources/service to produce a desired result) using traditional performance indicators (e.g., usage stats, entrants, assessment studies, ROI, et al.), and not effectiveness
Measures for evaluation
Levels of analysis
Individual (i.e., user experience)
Service (i.e., user group’s experience)
Organizational (i.e., all library users)
Societal (i.e., impact on local community)
MEA
SURE
S FO
R EV
ALUA
TIO
N
Inputs (resources; What
are needed)
Process (activities; What
to do)
Outputs (products; What the
results are)
Outcomes (effect/turnout/benefits; What the consequences
are)
MEA
SURE
S FO
R EV
ALUA
TIO
NBasic model (Richard Orr, 1973)
MEA
SURE
S FO
R EV
ALUA
TIO
NInputs Process Outputs Outcome
IL training courses
IL reference materials
Program Resource
persons Students
Trainings/Workshops for librarians
Development of IL program materials
modules IL sessions quizzes and
games self-paced IL
learning tools instructional IL
videos and guides
information products (e.g., pathfinders, guides, etc.)
Librarians know what relevant information are for appropriate users
Users become confident in finding authoritative sources
Users become independent learners and practice information ethics
Sample: Information Literacy Program
On impact and impact
evaluation (looking beyond the
numbers)
Let’s review: You identify your VMG Create your library’s purpose aligned to your
organization’s VMG Demonstrate evidence of your value to your
community Come up with initiatives (services, programs), noting
your purpose and aims Evaluate these initiatives
Are you creating impact?ON
IMPA
CT A
ND
IMPA
CT
EVAL
UATI
ON
Impact is... seeing change or difference in what you
do concerned about effectiveness any effect (maybe positive or negative)
of the service on either an individual, group or both
long-term; requires time a series/assessment of outcomes
ON
IMPA
CT A
ND
IMPA
CT
EVAL
UATI
ON
Measuring impactImpact evaluation (outcomes-based evaluation) largely focused on monitoring service
efficiency systematic causation (cause-effect) requires clear and consistent use of
terms
ON
IMPA
CT A
ND
IMPA
CT
EVAL
UATI
ON
Measuring impactImpact evaluation (outcomes-based evaluation)
Seeks to answer:
“How much better off are your patrons (in terms of service) after introducing a service/program?”
“Does the program/service/resource have a different impact on different groups?”
"Did the program/service/resource cause the impact?” “What would have happened if the program/service/resource had
not taken place?”ON
IMPA
CT A
ND
IMPA
CT
EVAL
UATI
ON
MET
HO
DS
FOR
IMPA
CT
EVAL
UATI
ON
QUANTITATIVEmeasuring changes in competences or behavior or to find correlations between library use and a person’s academic or professional success
QUALITATIVEevaluating users’ experiences and opinions
user skills tests before and after a training or the use of a library service
surveys (print, telephone, or online)
performance monitoring/data mining
Interviews
unobtrusive observation focus groups, discussion groups
analysis of citations in course work or research publications
users’ self-assessment of skills and competences gained
comparison of success data with use data
MET
HO
DS
FOR
IMPA
CT
EVAL
UATI
ON
Inputs Process Outputs OutcomeImpact(changes in confidence,
competencies/skills, behavior)
IL training courses
IL reference materials
Program Resource
persons Students
Trainings/Workshops for librarians
Development of IL program materials
modules IL sessions quizzes and
games self-paced IL
learning tools instructional IL
videos and guides
information products (e.g., pathfinders, guides, etc.)
Librarians know what relevant information are for appropriate users
Users become confident in finding authoritative sources
Users become independent learners and practice information ethics
• Librarians become library research consultants and collaborators
• Users become independent learners and manage information use on their own
• Users succeed in certification exams and in finding employment
Sample: Information Literacy Program
IMPA
CT
EVAL
UATI
ON
PR
OCE
SSStep 1: Impact ObjectivesStep 2: Impact IndicatorsStep 3: Evidence to collectStep 4:
Use the evidence to improve services and secure service sustainability
Focus on Step 1Impact areas Impact objectives
Identify key areas in your service/program that you want to make an impact (e.g., in IL, finding information)
Use specific terms Time-limited (e.g., achievable in 3 years) IM
PACT
EV
ALUA
TIO
N
PRO
CESS
Focus on Step 1Impact areas Impact objectives
Objectives are often confused with activities.
Objectives – what you want to achieve Activities – what you will do to make the
objectives happen
IMPA
CT
EVAL
UATI
ON
PR
OCE
SS
QuizObjective or Activity?
(Markless & Streatfield, 2013)
1. To support managers in carrying out their work effectively and efficiently.
2. To provide ICT-based learning in IL sessions3. To enable students to operate effectively in
an e-environment4. To provide materials appropriate to the
needs of graduate students
IMPA
CT
EVAL
UATI
ON
PR
OCE
SS
QuizObjective or Activity?
(Markless & Streatfield, 2013)
1. To support managers in carrying out their work effectively and efficiently.
2. To provide ICT-based learning in IL sessions
3. To enable students to operate effectively in an e-environment
4. To provide materials appropriate to the needs of graduate students
IMPA
CT
EVAL
UATI
ON
PR
OCE
SS
QuizObjective or Activity?
(Markless & Streatfield, 2013)
5. To increase the confidence and competence of socially excluded people in using library resources
6. To develop children as readers7. To collaborate with faculty in order to integrate
the library into their curriculum8. To develop the skills of the library staff in
delivering effective user-education
IMPA
CT
EVAL
UATI
ON
PR
OCE
SS
QuizObjective or Activity?
(Markless & Streatfield, 2013)
5. To increase the confidence and competence of socially excluded people in using library resources
6. To develop children as readers7. To collaborate with faculty in order to
integrate the library into their curriculum8. To develop the skills of the library staff in
delivering effective user-education
IMPA
CT
EVAL
UATI
ON
PR
OCE
SS
Before you even think about coming
out with a program/project/ser
vice for your users...
Think about the purpose for your initiatives first, and then make sure that it supports your organization's VMG.
Think about impact in planning your initiatives. Always integrate evaluation.
Be convinced that your library has value to your user community and organization. Think lifelong learning effect.
DEMONSTRATING THE LIBRARY’S IMPACT THROUGH ASSESSMENT AND EVALUATION 21 September 2016
Christine M. [email protected]
REFE
REN
CES
Bawden, D., Calvert, A., Robinson, L., Urquhart, C., Bray, C., & Amosford, J. (2009). Understanding our value; assessing the nature of the impact of library services. Library and Information Research, 33(105).
Blixrud, J. C. (n.d.). Evaluating library service quality: use of LibQUAL+. Retrieved from http://libqual.org/documents/admin/blixrud.pdf
Broady-Preston, J., & Lobo, A. (n.d.). Measuring the quality, value and impact of academic libraries: the role of external standards.
Markless, S., & Streatfield, D. (2013). Evaluating the impact of your library. London: Facet Publishing.
Matthews, J. R. (2007). The evaluation and measurement of library services. Westport, Connecticut: Libraries Unlimited.
Poll, R. (2012). Can we quantify the library’s influence? Creating an ISO standard for impact assessment. Performance Measurement and Metrics, 13(2), 121130. http://doi.org/10.18352/lq.7746
Poll, R., & Payne, P. P. (2006). Impact measures for libraries and information services. Library Hi Tech, 24(4), 547–562. http://doi.org/10.1108/07378830610715419
Streatfield, D., & Markless, S. (2009). What is impact assessment and why is it important? Performance Measurement and Metrics, 10(2), 134141.
Workshop