Date post: | 28-Dec-2015 |
Category: |
Documents |
Upload: | linette-simon |
View: | 214 times |
Download: | 0 times |
FROM DATA TO ASSESSMENT
Collecting, Presenting, and Using Statistics
Jim SelfUniversity of Virginia VIVA Stats WorkshopFebruary 2, 2010
THIS MORNING’S PROGRAMA few basic statistical conceptsDisplaying the dataExamples from U.Va. and
elsewhereMoving toward assessmentQuestions, comments
WHY MEASURE OR COUNT?To evaluateTo compareTo improve
WHEN NOT TO COLLECT DATA
When someone has already done itWhen the data will not be usefulWhen the results do not justify the
costs
MAKING DATA MEANINGFUL
SummarizeCompareAnalyzePresent
MEASURES OF CENTRAL TENDENCY
MedianMeanModeMidrange
MEASURES OF DISPERSION
RangeVarianceStandard DeviationInterquartile Range
SAMPLINGSelection of members is crucial
Strive for an unbiased sampleUnambiguous, pre-set selection process
Sample sizeLarge enough for appropriate precisionSmall enough to be economical
DETERMINING SAMPLE SIZE
Confidence Interval
Level of confidence
U.VA. STUDENT SURVEYQuestion: Do you own a computer
at U.Va?First Year Students
Yes 99.0%Graduate Students
Yes 88.2%
Do we know if these two groups are different in terms of computer ownership?
95% LEVEL OF CONFIDENCE
Population Size
Sample Size
Percent in Sample Answering “Yes”
Plus orMinusSampleError
Min.Value
Max.Value
Grads 5,400 443 88.2% 2.9% 85.3% 91.1%
1stYears
2,900 102 99.0% 1.9% 97.1% 99.9+%
PREPARATION FORDATA COLLECTION
Clarify your objectivesPrefer the simplest proceduresChoose the appropriate technique
Collect only data you can use
FOUR USEFUL ASSUMPTIONS
Your problem is not unique.
You have more data than you think.
You need less data than you think.
There is a useful measurement that is much simpler than you think.
Hubbard, D. (2007) “How to Measure Anything.”
PROCESSING THE DATA
Check and re-check the numbersSummarizeCompareFind the interesting pointsLook for explanations
COMMON PITFALLS Sampling with bias Surveys with a message Graphs with distorted figures Graphs with varying time frames Atypical projections Inappropriate grouping of data False precision Confusion of correlation and causality
“…BUT TO SUPPOSE THAT THE FACTS, ONCE ESTABLISHED IN ALL THEIR FULLNESS, WILL ‘SPEAK FOR THEMSELVES’ IS AN ILLUSION.”
Carl Becker
Annual Address of the President of the American Historical Association, 1931
DATA REPORTING
Integrate data, graphs, and narrative to tell a succinct and accurate story.
Show the data Give the source of the data Do not decorate graphics, let the data
speak Put your name on the report
UVA LIBRARY SURVEYS Faculty
1993, 1996, 2000, 2004Response rates 59% to 70%
Students 1994, 1998, 2001, 2005Separate analysis for grads and undergradsResponse rates 43% to 63%
LibQual+ 2006Response rates 14% to 24%
Annual Surveys 2008+Student samplesOne third of facultyResponse rates 29% to 47%
FINAL QUESTIONU.VA. LIBRARY SURVEY
Please rate your overall satisfaction with the library.
Not at all Very No
Satisfied Satisfied Opinion
1 2 3 4 5 X
2000 U.VA. FACULTY SURVEYOVERALL RATING
1 2 3 4 5 X
0 5 19 126 154 11
U.VA. FACULTY SURVEYSOVERALL RATING
1 2 3 4 5 X Total
1993 1 5 52 174 93 24 349
1996 1 8 36 146 136 12 339
2000 0 5 19 126 154 11 315
OVERALL SATISFACTIONU.VA. LIBRARY SURVEYS
Category Mean
1993 Faculty 4.09
1994 Undergraduates 4.01
Graduate Students 3.96
1996 Faculty 4.26
1998 Undergraduates 3.90
Graduate Students 3.97
2000 Faculty 4.41
2001 Undergraduates 4.07
Graduate Students 4. 02
2004 Faculty 4.20
2005 Undergraduates 3.94
Graduate Students 4.02
OVERALL SATISFACTIONUNIVERSITY OF VIRGINIA LIBRARY
1993/94
1996/98
2000/01
2004/05 2008/09
Faculty 4.09 4.26 4.41 4.20 4.23
Graduate Students
3.96 3.97 3.97 4.02 4.17
Undergrads
4.01 3.90 3.97 3.94 4.12
OVERALL SATISFACTIONUNIVERSITY OF VIRGINIA LIBRARY1993-2009
1993/94 1996/98 2000/01 2004/05 2008/093.80
4.00
4.20
4.40
4.60
4.09
4.23
4.01
4.12
3.96
4.17
Faculty
Undergrads
Grads
U.Va. Library Surveys 1993-2009 http://www2.lib.virginia.edu/mis/reports/
U.VA. FACULTY AND STUDENTSWHO PHYSICALLY VISIT A LIBRARYAT LEAST ONCE A WEEK
1993/94 1996/98 2000/01 2004/05 2008/090%
20%
40%
60%
80%
100%
79%
26%
69%72%
84%
43%
Faculty
Grads
Undergrads
DATA PRESENTATION ACCORDING TO DILBERT
THE LIBRARY BUDGET AS PIE
DR1% EI, DI
4%
EF, ER, EU9%
FA2%
GF1%
LG2%
LS1%
SE2%
SG77%
SR,SS0%
Sources of Funding FY09
CORROBORATIONData are more credible if they are
supported by other informationJohn Le Carre’s two proofs
ANALYZING SURVEY RESULTS
Two Scores for Resources, Services, FacilitiesSatisfaction = Mean Rating (1 to 5)Visibility = Percentage Answering the Question
Permits comparison over time and among groups
Identifies areas that need more attention
UVA REFERENCE ACTIVITY AND
REFERENCE VISIBILITY IN STUDENT SURVEYS
1,756
6,008
34%Visibility
39% Visibility
75% Visibililty
64% Visibility
1,000
7,000
1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004
Ref
eren
ce Q
ues
tion
sR
eco
rded
per
Wee
k i
n A
nn
ual
Sam
ple
10%
Ref
eren
ce V
isib
ility
a
mo
ng
Un
der
gra
du
ate
COLLECTING AND USING DATA AT THE U.VA. LIBRARY
Customer Surveys Staff Surveys Mining Existing Records Comparisons with peers Qualitative techniques Statistical compilations Revealing the budget process and issues
www.arl.org
DATA MININGAcquisitionsCirculationFinanceUniversity Records
COLLECTIONS EXPENDITURES BY FORMAT
FY1993
FY1994
FY1995
FY1996
FY1997
FY1998
FY1999
FY2000
FY2001
FY2002
FY2003
FY2004
FY2005
FY2006
FY2007
$0
$500,000
$1,000,000
$1,500,000
$2,000,000
$2,500,000
$3,000,000
$3,500,000
$4,000,000
University of Virginia Library
Electronic Resources
Print Monographs
Print Serials
Manuscripts
Miscellaneous
Microforms
Video or Film
Fiscal Year
Ex
pe
nd
itu
res
USAGE DATA AT UVA – CIRC, REF, E-JRNLS
1992
/93
1993
/94
1994
/95
1995
/96
1996
/97
1997
/98
1998
/99
1999
/00
2000
/01
2001
/02
2002
/03
2003
/04
2004
/05
2005
/06
2006
/07
0
500,000
1,000,000
1,500,000
2,000,000
2,500,000
Circulation
Reference
E-Journal Articles
INVESTMENT AND CUSTOMER ACTIVITYUNIVERSITY OF VIRGINIA LIBRARY1993-2006
Customer Activities
0
600,000
1,200,000
1,800,000
2,400,000
FY93 FY94 FY95 FY96 FY97 FY98 FY99 FY00 FY01 FY02 FY03 FY04 FY05 FY06
E-Journals
Circulation
Reference
Acquisitions Expenditures by Format
$0
$1,000,000
$2,000,000
$3,000,000
FY93 FY94 FY95 FY96 FY97 FY98 FY99 FY00 FY01 FY02 FY03 FY04 FY05 FY06
Electronic Resources
Print Monographs
Print Serials
TOTAL EXPENDITURES AT UVA 1989—2003
-50%
300%
Fiscal Year
% C
hang
e si
nce
1989 Other Academic Support
(+200%)
Research (+219%)
Total Academic Division (+140%)
Libraries (+81%)
Instruction (+80%)
LOOKING AT LIBQUAL+
22 core questions1-9 scaleRatings of minimum, desired, perceived
Locally selected questions General satisfaction ratings
SELECTING AND DISPLAYING LIBQUAL+
Choose the meaningful data Ignore the chaff
Summarize those data Compare
By category and subgroupsOver timeWith peers
Present findings transparently
LibQUAL+ 2006ARL Composite Faculty
5
6
7
8
9
AS-1 AS-2 AS-3 AS-4 AS-5 AS-6 AS-7 AS-8 AS-9 IC-1 IC-2 IC-3 IC-4 IC-5 IC-6 IC-7 IC-8 LP-1 LP-2 LP-3 LP-4 LP-5
5
6
7
8
9
AS-1 AS-2 AS-3 AS-4 AS-5 AS-6 AS-7 AS-8 AS-9 IC-1 IC-2 IC-3 IC-4 IC-5 IC-6 IC-7 IC-8 LP-1 LP-2 LP-3 LP-4 LP-5
LibQUAL+ 2006University of Virginia Faculty
5
6
7
8
9
Faculty AS-3 Grad AS-3 Undergrad AS-3 Library Staff AS-3
LibQUAL+ 2006Ratings of AS-3:
Employees who are consistently courteous
Faculty
Grads Undergrads
Library Staff
USING DATA AT UVA
Additional resources for the science libraries (1994+)
Redefinition of collection development (1996)
Initiative to improve shelving (1999)
Clemons Library open 24 hours (2000)
Additional resources for the Fine Arts Library (2000)
Support for transition from print to e-journals (2004)
New and improved study space (2005-06)
Increased appreciation of the role of journals (2007)
Re-design of main floor of Clemons Library (2008)
USING DATA:FROM STATS TO ASSESSMENT
CountingTraditional statistics emphasize inputs:
expenditures, acquisitions, holdings Assessing
More recent efforts assess outputs: service quality
WHY ASSESS?Improve services Justify resourcesCompare with othersIdentify changing needsIdentify questionable servicesPromote services
PARALLEL TRENDSCustomer-Centered
Library
All services and activities are viewed through the eyes of the customers
Customers determine quality
Library services and resources add value to the customer
Culture of Assessment
Organizational environment in which decisions are based on facts, research and analysis.
Services are planned and delivered to maximize positive customer outcomes
EFFECTIVE ASSESSMENT
Is customer-focused Uses representative groups of users Asks fair and unbiased queries Develops criteria for success Uses multiple assessment methods Uses corroboration from other sources Provides results that can be used to
improve libraries
A CULTURE OF ASSESSMENT
Decisions based on facts, research, and analysis
Wide recognition and acceptance of the importance of assessment
Expertise and staff development related to assessment