Efficiency as a Measure of Knowledge Production of
Research Universities
Amy W. Apon* Linh B. Ngo*Michael E. Payne* Paul W. Wilson+
School of Computing* and Department of Economics +
Clemson University
1
Content• Motivation• Methodology• Data Description• Case Studies• Conclusion
2
Motivation
• Recent economic and social events motivate universities and federal agencies to seek more measures from which to gain insights on return on investment
3
Motivation
• Traditional measures of productivity:– Expenditures, counts of publications, citations, student
enrollment, retention, graduation …
• These may not be adequate for strategic decision making
• Traditional Measures of Institutions’ Research Productivity:– Are primarily parametric-based– Often ignore the scale of operation
4
Research Question
• What makes this institution more efficient in producing research?
• What makes this group of institutions more efficient in producing research?
• How do we show statistically that one group of institutions is more efficient than the other group
5
Efficiency as a Measure
• Using efficiency as a measure of knowledge production of universities– Extends traditional metrics– Utilizes non-parametric statistical methods
• Non-parametric estimations of relative efficiency of production units
• No endogeneity: we are not estimating conditional mean function because we are not working in a regression framework
• Scale of operations is taken into consideration
– Rigorous hypothesis testing
6
Background
• We define as the set of feasible combinations of p inputs and q outputs, also called the production set.
• There exists a maximum level of output on a given input (the concept of efficiency)
• The efficiency score is an estimation with regard to the true efficiency frontier
• Range: [0,1]Input
Output
Infeasible se
t
Feasible set
7
Hypothesis Testing Procedure
8
Convexity
• Test for Convexity– Null hypothesis: The production set is convex– Alternative: The production set is not convex
Input
Output
Infeasible se
t
Feasible set
Input
Output
Infeasible se
t
Feasible set
Input
Output
Infeasible se
t
Feasible set
9
Constant Returns to Scale
• Test for Constant Returns to Scale– Null hypothesis: The production set has constant returns to
scale– Alternative: The production set has variable returns to scale
Input
Output
Infeasible se
t
Feasible set
Input
Output
Infeasible se
t
Feasible set
10
Group Distribution Comparison
• Test for Equivalent Means (EM)– Null hypothesis: – Alternative:
• Test for First Order Stochastic Dominance (SD) between the two efficiency distributions:– Null hypothesis: distribution 1 does not dominate
distribution 2– Alternative: distribution 1 dominates distribution 2
11
Case Studies
• University Level• Departmental Level • Grouping Categories– EPSCoR vs. NonEPSCoR– Public vs. Private– Very High Research vs. High Research– “Has HPC” versus “Does not have HPC”
12
Hypotheses
• Institutions from states with more federal funding (NonEPSCoR) will be more efficient than institutions from states with less federal funding (EPSCoR)
• Private institutions will be more efficient than public institutions
• Institutions with very high research activities will be more efficient than institutions with high research activities
13
University: Data Description
• NCSES Academic Institution Profiles• NSF WebCASPAR• Web of Science• Aggregated data from 2003-2009• Input: Faculty Count, Federal Expenditures• Output: PhD Graduates, Publication Counts
14
University
• Test of Convexity:– p = 0.4951: Fail to reject the null hypothesis of convexity
• Test of Constant Returns to Scale:– p = 0.9244: Fail to reject the null hypothesis of constant
return to scale
15
University: EPSCoR vs NonEPSCoR
• While the first set of EM/SD tests indicates that the distribution of efficiency scores for EPSCoR institutions does not dominate the distribution of efficiency scores for NonEPSCoR institutions,
• The second set of EM/SD tests also rejects the notion that the distribution of efficiency scores for NonEPSCoR institutions is greater than the distribution of efficiency scores for EPSCoR institutions.
• This implies that NonEPSCoR institutions are at least as efficient as EPSCoR institutions
EPSCoR NonEPSCoR p-values for EM and SD testsGroup 1: EPSCoR
Group 2: NonEPSCoR
p-values for EM and SD testsGroup 1: NonEPSCoR
Group 2: EPSCoR
Count 45 118 EM SD EM SD
Mean Efficiency
0.325 0.385 0.993 0.999 --
16
University: Public vs. Private
• The first set of EM/SD tests indicates that the distribution of efficiency scores for public institutions dominates the distribution of efficiency scores for private institutions,
• The second set of EM/SD tests also supports this result by rejects the notion that the distribution of efficiency scores for public institutions is greater than the distribution of efficiency scores for private institutions.
• This result shows strong evidence that public institutions are more efficient than private institutions
Public Private p-values for EM and SD testsGroup 1: PublicGroup 2: Private
p-values for EM and SD testsGroup 1: PrivateGroup 2: Public
Count 110 53 EM SD EM SD
Mean Efficiency
0.396 0.311 0.011 0.999 --
17
University: VHR vs. HR
• This result shows strong evidence that institutions with very high research activities are more efficient than institutions with only high research activities
VHR HR p-values for EM and SD testsGroup 1: VHRGroup 2: HR
p-values for EM and SD testsGroup 1: HR
Group 2: VHRCount 80 83 EM SD EM SD
Mean Efficiency
0.398 0.338 0.021 0.999 --
18
Department: Data Description
• National Research Council: Data-Based Assessment of Research-Doctorate Programs in the U.S. for 2005-2006
• Input: Faculty Count, Average GRE Scores• Output: PhD Graduates, Publication Counts• 8 academic fields have sufficient data:
– Biology – Chemistry– Computer Science– Electrical and Computer Engineering– English– History– Math– Physics
19
DepartmentDepartment p-values
Test for Convexity Test for Constant Returns to Scale
Biology 0.032 --
Chemistry 0.466 0.060
Computer Science 0.368 0.999
Electrical and Computer Engineering
0.078 --
English 0.003 --
History --
Mathematics 0.626 0.894
Physics 0.214 0.999
20
Department: EPSCoR vs NonEPSCoR
EPSCoR NonEPSCoRp-value for EM/SD tests:
Group 1: EPSCoRGroup 2: NonEPSCoR
p-values for EM/SD tests:Group 1: NonEPSCoR
Group 2: EPSCoR
Count/Mean Efficiency EM SD EM SDBiology 35/0.81 86/0.88 0.997 0.999 --
Chemistry 54/0.39 126/0.51 0.858 0.999 --Computer
Science 30/0.3 97/0.49 0.999 0.999 --
Electrical and Computer
Engineering34/0.66 102/0.87 0.999 0.999 --
English 27/0.91 92/0.89 0.648 0.999 --History 30/0.92 107/0.92 0.0000 0.802 0.999 --
Mathematics 32/0.48 95/0.59 0.953 0.999 --Physics 41/0.44 120/0.59 0.999 0.999 --
21
Department: Public vs. Private
Public Privatep-value for EM/SD tests:
Group 1: PublicGroup 2: Private
p-values for EM/SD tests:Group 1: PrivateGroup 2: Public
Count/Mean Efficiency EM SD EM SDBiology 82/0.85 39/0.89 -- 0.230
Chemistry 130/0.45 50/0.53 -- 0.096Computer
Science 92/0.42 35/0.5 0.984 0.999 --
Electrical and Computer
Engineering97/0.79 39/0.86 -- 0.127
English 81/0.89 38/0.92 -- 0.3626History 87/0.92 50/0.91 0.9999 -- 0.9318
Mathematics 90/0.55 37/0.59 -- 0.8265 --Physics 11/0.54 50/0.59 -- 0.0861 0.1917
22
Department: VHR vs. HR
VHR HRp-value for EM/SD tests:
Group 1: VHRGroup 2: HR
p-values for EM/SD tests:Group 1: HR
Group 2: VHRCount/Efficiency EM SD EM SD
Biology 67/0.89 40/0.79 0.999 -- 0.999Chemistry 115/0.56 57/0.35 0.010 0.989 --Computer
Science 95/0.5 29/0.28 0.999 -- 0.999
Electrical and Computer
Engineering94/0.83 37/0.77 0.999 -- 0.950
English 85/0.89 32/0.91 0.999 -- 0.246History 101/0.92 33/0.91 0.999 -- 0.0000 0.935
Mathematics 94/0.61 32/0.42 0.0001 0.999 --Physics 117/0.63 42/0.35 0.968 -- 0.999 --
23
Implication
• Efficiency estimations, together with hypothesis testing, provide insights for strategic decision making, particularly at departmental level.
• Lower efficiency estimate does not mean a program is not doing well.
• Issues:– Lack of data and integration/curation of data
24
Questions
25
