Post on 13-Oct-2020
transcript
Cost Benchmarking Challenges in Higher Education
Tom Eleuterio Program Manager
Higher Education Consortia tommyu@udel.edu
Presentation Overview
Disruption & Higher Education Benchmarking and disruption
Responding to challenges
Questions & Comments
What does disruption in higher education mean in the context of the Delaware Cost Study ?
D - data definitions that may only provide limited clarity
I – innovation in multi-disciplinary, interdisciplinary “CIP defying” programs
S – service to students, the institution, to the discipline and to the community
R – rankings with their questionable fundamentals but unquestionable impact
U – utility of the metrics to decision makers
P – Performance funding driving decision making from outside the institution
T – transparency in sources and methods
I – interpretation of metrics in context of national results
O – on-line learning and its variable impact
N – Networks that are dynamic and unpredictable
Continuous Improvement: Why Benchmark?
• The most integrated data system offers only clear insights into your institution (Case Study).
• Highly effective institutions engage in comprehensive benchmarking processes.
• Internal benchmarking refers to measuring similar operations, functions, or activities within the same unit or organization.
• External benchmarking refers to measuring similar operations, functions, or activities outside the same unit or organization.
What is the Delaware Cost Study? • The National Study of Instructional Costs and Productivity (Delaware Cost Study) is a
benchmarking project and data sharing consortia among four-year colleges and universities with over 200 institutions participating annually.
• Since 1996, over 660 institutions have participated, and over the past two decades,
the Delaware Cost Study has become the “tool of choice” for comparative analysis of faculty teaching loads, direct instructional costs, and separately budgeted scholarly activity, within academic disciplines.
• Currently, the Cost Study is used major data and state agencies including:
– Association of American Universities Data Exchange (AAUDE) – Southern Universities Group (SUG) – University of North Carolina (UNC) System – Pennsylvania State System of Higher Education (PaSSHE) – University of Missouri Syste – City University of New York (CUNY) System
Growing the Delaware Cost Study
188 201 196
174
197 188
176
220
253
0
50
100
150
200
250
300
2006 2007 2008 2009 2010 2011 2012 2013 2014
Institutional Participation, 2006 - 2014
30.0% 23.5%
11.3%
7.9%
7.1%
4.9%
48.6%
48.4%
39.5%
12.6% 21.0%
44.2%
0.9%
0.0%
20.0%
40.0%
60.0%
80.0%
100.0%
120.0%
2014 Delaware Cost Study (n=253) All-Time Delaware Cost Study (n= 638) Carnegie Classification (n=1,831)
Delaware Cost Study Participation by Carnegie Classification Relative to Carnegie Classification
Research Doctoral Master's Baccalaureate Special Focus
5.2% 10.8%
23.4% 28.4% 33.6%
54.0% 47.1% 47.7%
46.3%
51.9% 49.3%
36.4%
5.8% 5.7%
5.1%
12.3% 8.7% 4.8%
40.7% 35.8%
25.1%
7.4% 8.4% 4.8% 1.2%
0.0%
20.0%
40.0%
60.0%
80.0%
100.0%
120.0%
2014 Delaware Cost Study-Public
All Delaware Cost StudyCarnegie Classification -
Public
Carnegie Classification -Public
2014 Delaware Cost Study -Private
All Delaware Cost StudyCarnegie Classification -
Private
Carnegie Classification -Private
Delaware Cost Study Participation and Carnegie Classification by Institutional Control
Baccalaureate Master's Doctoral Research Intensive Special Focus
Historical Objective of the Delaware Cost Study
Answering the questions: Who is teaching what to whom and at what cost? What are the expenditures made for sponsored research and service?
Delaware Cost Study – Strengths and Limitations • Strengths
– Ideal tool for benchmarking instructional costs, research, and public service expenditures at the academic discipline level
– Most systematic and rigorous conceptualization based on CIP levels – Assists institutional data and unit alignment – Assists in identifying cost distortions – Ideal for program reviews and accreditation – Ideal for establishing new program or department projections – Higher Education Consortia is a useful intermediary partner
• Limitations
– Not a “whole cost” benchmarking tool for expenditure or tuition – Not a perfect 1:1 Program/Department/Academic Budget Unit match – Not a tool for performance funding
Why do institutions participate in the Delaware Cost Study?
• According to the National Center for Education Statistics, 76 – 82 percent of the variation in cost is located at the academic disciplinary level.
$136 $164
$171
$140
$217
$231
$152
$181
$301 $167 $164
$213
$226
$173
$250
$305
$185
$240
$405
$228 $187 $296
$289
$218
$312
$384
$251
$314
$493
$308
$0
$100
$200
$300
$400
$500Communications
English
Biology
Mathematics
Chemistry
Physics
Psychology
Political Science
Music
History
FirstQuartile
Median
ThirdQuartile
Quartile Bands for Direct Instructional Expenditure/Student Credit Hour by Ten Most Frequently Reported
Benchmark basis: “Who is teaching what to whom and at what cost?”
Who – what faculty rank as defined as FTE in HR data What - Discipline defined by CIP by NCES definitions To Whom - defined by undergraduate class level, graduate or individual instruction What Cost – Expense on direct instructional activities: salary, benefits, other expenses in support of instruction
Data Checklist for the Delaware Cost Study
“Who is teaching what to whom, and at what cost?”
Unit of Analysis: Academic Budget Units as identified by Classification of Instructional Programs (CIP) Code.
Method: Origin of Instructor - Instructional activity stays with the FTE instructor within unit where they are funded (“follow the money”).
What does disruption in higher education mean in the context of the Delaware Cost Study ?
D - data definitions that may only provide limited clarity
I – innovation in multi-disciplinary, interdisciplinary “CIP defying” programs
S – service to students, the institution, to the discipline and to the community
R – rankings with their questionable fundamentals but unquestionable impact
U – utility of the metrics to decision makers
P – Performance funding driving decision making from outside the institution
T – transparency in sources and methods
I – interpretation of metrics in context of national results
O – on-line learning and its variable impact
N – Networks that are dynamic and unpredictable
Data Definitions and the Delaware Cost Study
Answering the questions: What cost differences are there in STEM versus non-STEM disciplines? What definition of STEM is being used and what disciplines are included or excluded?
Which of the following are STEM disciplines ?
Social psychology International Relations Philosophy of Science Cultural Anthropology
Urban and Regional Planning
Innovative programs and the Delaware Cost Study Defining and defending the cyber-landscape
What does the NCES say about cybersecurity ?
Detail for CIP Code 52.2101 Print Title: Telecommunications Management. Definition: A program that prepares individuals to apply business skills to design, implement, and manage the voice, video, and data networking systems of organizations. Includes instruction in telecommunications concepts and technologies, network operations and management, wireless communications and mobile computing, cybersecurity, regulation and public policy, business practices and management, and written and oral communications.
Searching for the program using “computer security”
Service and the Delaware Cost Study
Answering the question what type of service and to what constituent group: Service to the institution? General education requirements; service in behalf of a discipline. Service to the community? How to measure this in a standard metric? What are the expenditures made for sponsored research and service?
Benchmarking Unit Activity by Carnegie Classification
Rankings and the Delaware Cost Study
Answering the questions: How do we compare to our current and aspirational peers? What do the national norms illuminate about the student experience in a program?
Benchmarking Instructional Activity – Psychology
90
62
79 79 84
0 38 21 21
16 10
Undergraduate Student Credit Hour Share by Faculty Rank relative to Carnegie Classification
00 00 00 00 0
77 76 76
62 60
5 0 6 0 7
0
15
9
15 18 17 18
22
0 0 0
10
20
30
40
50
60
70
80
90
100
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
2008 2009 2010 2011 2012 T/TT Faculty Regular Faculty Supplemental Faculty Teaching Assistant
Carnegie T/TT Carnegie ORF Carnegie Supplemental Carnegie TA
Utility of the results of the Delaware Cost Study
Answering the questions: What variability in cost can be expected from year to year across categories of disciplines? What are the likely impacts on cost cutting measures?
Delaware Cost Study Data Uses
Academic/Accreditation Program Review Faculty Hiring/Disparity Chair Key Performance Indicators Deans Dashboard Senior Budgeting/President/Provost Planning/Projections Identifying Cost Distortions in Budgeting Formulas Developing New Programs/Departments Grants and Research Expenditure Benchmarking External Audit/System Review Tool Student/Faculty Recruitment/Retention Tool General Unit and Institutional Improvement
Advanced Modeling Techniques Utilizing Delaware Cost Study Data
• An Ordinary Least Squares (OLS) Regression model allows for the researcher to predict DIE/FTE Student for RVH Physics. (Weighting for Percent Undergraduate Degree)
• Findings indicate for every one percent increase
in personnel cost as a percentage of total departmental expenditure is associated with a $142.84 decrease in DIE/FTE Student (p=.01).
• For every one dollar increase in research
expenditure per T/TT FTE faculty is associated with a $0.007 increase in DIE/FTE Student (p=.03).
Advanced Modeling Techniques Utilizing Delaware Cost Study Data
• Monte Carlo methods allow for the simulation of estimated future costs. When simulating the DIE/FTE Student, 1,000,0000 times, a 90 percent confidence interval can be estimated.
• For RVH Physics, the average DIE/FTE
Student is $9,340.
• For RVH Physics departments, there is a 5% chance a department’s DIE/FTE Student cost will be above $12,450. For RVH Physics, there is a 5% chance a department’s DIE/FTE Student cost will be below $6,680.
• Monte Carlo sensitivity analyses allow for
the researcher to manipulate variables to influence projections.
Politics and the impact of the Delaware Cost Study
Answering the questions: Are faculty meeting legislated productivity standards measured by student credit hours and organized class sections taught? What are the comparative expenditures made for instructional activity, sponsored research and service relative to national norms?
How Many Student Credit Hours by Discipline, Course Level and Type are Required to Generate the Need for One Instruction Full-Time Equivalent Person?
Assessing Student Credit Hour Production and Direct Instructional Expenditure
Direct Instructional Expenditure per FTE Student as Percentage of Carnegie Classification
Student Credit Hours per FTE Faculty as Percentage of Carnegie Classification
Low Moderate High
Low Theater Physics, Chemical Engineering
Moderate Criminal Justice, Political Science
Biology, Music Chemistry
High Communication, English, History, Psychology, Sociology
Mathematics
Transparency and the Delaware Cost Study
Answering the questions: How does our program compare with respect to the cost of others in our institution and with similar programs at other comparable institutions? What are the normative expectations for expenditures made for sponsored research and service?
Academic Program Review - National Carnegie Comparison -
Interpretation of the Delaware Cost Study results
Answering the questions: Who is teaching what to whom and at what cost? What are the expenditures made for sponsored research and service?
On-line Learning and the Delaware Cost Study
Answering the questions: What have we been missing in answering “Who is teaching what to whom and at what cost”? What is the impact of digital learning on the effectiveness and cost of instruction?
February 2015 GRADE LEVEL TRACKING ONLINE EDUCATION IN THE UNITED STATES Background: While the number of students taking distance courses has grown by the millions over the past decade, it has not come without considerable concerns. Faculty acceptance has lagged, concerns about student retention linger, and leaders continue to worry that online courses require more faculty effort than face-to-face instruction. Evidence: Chief academic officers report no major improvements for any of these areas of concern.
February 2015 GRADE LEVEL TRACKING ONLINE EDUCATION IN THE UNITED STATES Only 28.0% of chief academic officers say that their faculty members accept the “value and legitimacy of online education,” a rate substantially the same as it was in 2003. Most academic leaders (68.3%) continue to believe that “Students need more discipline to succeed in an online course than in a face-to-face course.” Increasing numbers of academic leaders think that retaining students is a greater problem for online courses than for face-to-face courses (44.6% in 2014 versus 40.6% in 2013, 28.4% in 2009, and 27.2% in 2004). Additional effort required to deliver an online course represents a barrier for online instruction for 78.0% of academic leaders.
Networks and the Delaware Cost Study
Answering the questions: What are the important social networks that are driving change in higher education? What is the evidence that administrative decisions are in-tune with social realities?
Advanced Modeling Techniques Utilizing Cost Study Peer Ratio Tool
• Physics is a discipline with relatively high costs and high variability, even among comparable Research Very High (RVH) institutions.
• Conducting RVH cluster analyses with Cost Study data enables the researcher to explore the relationship between key variables related to Direct Instructional Expenditure (DIE)/Full-Time Equivalent (FTE) Student.
• High research expenditures is associated with lower personnel costs as a percentage of total expenditure as units are spending more on other than personnel costs.
Peer Ratio Access – Institutional Login • Once you have logged institutionally, click on “Peer Analysis” and then “Peer
Ratio Data.”
Peer Ratio Access – Institutional Login • Once you have selected “Peer Ratio Data,” select ten or more institutions, the
results will be generated in a manner that can be viewed and/or downloaded into Microsoft Excel. Additional instruction are on the right of the page.
Disruption and the Delaware Cost Study
Answering the questions: How do we facilitate good decision-making when the major constant to expect is change? What short-term decisions are prudent in the context of their long-term consequences?
What does disruption in higher education mean in the context of the Delaware Cost Study ?
D - data definitions that may only provide limited clarity
I – innovation in multi-disciplinary, interdisciplinary “CIP” defying programs
S – service to students, the institution, to the discipline and to the community
R – rankings with their questionable fundamentals but unquestionable impact
U – utility of the metrics to decision makers
P – Performance funding driving decision making from outside the institution
T – transparency in sources and methods
I – interpretation of metrics in context of national results
O – on-line learning and its variable impact
N – Networks that are dynamic and unpredictable
Future Directions: Higher Education Consortia
• Over the past two years, the Higher Education Consortia has worked with four-year colleges, universities, and systems to facilitate unit and institutional improvement on a limited basis.
• Data Alignment
• Statistical Solutions ( OLS, HLM, SEM, Survival – time to event)
• Academic Program Review
• Internal and External Benchmarking • Special Projects
Addressing Your Questions and Comments
• What questions do you have at this time?
Moving Forward: Implementation and Support
Online Data Collection Walkthrough Web Portal Access for Data Submission
Relational Database Historical Data
Personal Support/Consultation
First Year Conference Call Prospective Conferences/Workshops ( NEAIR, SCUP)
What academic level is being taught…( as a percent of instruction)?
Who is teaching what to whom…(as a percent of instruction)?
How many student credit hours and FTE students…?
What level of expenditure for instruction, research and service…?
• Cluster analysis is a series of statistical techniques designed to identify how similar (or different) some observations are from one another.
• Cluster analysis is a data classification technique
rather than a test for statistical significance. • The next-nearest-neighbor is a cluster analysis
approach designed to assess how close data points are to a specific point based upon majority. If k=3, red triangle. If k=5, blue square.
Advanced Modeling Techniques Utilizing Cost Study Peer Ratio Tool
Getting the Most from Delaware Cost Study Data
Projection
Prediction Description
Regression: Best Fit Lines
• A regression is a best-fit line that lies closer to the data points than any other possible line according to a least squares standard statistical measure of closeness.
• Ordinary Least Squares (OLS) regression is a statistical improvement bivariate statistical analyses because they allow the researcher to “control for” or “separate” certain aspects of independent variables on a dependent variable.
• Regression analysis is like a mosaic of real life experiences that allow complicated patterns of interaction to be disentangled on a statistical level.
Take Away: The Delaware Cost Study is a resource.
• The Delaware Cost Study is a useful tool for providing comparative analysis of faculty teaching loads, direct instructional costs, and separately budgeted scholarly activity, all within academic disciplines.
• Delaware Cost Study is useful for data alignment, system integration,
unit improvement, and institutional effectiveness.
• A failure to monitor these variables may lead to expenditure distortion in planning and budget models (ABB, RCM, PBB, ZBB).
• Cost Study data (courses taught, research and public service
expenditures) may be useful for Academic Program Reviews (APR). • Cost Study data may be useful for proposing, planning and developing
new departmental programs.