Research Performance Measures Survey 2014

Bill StoneResearch AdministratorMissouri Department of Transportation

Presented at 2014 AASHTO Research Advisory Committee Meeting,Madison WI, July 24, 2014

AASHTO RAC SurveyState DOT Survey -- March 2010

Identified very broadly defined performance measures used by state DOTs at the timeMeasures taken from public sources such as accountability reports, state DOT websites, and other web-based resources Out of 40 states, 10 had published measures and 30 had no public informationSurvey lists metric, source and URL

DOT State Statshttp://stats.mtkn.org The DOT State Stats report is a synthesis of facts, figures, statistics and metrics pulled from accountability reports, online performance measurement dashboards and fact books. 2009: Need existed to easily find and access local statisticsEditions available: 2008-2012

DOT State StatsAnnual figures compiled in easy-to-use synthesis report online. Excel dataset also available online.Created and launched by previous MoDOT Librarian; assisted by members of the Midwest Transportation Knowledge Network

(MTKN) and hosted on their website.2012 edition included metrics from 42 states.Publication is used by the transportation community at large.

DOT State Stats

State Department of Transportation Benchmarks, Facts, and StatisticsCurrent as of Friday Mar. 22nd, 2012

State Measure Year CitationIA Number of new transportation research dollars secured 2011WA Total value of 2011-2013 research projects 2011WA Strategic planning & research fund distribution 2011WA Number of WSDOT research projects planned 2011

Searching on “research” retrieves only 4 results out of 1,580 measures

Synthesis report no indexDataset searchable but not categorizedNo longer current not updated since 2012

RAC Value of Research Task ForceState DOT Survey -- May 2014

Obtain input from research administrators actual or anticipated use of research performance measures to document the progress and success of their research programs

Measures with performance targets or metrics that track activityMeasures that were program-specific not project-specificMeasures from all research program areas (pure research projects, library and/or technology transfer program)

2014 Survey Results

49% response rate(25 states)

Does Your State Track Research Program Performance?

15, 60%

10, 40%

Yes No

For States That Don’t Track Measures

Have future plans (5)

High interest (1)

Have potential measure in

mind (1)

Discussed it (1)

Track but not as measure (1)

No comment (1)

Research Performance Measure Characteristics

• Track activity

56%• Have performance

targets28%

How Frequently Are Measures Collected or Reported?All Responses

02468

10

98

1 1

Annually Quarterly Monthly Bi-Annually

20% of states collect or report statistics for more than just one time period

Are the Measures Published?All Responses-13% of States Do Both

73%

13%

13%

Internally

Externally

Do Both

How Does Your State Use the Measures?

CURRENT USECustomer satisfaction (AZ)Monitor progress toward strategic goals (DC); adjust performance (TX)Make a difference in organizational business processes (MO)Determine program effectiveness (IA)Identify strengths and weaknesses (IN); what’s working or not working (MD)Tied to individual performance evaluations (LA);

How Does Your State Use the Measures?

CURRENT USE (CONT’D)Assess researcher performance and the usefulness of completed research (NC)Select new projects (TX)Ensure that projects move forward according to set budget and schedule (UT, WY)Develop and monitor SPR Part 2 program (WI)

FUTURE USEWill use to direct funding (FL)Will use to gauge and improve performance (IL)

Measure Categories

Applied 31 unique categories to 103 submitted individual measuresMeasures by Status

Current: 69 measuresPlanned: 21 measuresConsidering: 10 measuresPast: 2 measuresIn progress: 1 measures

Some measures have multiple categories applied

Measure CategoriesTOP CURRENT: Completion (14), Implementation (9), Library Utilization/Processing (8), Training (7), Satisfaction (6)TOP IN PROGRESS/PLANNED/POTENTIAL: Implementation (6), Completion (3), Cost Savings (3), Timeliness (3), Within Budget (3)

Completion CategoryExamples of Measures

Related to numberNo. of completed research projects

Related to on-time or on-budget% of research projects meeting original completion dateNo. of projects completed in the FY on schedule% of projects completed according to the initial budget% of projects completed on time/on budget% of studies completed within the approved schedule of the Work Program

Completion CategoryExamples of Measures Cont’d

Related to satisfaction% of completed studies deemed satisfactory by the project sponsorCustomer satisfaction survey for completed Research

Related to implementationIn past 5 years, 75% of completed research projects provide recommendations for implementation of results endorsed by the Project Review Committee

Implementation CategoryExamples of Measures

Related to numberNo. of research results and best practices implementedNo. of NCHRP and other external research program results implemented% of projects implemented

Fully, partially, later, cannot/or should not be implemented, within 2 years of final research report (using 5 years of data)

% of projects actively utilized in the field

Implementation CategoryExamples of Measures Cont’d

Related to number cont’d% of projects that have resulted in a spec, policy, procedure, manual, requirement or material changeIn past 5 years, 75% of completed research projects provide recommendations for implementation of results endorsed by the Project Review CommitteeMulti-year tracking of implementation work

Related to fundingAmount of funding for implementation activities

Level of Effort to Compile Measures

A lot of effort Moderate Minimal0%5%

10%15%20%25%30%35%40%45%

27%33%

40%

Tools Used to Collect MeasuresAll Responses-75% of States Use More Than One

State DOT softwareSiteManager

OtherGovt databases

WebsiteCustom built software

Project mgmt softwareAccessSurvey

Excel

0 2 4 6 8 10 12111

22

33

68

10

How Have You Used the RPM Website?

CA and MN: ReviewedDC & LA: Used as basis for developing PMsIL: Used to develop a structure for functional PMsIN: Exploring how to move measures to RPMUT: Use to upload measuresID, LA, MI, MO, PA and UT: Use for HVR submittals

RPM Website Tools: How Useful?

Not Used Tools43%

Somewhat Useful

29%

Extremely Useful

14%

Somewhat Not Useful

14%

RPM Web CommentsPROS

“RPM Web was extremely useful in helping us grasp the different types of performance measures and developing a structure for functional performance measures.” “Need to re-visit the site now that our program is more developed to see how it might help.”

CONS“Adapting to RPM Web would require a lot of work.”“Have mixed feelings about the usefulness.”“Did not find the tools very beneficial, because the benefits need to be customized for individual states …”“Information may be duplicated elsewhere … Would need to build the measures into projects prior to start … Input not required/hasn’t been a priority.”

Observations/Questions

Who bears the cost of developing measures for each individual project? Researcher? Program director?What is the benefit/cost difference of applying measures to all projects versus a sample?What is the right balance between qualitative vs. quantitative data to demonstrate program effectiveness?Issues/challenges with what administration wants or needs (prediction versus real cost or savings)

Contact Information

Bill Stone, PEConstruction, Materials and Research

1617 Missouri Blvd. P.O. Box 270

Jefferson City Missouri 65102-0270573-526-4328

William.Stone@modot.mo.gov

ResearchAhead