ASSESSING THE IMPACT OF COLLABORATIVE
RESEARCH PROJECTS ON NWS PERFORMANCE
Jeff Waldstreicher
Scientific Services Division – Eastern Region
Northeast Regional Operational Workshop (NROW)
November 4-5, 2003
Albany, NY
ASSESSING THE IMPACTS OF COLLABORATIVE PROJECTS
• IMPACT ON VERIFICATION SCORES– Performance Metrics
• CASE STUDY ANALYSIS– Event Verification– Subjective Evaluation of Impact on Forecast
Process• AFDs• Event Reviews and Impact Reports
COMPLICATIONS
• Many Factors Influence Performance Metrics– Factors Not Independent– Difficult to tie performance changes to a specific factor
• Cannot Analyze “Null Case” – If forecasters have “knowledge” or “data,” cannot
directly answer “What if they did not have the knowledge?”
FACTORS THAT INFLUENCE WARNING VERIFICATION SCORES
• Infusion of New Technologies – Hardware (New systems or Processors) – Software (New algorithms or Models)
• Applied Research and Development
– National (Research Laboratory)– Local/Regional– Independent– Collaborative
• Changes to Operational Procedures– Implementation of Best Practices
FACTORS THAT INFLUENCE WARNING VERIFICATION SCORES
• Climate Variability– Variations in frequency and type of events
• External Outreach and Education– Including development of spotter networks
• Personnel/Staffing Issues– Including forecaster experience levels
• Training– All other factors are also tied to training issues
HOW DO THESE FACTORS IMPACT PERFORMANCE METRICS?
• Can be global (national), regional, or local– Technology usually national
– Climate Variability typically regional
– Staffing usually local • Impact can be long or short term
– Technology infusion typically has long term impact
– Climate Variability is usually short term
• Factors are often inter-related– New technology or staffing changes require training
– Applied R&D often yields/suggests changes to operations
COMET PROJECTS IN EASTERN REGION
• Different types of projects– Cooperative (2-3 year ~$35K/yr)
• 19 ER Cooperative Projects funded since 1991
– Partners (1 year ~$9K/yr)• 39 ER Partners Projects funded since 1991
• 20 ER WFOs and 3 RFC have participated– 90 Offices Nationwide
• 21 Universities have participated in ER projects– 70+ Universities Nationwide
EVALUATING COMET PROJECTS
• Examined COMET Cooperative and Partners Projects in Eastern Region– Projects completed between 1995 and mid-2001– Projects specifically addressing warning programs
• Tornado• Severe Thunderstorms• Flash Flooding• Winter Storms
• Study was designed to minimize as much as possible the impact of the factors previously discussed.
METHODOLOGY• 3-year running verification scores used
– Minimize impact of short-term factors such as variability of events
• Compared 3 years before project to 3 years following project– 1996-mid 2001 period helped ensure 3-years of post-88D data in “before” scores
and a full 3 years of “after” scores
• 3-year “expected” improvements calculated based on long term trend of ER-wide scores– Compared rate of improvement for WFO involved in collaborative project to ER-
wide improvement– Use of ER scores as a baseline minimizes the impact of national/region-wide
factors such as AWIPS and radar system improvements
• Very difficult to evaluate the impact of project results beyond the primary WFO
INDIVIDUAL PROJECT COMPARISON
• Compare 3-year performance change of WFOs involved in COMET projects to the region-wide improvements for the same time period
PROBABILITY OF DETECTION (POD)
# of COMET Projects With
WFO Greater Improvement
# of COMET Projects With ER Greater
Improvement
# of COMET Projects With No
Difference in Improvement
Tornado Warnings 3 2 1
Severe Tstm Warnings 6 1 0
Flash Flood Warnings 5 1 0
Winter Storm Warnings 4 3 0
FALSE ALARM RATIO (FAR)
# of COMET Projects With
WFO Greater
Improvement
# of COMET Projects With ER Greater
Improvement
# of COMET Projects With No
Difference in Improvement
Tornado Warnings 2 4 0
Severe Tstm Warnings 3 4 0
Flash Flood Warnings 3 3 0
Winter Storm Warnings 3 3 1
LEAD TIME
# of COMET Projects With
WFO Greater Improvement
# of COMET Projects With ER Greater
Improvement
# of COMET Projects With No
Difference in Improvement
Tornado Warnings 5 1 0
Severe Tstm Warnings 5 1 1
Flash Flood Warnings 4 1 1
Winter Storm Warnings 4 3 0
IMPACT OF LONG TERM COLLABORATIVE ACTIVITIES
• WFO RAH and NCSU– Continuous collaborative projects since January 1991– COMET Projects 1991-2000
• 3 Cooperative Projects• 3 Partners Projects• 1 Graduate Fellowship
– CSTAR Projects• 2000-2003• 2003-2006
• Unique “laboratory” to examine the impact of long-term collaborative activities
IMPACT OF LONG TERM COLLABORATIVE ACTIVITIES
• WFO ALY and U-Albany– Continuous since February 1995– Numerous COMET Projects
• 3 Cooperative Projects
• 7 Partners Projects
• Several additional projects between UA and NCEP
– CSTAR PROJECT 2001-2004
SUMMARY
• Verification scores for ER WFOs involved in COMET collaborative research projects appear to improve at a greater rate than overall ER performance
• There are indications that certain
performance metrics (e.g., lead time) are more responsive to improved scientific understanding, while others (POD) are more dependent on technology upgrades.