33
1 © CFI Group National Weather Service Marine/Tropical Survey Presentation of Customer Satisfaction Results March 16, 2004 Federal Consulting Group
1© CFI Group
National Weather ServiceMarine/Tropical Survey
Presentation ofCustomer Satisfaction Results
March 16, 2004
Federal Consulting Group
2© CFI Group
Introduction
3© CFI Group
Snapshot of the American Customer Satisfaction Index (ACSI)
• The #1 national economic indicator of customer satisfaction
• Compiled by the National Quality Research Centerat University of Michigan since 1994 usingmethodology licensed from CFI Group
• Measures 30 industries, 180+ organizations covering 75% of the U.S. economy
– Over 70 U.S. Federal Government agencies have used ACSI to measure more than 120 programs/services
• Advanced methodology quantifiably measures and links satisfaction levels to performance and prioritizes actions for improvement
• Results from all surveys are published quarterly in The Wall Street Journal
4© CFI Group
Scope of the 2003 NWSCustomer Satisfaction Program Four segments measured in 2003
– Media – completed
– Emergency managers – completed
– Aviation (pilots and dispatchers) – completed
– Marine/Tropical – completed
5© CFI Group
Marine Customer Satisfaction Results
6© CFI Group
Project Background
• Target segment: Users of Marine and Coastal, Tropical, and/or Great Lakes products and services, including recreational boaters, coastal residents, beachgoers, and professionals.
• Survey content: Survey designed over July and August 2003, with feedback from process owners in NWS (Silver Spring and regions).
• Survey method: The survey was conducted online and hosted on CFI Group’s secure server.
• Customers contacted: NWS publicized the survey to interested parties and provided links to the survey on multiple web sites, including the National Hurricane Center.
• Response totals: The online survey accepted responses from September 9 through October 6, 2003. Completed a total of 6298 interviews.
7© CFI Group
0
100
200
300
400
500
600
700
800
900
1000
9-S
ep
11-S
ep
13
-Se
p
15
-Se
p
17
-Se
p
19
-Se
p
21
-Se
p
23
-Se
p
25
-Se
p
27
-Se
p
29
-Se
p
1-O
ct
3-O
ct
5-O
ct
Completed Surveys, by Date
Isabel Survey link removed from NHC site
8© CFI Group
Who Are the Marine Respondents?Services Used
9%
13%
60%
71%
0% 10% 20% 30% 40% 50% 60% 70% 80%
Great LakesServices
Other Services
Marine andCoastal Services
Tropical Services
9© CFI Group
Who Are the Marine Respondents?Primary Operating Area
0%
2%
3%
7%
9%
16%
20%
42%
0% 10% 20% 30% 40% 50%
Alaska
Pacific
West Coast
Great Lakes
Other
Tropical Atlantic/Caribbean
Gulf of Mexico
Atlantic
10© CFI Group
0%
1%
1%
1%
1%
2%
4%
12%
15%
29%
34%
0% 5% 10% 15% 20% 25% 30% 35% 40%
Cruise ship
Freight transport
Passenger transport
Commercial fishing
Law enforcement
Search and rescue
Science/research
Beachgoer
Other
Coastal resident
Recreational Boater
Who Are the Marine Respondents?Primary Marine Activity
11© CFI Group
Who Are the Marine Respondents? Most-Used Products: Marine and Coastal
43%
51%
62%
70%
74%
78%
79%
85%
91%
91%
0% 20% 40% 60% 80% 100%
Surf Zone Forecasts
High Seas Forecasts
Offshore Waters Forecasts
Marine Weather Statements
Special Marine Warnings
Area Forecast Discusssions
Coastal Flood Watch/Warnings
Short Term Forecasts
Hazardous Weather Outlooks
Coastal Waters Forecasts
12© CFI Group
26
29
31
19
30
32
35
31
74
40
83
0 10 20 30 40 50 60 70 80 90 100
Inmarsat-C SafetyNET
NAVTEX
Radiofacsimile
Private vendor
Phone
NOAA Weather Wire, FOS, etc.
EMWIN
Local or cable TV/radio
U.S. Coast Guard broadcasts
Internet access
NOAA weather radio
Sea Land
18
18
20
32
31
41
45
68
42
90
69
0 10 20 30 40 50 60 70 80 90 100
Inmarsat-C SafetyNET
NAVTEX
Radiofacsimile
Private vendor
Phone
NOAA Weather Wire, FOS, etc.
EMWIN
Local or cable TV/radio
U.S. Coast Guard broadcasts
Internet access
NOAA weather radio
Sea Land
26
29
31
19
30
32
35
31
74
40
83
18
18
20
32
31
41
45
68
42
90
69
0 10 20 30 40 50 60 70 80 90 100
Inmarsat-C SafetyNET
NAVTEX
Radiofacsimile
Private vendor
Phone
NOAA Weather Wire, FOS, etc.
EMWIN
Local or cable TV/radio
U.S. Coast Guard broadcasts
Internet access
NOAA weather radio
Sea Land
Usefulness of Dissemination Methods
97% of respondents rate land methods
58% of respondents rate sea methods
13© CFI Group
CSI Comparison across Government Agencies
62
64
67
70
73
80
84
76
77
78
80
70
74
0 10 20 30 40 50 60 70 80 90
IRS
FAA
FDA
Export - Import Bank
Bureau of the Census
GSA FCIC
U.S. Mint - Collectors
NWS Media Personnel
NWS Aviation
NWS Marine
NWS Emergency Managers
Federal Gov't Aggregate
ACSI Overall
Customer Satisfaction Index (CSI) scores are based on three questions: • Overall satisfaction with products and services • Products and services compared to expectations • Products and services compared to ideal
14© CFI Group
Satisfaction ScoresServices Used
81
80
77
76
Other Services
Tropical Services
Marine and CoastalServices
Great Lakes Services
13% of respondents
60% of respondents
71% of respondents
9% of respondents
15© CFI Group
Satisfaction ScoresPrimary Operating Area
80
79
78
78
77
75
74
Tropical Atlantic/Caribbean
Other
Atlantic
Gulf of Mexico
Pacific
Great Lakes
West Coast
9% of resp.
3% of resp.
7% of resp.
2% of resp.
20% of resp.
42% of resp.
16% of resp.
16© CFI Group
Satisfaction ScoresPrimary Marine Activity
82
81
75
73
81
81
79
79
78
78
Law enforcement
Passenger transport
Coastal resident
Beachgoer
Science/research
Search and rescue
Freight transport
Other
Recreational boater
Commercial fishing
1% of resp.
34% of resp.
15% of resp.
1% of resp.
2% of resp.
4% of resp.
12% of resp.
29% of resp.
1% of resp.
1% of resp.
17© CFI Group
NWS Marine CSI Model
Confidence in NWS
Confidence in NWS
Take ActionTake Action
Contact NWSContact NWS
OUTPUTSOUTPUTSSATISFACTIONSATISFACTION
CustomerSatisfactionCustomer
Satisfaction
INPUTSINPUTS
Great Lakes Marine ForecastsGreat Lakes Marine Forecasts
Great Lakes Short-FusedAnd Other Forecasts
Great Lakes Short-FusedAnd Other Forecasts
Great Lakes Ice ForecastsGreat Lakes Ice Forecasts
NHC ForecastsNHC Forecasts
Tropical Warnings and WatchesTropical Warnings and Watches
Hurricane Local StatementsHurricane Local Statements
Coastal /Offshore/High Seas ForecastsCoastal /Offshore/High Seas Forecasts
Marine/Coastal Short-Fused and Other Forecasts
Marine/Coastal Short-Fused and Other Forecasts
Coastal Flood and Surf Zone InformationCoastal Flood and Surf Zone Information
Marine Forecast AccuracyMarine Forecast Accuracy
General Forecast PerformanceGeneral Forecast Performance
General Forecast AvailabilityGeneral Forecast Availability
General Forecast VerificationGeneral Forecast Verification
Quality of Ice Freeze-up OutlooksTimelines of Ice Break-up Outlooks
Usefulness of tropical forecast productsEase of understanding tropical
forecast products
Clarity and utility of hurricane and tropical storm watches and warnings
How well HLSs support mission needs in geographic area of interest
Accuracy of Marine WarningsOverall accuracy of the Marine Forecasts
NWS overallNWS vs. expectationsNWS vs. ideal
Contacted NWS to report problem or make suggestion
Likelihood to take action based on NWS information
Confidence in NWS
Quality of Short Term ForecastsTimelines of Short Term Forecasts
Quality of ForecastsTimelines of Forecasts
Quality of Short Term ForecastsTimelines of Short Term Forecasts
Quality of Area Forecast DiscussionsTimelines of Special Marine Warnings
Quality of ForecastsTimelines of Forecasts
Ease of understanding the data providedEase of reading charts and graphs
Timelines of warnings broadcast over NOAA Weather Radio
Frequency of forecast updates
Frequency NWS forecast events occur within six hours of predicted time frame
18© CFI Group
NWS Marine Model
Confidence in NWS
Confidence in NWS
Take ActionTake Action
Contact NWSContact NWS
INPUTSINPUTS SATISFACTIONSATISFACTION OUTPUTSOUTPUTS
Great Lakes Marine ForecastsGreat Lakes Marine Forecasts
Great Lakes Short-FusedAnd Other Forecasts
Great Lakes Short-FusedAnd Other Forecasts
Great Lakes Ice ForecastsGreat Lakes Ice Forecasts
NHC ForecastsNHC Forecasts
Tropical Warnings and WatchesTropical Warnings and Watches
CustomerSatisfactionCustomer
Satisfaction
Hurricane Local StatementsHurricane Local Statements
Coastal /Offshore/High Seas ForecastsCoastal /Offshore/High Seas Forecasts
Marine/Coastal Short-Fused and Other Forecasts
Marine/Coastal Short-Fused and Other Forecasts
Coastal Flood and Surf Zone InformationCoastal Flood and Surf Zone Information
Marine Forecast AccuracyMarine Forecast Accuracy
General Forecast PerformanceGeneral Forecast Performance
General Forecast AvailabilityGeneral Forecast Availability
General Forecast VerificationGeneral Forecast Verification
0.9
0.3
0.0
0.9
0.5
0.1
0.6
0.4
0.0
0.9
0.7
0.4
0.5
Impact: Measures leverage of component on target variable. If NHC Forecasts improves 5 points, CSI will improve 0.9 points.
76
78
63
85
85
80
76
79
78
74
78
76
69
78
13%
90
87
Score: Measures performance on a 0-100 scale.
19© CFI Group
Priority Matrix: Where to Focus Attention
Maintain Building on Success
Marine/Coastal Short-Fused and Other Forecasts 79/0.4
General Forecast Availability 76/0.4Great Lakes Short-Fused
and Other Forecasts 78/0.3Hurricane Local Statements 80/0.1
Coastal Flood and Surf Zone Information 78/0.0
Monitor
Top PriorityGreat Lakes Ice Forecasts 63/0.0
Marine Forecast Accuracy 74/0.9General Forecast Verification 69/0.5
General Forecast Performance 78/0.7Great Lakes Marine Forecasts 76/0.9
LOW Impact on Satisfaction (CSI) HIGH
NHC Forecasts 85/0.9
Coastal/Offshore/High Seas Forecasts 76/0.6Tropical Warnings and Watches 85/0.5
LO
W
Sco
re
H
IGH
20© CFI Group
Marine Forecast AccuracyImpact on CSI: 0.9
74
76
75
72
67
Marine ForecastAccuracy
Accuracy of MarineWarnings
Overall accuracy of theMarine forecasts
Accuracy of the windforecast
Accuracy of the waveforecast
21© CFI Group
NHC ForecastsImpact on CSI: 0.9
85
87
87
86
82
NHC Forecasts
Usefulness of theNWS/NHC tropicalforecast products
Overall quality ofNWS/NHC tropical
forecast products andservices
Improvements inNWS/NHC tropical
forecasting over the pastfive years
Ease of understandingthe NWS/NHC tropical
forecast products
• The usefulness of additional probabilistic information is rated an 86 – very useful• When asked which area needs improvement most, 51% of respondents say ‘Track’. No other
feature is mentioned more than 11% of the time.
22© CFI Group
General Forecast PerformanceImpact on CSI: 0.7
78
80
79
76
75
General ForecastPerformance
The ease of readingcharts and graphs
Ease of understandingthe data provided
Timeliness of responsesto your queries
Availability ofinformation to explain
the data
23© CFI Group
Coastal/Offshore/High Seas ForecastsImpact on CSI: 0.6
76
74
74
77
76
75
75
Coastal/Offshore/HighSeas Forecasts
Quality of CoastalWaters Forecasts
Timeliness of CoastalWaters Forecasts
Quality of High SeasForecasts
Timeliness of High SeasForecasts
Quality of OffshoreWaters Forecasts
Timeliness of OffshoreWaters Forecasts
24© CFI Group
Tropical Warnings and WatchesImpact on CSI: 0.5
85
85
Tropical Warnings andWatches
Clarity and utility of NWShurricane and tropical
storm watches andwarnings
• Most-mentioned minimum desired probability associated with a watch: 30%• Most-mentioned minimum desired probability associated with a warning: 70%• 50% of respondents prefer break points over county boundaries, and 28% are indifferent
25© CFI Group
General Forecast AvailabilityImpact on CSI: 0.4
76
81
72
General ForecastAvailability
The timeliness ofwarnings broadcast over
NOAA Weather Radio
The frequency offorecast updates
26© CFI Group
Marine and Coastal Short-Fused/OtherImpact on CSI: 0.4
79
78
78
78
78
77
76
82
81
79
79
Marine/Coastal Short-Fused and Other Forecasts
Quality of Special Marine Warnings
Quality of Hazardous Weather Outlooks
Quality of Marine Weather Statements
Timeliness of Special Marine Warnings
Quality of Area Forecast Discussions
Quality of Short Term Forecasts
Timeliness of Marine Weather Statements
Timeliness of Hazardous Weather Outlooks
Timeliness of Short Term Forecasts
Timeliness of Area Forecast Discussions
27© CFI Group
Coastal Flood and Surf Zone InfoImpact on CSI: 0.0
78
80
79
72
71
Coastal Flood and SurfZone Information
Quality of Coastal FloodWatch/Warnings
Timeliness of CoastalFlood Watch/Warnings
Timeliness of Surf ZoneForecasts
Quality of Surf ZoneForecasts
28© CFI Group
Other Aspects of NWS Performance
82
48
53
45
The use of Englishversus metric units
The availability of datain languages other than
English
Usefulness of 500milibar forecasts
Usefulness of waveperiod forecasts
29© CFI Group
What Customers Want
4%
30%
36%
60%
78%
0% 20% 40% 60% 80% 100%
Prefer other
Prefer gridded and/or vector forecasts
Prefer audio forecasts
Prefer text-based forecasts andwarnings services
Prefer graphical representation offorecasts and warnings services
• Customers express a clear preference for graphical forecasts.• A constantly-evolving forecast is preferred over a regularly-scheduled
forecast, 79% to 21%.
30© CFI Group
What Customers Want (cont.)
4%
25%
26%
46%
70%
72%
0% 20% 40% 60% 80%
Other
Graphical Waterspout Outlooks
Graphical Rip Current Outlooks
Graphical Probability-based WaveHeight Forecasts
Graphical Hurricane Local Statements
Graphical Hazardous Weather Outlooks
• When asked if they want any weather information not currently provided by NWS, 82% say no.
• Needs are varied, from better barometric readings to more accurate lightning forecasts to better wave forecasts.
31© CFI Group
Recommendations
• Keep up the good work! (e.g., Isabel forecasting)• Improve perceived forecast accuracy by increasing the number of
official forecasts • Give customers more information to explain the forecast data• Educate customer regarding reasonable expectations, especially
with wave height and period forecasts• As in the Media survey, improve the consistency of Area Forecast
Discussions and terminology use• Ensure storm track is a top priority for tropical/hurricane forecasts• Focus on graphical products, but keep in mind that radio is still a
dominant method of information acquisition• Examine possible causes of lower scores for Great Lakes Ice and
Coastal Surf Zone forecasts• Consider how different regions/users may respond to the survey
differently– These overall findings will have varying applicability to specific sub-
populations
32© CFI Group
Discussion
33© CFI Group
For More Information
Brian LaMarre, NOAAMarine Program Manager(301) 713-1677 [email protected]
Doug Helmreich, CFI GroupSenior Consultant(734) [email protected]
Bernie Lubran, Federal Consulting GroupProject Manager(202) [email protected]
