Climate Modeling LaboratoryMEASNC State University An Extended Procedure for Implementing the...

Climate Modeling LaboratoryMEASNC State University

An Extended Procedure for Implementing the

Relative Operating Characteristic

Graphical MethodRobert J. MeraMarch 6, 2008

Marine, Earth and Atmospheric SciencesNorth Carolina State University


Outline• Objective and Motivation• Principles of Ensemble Forecasting• Traditional Relative Operating Characteristic

(ROC) and Economic Value (EV) analysis• Extended Relative Operating Characteristic

(EROC)• Conclusions • Applications and Future Work


MotivationClimate prediction is becomingincreasingly important for different sectors of the economy worldwide

Courtesy: El UniversoA case of false alarm/miss


Ensemble forecasting

• Ensemble Prediction System (EPS) forecasting is a method used to account for uncertainties and errors in the forecasting system (recall Chaos theory)

• Through the ensemble approach one can generate probabilistic forecasts for assessing a future event such as excessive rains, droughts, etc.


Contingency Matrix

No Yes

No No cost ()

Miss()

Yes False Alarm()

Hit()

Observations

EP

S F

orec

ast

Hit Rate: H= δ/( +δ)False Alarm Rate: F= /( + )

• A decision maker becomes a user of weather forecasts if he/she alters his/her actions based on forecast information

• A cost-loss analysis can be assessed based on a 2x2 matrix in which we evaluate the skill of a probabilistic forecast


Ensemble numbers

• In our contingency matrix we compute the hit rate and false alarm for an array of ensemble member groups

• Example: 15 ensembles used– Hit rate and false alarm calculated for

only 1 out of 15, 2/15 . . . n/N• We can use this information to

analyze the skill of a model


The Relative Operating Characteristic (ROC)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.2 0.4 0.6 0.8 1False Alarm Rate

Hit

Rat

e

• The ROC method is widely used for estimating the skill of ensemble prediction systems (EPS) (Marzban, 2004)

• The closer a curve is to the upper-left-hand corner, the more skillful the forecast system is

• A perfect forecast system would have a ROC area (ROCA) of 1

• A system with no capability of distinguishing in advance between different climate events has a score of 0.5, i.e. lying on the diagonal defined by (0,0) and (1,1)

Using the information in our contingency matrix we can compute ROC

Each point on the curve represents a group of ensembles (1/15, 2/15, etc.)


• The ultimate utility of climate forecasts is economic and other benefits associated with their actual use in the daily decision-making process of individuals and different organizations

• Simplistically, users of climate forecasts either DO or DO NOT take action, but the relative value of the forecasts varies with model performance (i.e. hit rate, false alarm)

Utility of Climate Predictions


0

0.15

0.3

0 0.33 0.66C/L

Re

lati

ve

Va

lue

Economic value (EV) graph based on the ROC graph

• The economic value (EV) graphical method provides a measure of the EPS performance in relative economic terms for a specific hypothetical range of end users (C/L,) varying from 0 to 1 (Richardson 2000a,b)

• EV computes the relative economic value using the hit rate and false alarm rate, where a value of 1 translates to a perfect forecast.

For the sake of brevity, we will not discuss the mathematics involved.


EROC Procedure• Based on the ROC skill alone it is not possible

to determine if a useful level of skill has been achieved for a specific end user.

• The EV method is more cumbersome to use and its approach could result in some oversimplification of the actual situation for many users because in reality they may have an infinite range of available mitigation options. Also, EV does not provide certain features of ROC that help to diagnose specific characteristics of EPS.

• Our goal was to develop an alternative procedure similar to the traditional implementation of the ROC graphical method, but one that also provides evaluation for a specific end user.


ROC and EV relationship

• ROC measures skill only for the C/L (μ) with the maximum value (Vopt)

• This effectively ignores any other user

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1


Hit

Rat

e

0

0.15

0.3

0 0.33 0.66C/L

Re

lati

ve

Va

lue


EROC• Specific users are more interested in the economic

value related their own mitigation options• EROC allows us to build different base lines for

different users

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1


Hit

Rat

e

Base line for μ=ō(=0.33), Vmin=0

ROC

Base line for μ=0.25(<ō), Vmin=0

Base line for μ=0.40(>ō), Vmin=0

0

0.15

0.3

0 0.33 0.66C/L

Re

lati

ve

Va

lue

μ= 0.25μ= 0.40

μ= V(opt)

Two users


0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1


Hit

Ra

te


ROC



0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1


Hit

Ra

te

Base line for μ=ō(=0.33), Vmin=0ROCBase line for μ=0.25(<ō), Vmin=0.1Base line for μ=0.40(>ō), Vmin=0.1

minV

0

0.15

0.3

0 0.33 0.66C/L

Re

lati

ve

Va

lue

0

0.15

0.3

0 0.33 0.66C/L

Re

lati

ve

Va

lue

What if the end-user decides to use a forecast only if the Vmin is at a certain value?EROC allows for this.Initial: Vmin=0

Initial: Vmin=0.1

Notice the shift


0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1


Hit

Rat

e


ROC



Additional advantage• Each curve in the EV plot represents a particular group of

ensembles on the ROC plot (i.e. a point on the ROC curve)• EROC preserves ROC’s ability to diagnose each ensemble

group’s skill and its relative value for a specific user• This translates into a possibility of using a smaller number

of ensemble members, say 3 or 4 model runs instead of 15 (i.e. a less expensive forecast)

0

0.15

0.3

0 0.33 0.66C/L

Re

lati

ve

Va

lue


Conclusions• An extended ROC (EROC) procedure has

been developed from the traditional ROC and the EV graphical methods used for evaluating the performance of ensemble climate/weather prediction systems

• In the proposed EROC approach we recommend construction of user-specific baselines that provide us with an analysis of both skill and value of an EPS forecast that is tailor-made for a specific user

• EROC allows for a clearer picture of minimum value and an ensemble group’s skill for a particular user


Applications and Future Work

• Implementation of a routine to calculate EROC and EV plots to the CML R statistical library (currently in its final testing stage)

• You can view the progress on http://climlab.meas.ncsu.edu/erocroutine.html

• An ongoing project in partnership with NCAR using WRF-DART simulations output for EROC implementation


Acknowledgements

Professor Fred SemazziNeil Davis

Matt NormanRichard Anyah

NCSU CLIMLABhttp://climlab.meas.ncsu.edu

An Extended Procedure for Implementing the Relative Operating Characteristic Graphical MethodFredrick H. M. Semazzi, and Roberto J. MeraJournal of Applied Meteorology and Climatology, Semptember 2006


Questions?



Criteria for Issuing a forecast

Decision to issue a forecast of an event (E) to occur is probabilistically based on the criteria:

p nN pt

Where:(N): size of the ensemble(n): number of the runs in the ensemble for which (E) actually occurs(p): probability given by the ratio (n/N)

This is the threshold fraction above which the event (E) is predicted to occur based on the model forecast

Date post:	26-Mar-2015
Category:	Documents
Upload:	andrew-zimmerman
View:	213 times
Download:	0 times

Climate Modeling LaboratoryMEASNC State University An Extended Procedure for Implementing the...

Documents