Model Performance Evaluation Dealing with uncertainties in hydrologic modeling and forecasting Prof. Ke-Sheng Cheng Department of Bioenvironmental Systems Engineering National Taiwan University INTRODUCTION Rainfall-runoff modeling Empirical models regression, ANN Conceptual models Nash LR Physical models kinematic wave Regardless of which types of models are used, almost all models need to be calibrated using historical data. 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY Model calibration encounters a range of uncertainties which stem from different sources including data uncertainty, parameter uncertainty, and Model structure uncertainty. Very often, in hydrology, the problems are not clearly understood or cannot be fully explained by the models. All sources of uncertainties are involved. 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY The uncertainties involved in model calibration inevitably propagate to the model outputs. Model performance needs to be evaluated concerning the uncertainties in the model outputs. 7/14/ MPE needs to address uncertainties in model performance evaluation itself. 10th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY SOME NATURES OF REAL-TIME FLOOD FORECASTING Incomplete knowledge of the hydrological process under investigation. Uncertainties in model parameters and model structure when historical data are used for model calibration. It is often impossible to observe the process with adequate density and spatial resolution. Due to our inability to observe and model the spatiotemporal variations of hydrological variables, stochastic models are sought after for flow forecasting. 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY A unique and important feature of the flow at watershed outlet is its persistence, particularly for the cases of large watersheds. Even though the model input (rainfall) may exhibit significant spatial and temporal variations, flow at the outlet is generally more persistent in time. 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY Illustration of persistence in flood flow series 7/14/ A measure of persistence is defined as the cumulative impulse response (CIR). 10th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY The flow series have significantly higher persistence than the rainfall series. We analyzed flow data at other locations including Hamburg, Iowa of the United States, and found similar high persistence in flow data series. 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY CRITERIA FOR MODEL PERFORMANCE EVALUATION Relative error (RE) Mean absolute error (MAE) Correlation coefficient (r) Root-mean-squared error (RMSE) Normalized Root-mean-squared error (NRMSE) 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY Coefficient of efficiency (CE) (Nash and Sutcliffe, 1970) Coefficient of persistence (CP) (Kitanidis and Bras, 1980) Error in peak flow (or stage) in percentages or absolute value (Ep) 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY CE is widely applied for MPE. In this study we focus on MPE using CE and CP. Coefficient of Efficiency (CE) The coefficient of efficiency evaluates the model performance with reference to the mean of the observed data series. Its value can vary from 1, when there is a perfect fit, to. A negative CE value indicates that the model predictions are worse than predictions using a constant equal to the average of the observed data. 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY Model performance rating using CE (Moriasi et al., 2007) Moriasi et al. (2007) emphasized that the above performance rating are for a monthly time step. If the evaluation time step decreases (for example, daily or hourly time step), a less strict performance rating should be adopted. 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY Coefficient of Persistency (CP) It focuses on the relationship of the performance of the model under consideration and the performance of the nave (or persistent) model which assumes a steady state over the forecast lead time. A small positive value of CP may imply occurrence of lagged prediction, whereas a negative CP value indicates that performance of the considered model is inferior to the nave model. 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY Demonstration of uncertainties in modeling 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY 15 Illustrative diagram showing the process of (1) parameter estimation, (2) forecasting, (3) MPE criteria calculation, and (4) uncertainty assessment of MPE criteria. 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY 16 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY 17 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY 18 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY 19 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY 20 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY 21 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY 22 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY 23 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY 24 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY 25 Figures intended to show how well predictions agree with observations often only provide limited information because long series of predicted data are squeezed in and lines for observed and predicted data are not easily distinguishable. Not all authors provide numerical information, but only state that the model was in good agreement with the observations (Seibert, 1999). Examples of flood stage forcasting 7/14/ Model forecasting CE= ANN model observation 10th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY 7/14/ Model forecasting CE= CP= Naive forecasting CE= ANN model observation Nave model 10th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY 28 Model forecasting CE= 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY 29 Model forecasting CE= CP= Naive forecasting CE= 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY 30 Model forecasting CE= 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY 31 Model forecasting CE= CP= Naive forecasting CE= Examples of flood flow forecasting 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY 32 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY 33 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY 34 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY 35 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY 36 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY 37 Bench Coefficient Seibert (2001) addressed the importance of choosing an appropriate benchmark series with which the predicted series of the considered model is compared. 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY The bench coefficient provides a general form for measures of goodness-of-fit based on benchmark comparisons. CE and CP are bench coefficients with respect to benchmark series of the constant mean series and the nave-forecast series, respectively. 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY The bottom line, however, is what should the appropriate benchmark series be for the kind of application (flood forecasting) under consideration. Taking the flow persistence into account, we propose to use the AR(1) or AR(2) model as the benchmark for flood forecasting model performance evaluation. 7/14/ A CE-CP coupled MPE criterion. 10th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY ASYMPTOTIC RELATIONSHIP BETWEEN CE AND CP Given a sample series { }, CE and CP respectively represent measures of model performance by choosing the constant mean series and the nave forecast series as benchmark series. The sample series is associated with a lag-1 autocorrelation coefficient. 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY 7/14/ [A] 10th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY Given a data series with a specific lag-1 autocorrelation coefficient, we can choose various models for one-step lead time forecasting of the given data series. Equation [A] indicates that, although the forecasting performance of these models may differ significantly, their corresponding (CE, CP) pairs will all fall on a specific line determined by. 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY Asymptotic relationship between CE and CP for data series of various lag-1 autocorrelation coefficients. 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY The asymptotic CE-CP relationship can be used to determine whether a specific CE value, for example CE=0.55, can be considered as having acceptable accuracy. The CE-based model performance rating recommended by Moriasi et al. (2007) does not take into account the autocorrelation structure of the data series under investigation, and thus may result in misleading recommendations. 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY Consider a data series with significant persistence or high lag-1 autocorrelation coefficient, say 0.8. Suppose that a forecasting model yields a CE value of 0.55 (see point C). With this CE value, performance of the model is considered satisfactory according to the performance rating recommended by Moriasi et al. (2007). However, it corresponds to a negative value of CP (-0.125), indicating that the model performs even poorer than the nave forecasting, and thus should not be recommended. 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY Asymptotic relationship between CE and CP for data series of various lag-1 autocorrelation coefficients. 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY 7/14/ 1 = CE=0.686 at CP=0 1 = CE=0.644 at CP=0 1 = CE=0.816 at CP=0 10th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY For these three events, the very simple nave forecasting yields CE values of 0.686, 0.644, and respectively, which are nearly in the range of good to vary good according to the rating of Moriasi et al. (2007). 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY In the literature we have found that many flow forecasting applications resulted in CE values varying between 0.65 and With presence of high persistence in flow data series, it is likely that not all these models performed better than nave forecasting. 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY Misuse of CE and CP for MPE 7/14/ Artifactual series CE= Individual events CE= CE= CE= CE= CE= CE= CE= CE= th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY 52 Individual events CP= CP= CP= CP= CP= Artifactual series CP= Misuse of CE and CP for MPE A CE-CP COUPLED MPE CRITERION Are we satisfied with using the constant mean series or nave forecasting as the benchmark? Considering the high persistence nature in flow data series, we argue that performance of the autoregressive model AR(p) should be considered as a benchmark comparison for performance of other flow forecasting models. 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY From our previous experience in flood flow analysis and forecasting, we propose to use AR(1) or AR(2) model for benchmark comparison. 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY The asymptotic relationship between CE and CP indicates that when different forecasting models are applied to a given data series (with a specific value of 1, say *), the resultant (CE, CP) pairs will all fall on a line determined by Eq. [A] with 1 = *. 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY In other words, points on the asymptotic line determined by 1 = * represent forecasting performance of different models which are applied to the given data series. Using the AR(1) or AR(2) model as the benchmark, we need to know which point on the asymptotic line corresponds to the AR(1) or AR(2) model. 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY CE-CP relationships for AR(1) model AR(1) 7/14/ [B] 10th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY CE-CP relationships for AR(1) and AR(2) models AR(2) 7/14/ [C] 10th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY Example of event-1 7/14/ AR(1) model AR(2) model Data AR(2) modeling Data AR(1) modeling 1 = th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY Assessing uncertainties in (CE, CP) using modeled-based bootstrap resampling 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY Assessing uncertainties in MPE by bootstrap resampling (Event-1) 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY Assessing uncertainties in MPE by bootstrap resampling (Event-1) 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY 63 Seibert (2001) Obviously there is the risk of discouraging results when a model does not outperform some simpler way to obtain a runoff series. But if we truly wish to assess the worth of models, we must take such risks. Ignorance is no defense. 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY ASCE Task Committee, 1993 Although there have been a multitude of watershed and hydrologic models developed in the past several decades, there do not appear to be commonly accepted standards for evaluating the reliability of these models. There is a great need to define the criteria for evaluation of watershed models clearly so that potential users have a basis with which they can select the model best suited to their needs. Unfortunately, almost two decades have passed and the above scientific quest remains valid. 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY Conclusions Performance of a flow forecasting model needs to be evaluated by taking into account the uncertainties in model performance. AR(2) model should be considered as the benchmark. Bootstrap resampling can be helpful in evaluating the uncertainties in model performance. 7/14/ th International Conference on Hydroinformatics HIC 2012, Hamburg, GERMANY