Date post: | 27-Mar-2015 |
Category: |
Documents |
Upload: | samantha-hale |
View: | 215 times |
Download: | 1 times |
Applied Hydrology
Assessing Hydrological Model Performance Using Stochastic Simulation
Professor Ke-Sheng Cheng
Department of Bioenvironmental Systems Engineering
National Taiwan University
INTRODUCTION
• Very often, in hydrology, the problems are not clearly understood for a meaningful analysis using physically-based methods.
• Rainfall-runoff modeling – Empirical models – regression, ANN– Conceptual models – Nash LR– Physical models – kinematic wave
04/10/23 2Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
• Regardless of which types of models are used, almost all models need to be calibrated using historical data.
• Model calibration encounters a range of uncertainties which stem from different sources including – data uncertainty, – parameter uncertainty, and – model structure uncertainty.
04/10/23 3Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
• The uncertainties involved in model calibration inevitably propagate to the model outputs.
• Performance of a hydrological model must be evaluated concerning the uncertainties in the model outputs.
04/10/23 4
Uncertainties in model performance evaluation.
Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
ASCE Task Committee, 1993
• “Although there have been a multitude of watershed and hydrologic models developed in the past several decades, there do not appear to be commonly accepted standards for evaluating the reliability of these models. There is a great need to define the criteria for evaluation of watershed models clearly so that potential users have a basis with which they can select the model best suited to their needs”.
• Unfortunately, almost two decades have passed and the above scientific quest remains valid.
04/10/23 5Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
SOME NATURES OF FLOOD FLOW FORECASTING
• Incomplete knowledge of the hydrological process under investigation.– Uncertainties in model parameters and model
structure when historical data are used for model calibration.
• It is often impossible to observe the process with adequate density and spatial resolution. – Due to our inability to observe and model the
spatiotemporal variations of hydrological variables, stochastic models are sought after for flow forecasting.
04/10/23 6Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
• A unique and important feature of the flow at watershed outlet is its persistence, particularly for the cases of large watersheds. – Even though the model input (rainfall) may exhibit
significant spatial and temporal variations, flow at the outlet is generally more persistent in time.
04/10/23 7Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
Illustration of persistence in flood flow series
04/10/23 8
A measure of persistence is defined as the cumulative impulse response (CIR).
1
1CIR
p
ii
1
t
p
iitit xx
10
Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
• The flow series have significantly higher persistence than the rainfall series.
• We have analyzed flow data at other locations including Hamburg, Iowa of the United States, and found similar high persistence in flow data series.
04/10/23 9Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
CRITERIA FOR MODEL PERFORMANCE EVALUATION
• Relative error (RE)• Mean absolute error (MAE) • Correlation coefficient (r) • Root-mean-squared error (RMSE) • Normalized Root-mean-squared error (NRMSE)
04/10/23 10
obs
RMSENRMSE
Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
• Coefficient of efficiency (CE) (Nash and Sutcliffe, 1970)
• Coefficient of persistence (CP) (Kitanidis and Bras, 1980)
• Error in peak flow (or stage) in percentages or absolute value (Ep)
04/10/23 11
n
tt
n
ttt
m QQ
SST
SSECE
1
2
1
2
)(
)ˆ(11
n
tktt
n
ttt
N QQ
SSE
SSECP
1
2
1
2
)(
)ˆ(11
Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
04/10/23 12Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
Coefficient of Efficiency (CE)• The coefficient of efficiency evaluates the
model performance with reference to the mean of the observed data.
• Its value can vary from 1, when there is a perfect fit, to . A negative CE value indicates that the model predictions are worse than predictions using a constant equal to the average of the observed data.
04/10/23 13
n
tt
n
ttt
m QQ
SST
SSECE
1
2
1
2
)(
)ˆ(11
Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
Model performance rating using CE (Moriasi et al., 2007)
• Moriasi et al. (2007) emphasized that the above performance rating are for a monthly time step. If the evaluation time step decreases (for example, daily or hourly time step), a less strict performance rating should be adopted.
04/10/23 14Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
Coefficient of Persistency (CP)• It focuses on the relationship of the performance of
the model under consideration and the performance of the naïve (or persistent) model which assumes a steady state over the forecast lead time.
• A small positive value of CP may imply occurrence of lagged prediction, whereas a negative CP value indicates that performance of the considered model is inferior to the naïve model.
04/10/23 15
1 CP
n
tktt
n
ttt
N QQ
SSE
SSECP
1
2
1
2
)(
)ˆ(11
Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
An example of river stage forcating
04/10/23 16
Model forecasting
CE=0.68466
ANN model
observation
Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
04/10/23 17
Model forecasting
CE=0.68466
CP= -0.3314
Naive forecasting
CE=0.76315ANN model
observation
Naïve model
Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
04/10/23 Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
18
Model forecasting
CE=0.94863
04/10/23 Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
19
Model forecasting
CE=0.94863
CP= -0.218
Naive forecasting
CE=0.95783
04/10/23 Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
20
Model forecasting
CE=0.90349
04/10/23 Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
21
Model forecasting
CE=0.90349
CP= -0.1875
Naive forecasting
CE=0.91873
Bench Coefficient• Seibert (2001) addressed the importance of
choosing an appropriate benchmark series with which the predicted series of the considered model is compared.
04/10/23 22
n
ttbt
n
ttt
bench
QQG
1
2,
1
2
)(
)ˆ(1
n
tktt
n
ttt
QQCP
1
2
1
2
)(
)ˆ(1
n
tt
n
ttt
QQCE
1
2
1
2
)(
)ˆ(1
Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
• The bench coefficient provides a general form for measures of goodness-of-fit based on benchmark comparisons.
• CE and CP are bench coefficients with respect to benchmark series of the constant mean series and the naïve-forecast series, respectively.
04/10/23 23Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
• The bottom line, however, is what should the appropriate benchmark series be for the kind of application (flood forecasting) under consideration.
• We propose to use the AR(1) or AR(2) model as the benchmark for flood forecasting model performance evaluation.
04/10/23 24
A CE-CP coupled MPE criterion.
Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
ASYMPTOTIC RELATIONSHIP BETWEEN CE AND CP
• Given a sample series { }, CE and CP respectively represent measures of model performance by choosing the constant mean series and the naïve forecast series as benchmark series.
• The sample series is associated with a lag-1 autocorrelation coefficient .
04/10/23 25
ntxt ,,2,1,
1
Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
04/10/23 26
[A]
Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
• Given a data series with a specific lag-1 autocorrelation coefficient, we can choose various models for one-step lead time forecasting of the given data series.
• Equation [A] indicates that, although the forecasting performance of these models may differ significantly, their corresponding (CE, CP) pairs will all fall on a specific line determined by .
04/10/23 27
1
Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
Asymptotic relationship between CE and CP for data series of various lag-1 autocorrelation coefficients.
04/10/23 28
6.01
Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
• The asymptotic CE-CP relationship can be used to determine whether a specific CE value, for example CE=0.55, can be considered as having acceptable accuracy.
• The CE-based model performance rating recommended by Moriasi et al. (2007) does not take into account the autocorrelation structure of the data series under investigation, and thus may result in misleading recommendations.
04/10/23 29Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
• Consider a data series with significant persistence or high lag-1 autocorrelation coefficient, say 0.8. Suppose that a forecasting model yields a CE value of 0.55 (see point C). With this CE value, performance of the model is considered satisfactory according to the performance rating recommended by Moriasi et al. (2007).
• However, it corresponds to a negative value of CP (-0.125), indicating that the model performs even poorer than the naïve forecasting, and thus should not be recommended.
04/10/23 30Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
Asymptotic relationship between CE and CP for data series of various lag-1 autocorrelation coefficients.
04/10/23 31Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
04/10/23 32
1= 0.843
CE=0.686 at CP=0
1= 0.822
CE=0.644 at CP=0
1= 0.908
CE=0.816 at CP=0
Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
• For these three events, the very simple naïve forecasting yields CE values of 0.686, 0.644, and 0.816 respectively, which are nearly in the range of good to vary good according to the rating of Moriasi et al. (2007).
04/10/23 33Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
• In the literature we have found that many flow forecasting applications resulted in CE values varying between 0.65 and 0.85. With presence of high persistence in flow data series, it is likely that not all these models performed better than naïve forecasting.
04/10/23 34Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
A nearly perfect forecasting model
04/10/23 35
0
200
400
600
800
1000
1200
1400
1600
1 15 29 43 57 71 85 99 113 127 141 155 169 183 197 211 225 239 253 267 281 295 309 323 337 351 365 379 393 407 421
CE=0.85599
CE=0.79021
CE=0.66646
CE=0.79109
CE=0.80027
CE=0.62629
CE=0.77926
CE=0.76404
CE=0.84652
Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
A CE-CP COUPLED MPE CRITERION
• Are we satisfied with using the constant mean series or naïve forecasting as benchmark?
• Considering the high persistence nature in flow data series, we argue that performance of the autoregressive model AR(p) should be considered as a benchmark comparison for performance of other flow forecasting models.
04/10/23 36Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
• From our previous experience in flood flow analysis and forecasting, we propose to use AR(1) or AR(2) model for benchmark comparison.
04/10/23 37Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
• The asymptotic relationship between CE and CP indicates that when different forecasting models are applied to a given data series (with a specific value of 1, say *), the resultant (CE, CP) pairs will all fall on a line determined by Eq. [A] with 1= * .
04/10/23 38Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
• In other words, points on the asymptotic line determined by 1= * represent forecasting performance of different models which are applied to the given data series.
• Using the AR(1) or AR(2) model as the benchmark, we need to know which point on the asymptotic line corresponds to the AR(1) or AR(2) model.
04/10/23 39Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
CE-CP relationships for AR(1) model
• AR(1)
04/10/23 40
144 2 CPCPCE [B]Lab for Remote Sensing Hydrology and Spatial Modeling
Dept of Bioenvironmental Systems Eng, NTU
CE-CP relationships for AR(1) and AR(2) models
• AR(2)
04/10/23 41
31
4
1
84
1
422
22
222
CPCPCE [C]
tttt XXX 2211
Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
Example of event-1
04/10/23 42
AR(1) model
AR(2) model
Data AR(2) modeling
Data AR(1) modeling
1=0.843
Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
Assessing uncertainties in (CE, CP) using modeled-based bootstrap resampling
04/10/23 43Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
Assessing uncertainties in MPE by bootstrap resampling (Event-1)
04/10/23 44Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
Assessing uncertainties in MPE by bootstrap resampling (Event-1)
04/10/23 45Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
Conclusions• Performance of a flow forecasting model needs
to be evaluated by taking into account the uncertainties in model performance.
• AR(2) model should be considered as the benchmark.
• Bootstrap resampling can be helpful in evaluating the uncertainties in model performance.
04/10/23 46Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU
• Seibert (2001)“Obviously there is the risk of discouraging results when a model does not outperform some simpler way to obtain a runoff series. But if we truly wish to assess the worth of models, we must take such risks. Ignorance is no defense.”
04/10/23 47Lab for Remote Sensing Hydrology and Spatial Modeling Dept of Bioenvironmental Systems Eng, NTU