HydrologicalHydrological--Driven Validation of Driven Validation of MPE Precipitation EstimatesMPE Precipitation Estimates
Emad Habib & Boone F. LarsonEmad Habib & Boone F. LarsonUniversity of Louisiana at LafayetteUniversity of Louisiana at Lafayette
Jeffrey Jeffrey GraschelGraschel Brian R. NelsonBrian R. NelsonLMRFCLMRFC NCDCNCDC
Joint Assembly, Fort Lauderdale, FloridaMay 2008
ObjectivesObjectives
•• To provide an independent evaluation of MPE To provide an independent evaluation of MPE (NWS(NWS--RFC) at hydrological relevant scalesRFC) at hydrological relevant scales
•• To assess implications of subTo assess implications of sub--pixel variability for pixel variability for MPE evaluationMPE evaluation
•• To gain insight on practical value of MPE To gain insight on practical value of MPE products for hydrological applicationsproducts for hydrological applications
Data SourcesData Sources
•• MPE products:MPE products:– NWS LMRFC products
• Stage III (MPE 2002) ----> Stage IV (NCEP)
• Rain gauge network–– IndependentIndependent–– High quality of dataHigh quality of data
–– High density within scale of MPE productHigh density within scale of MPE product
Rain Gauge Network in Lafayette, LARain Gauge Network in Lafayette, LA
4 Km4 Km
Annual precipitation = 55-60 inches
724 -200
724 -201
723 -201
Rain Gauge StationDischarge Gauge Station
723 -200Vincent Rd.
Gulf South
LaNeuville Rd.
Carriage Light Loop
Millcreek Rd.
STM
FenstermakerCommission Blvd
Lafayette Vineyard
Covenant Church
722 -201
ÊÚ
ÊÚ
0 100 200 Kilometers
KPOE
KLCH New Orleans
KPOE
KLCH New Orleans
~ 116 km from KLCH
• Period of study – 2004-2006
• Scales of interest:– 4x4 km2
– Hourly/ Daily/ Monthly• Daily / Monthly Analysis
– all six pixels• Hourly Analysis
– Two 4-gauge pixels
0.00
0.05
0.10
0.15
0.20
0.25
0 1 2 3 4 5Number of Gauges
VRF
Variance Reduction Factor
Monthly ComparisonsMonthly Comparisons
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec0
100
300
500Rainfall 2004
Dep
th (m
m)
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec0
100
200
300Rainfall 2005
Dep
th (m
m)
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec0
50
150
250Rainfall 2006
Dep
th (m
m)
GaugeMPE
2004
2005
2006
MP
E(m
m)
0 10 20 30 40 50 60 700
10
20
30
40
50
60
70HOURLY
0 40 80 120 160 2000
40
80
120
160
200DAILY
0 100 200 300 400 5000
100
200
300
400
500MONTHLY
2004
Gauge (mm) Gauge (mm) Gauge (mm)2006
MP
E(m
m)
0 10 20 30 40 50 60 70 800
10
20
30
40
50
60
70
80HOURLY
0 20 40 60 80 100 120 1400
20
40
60
80
100
120
140DAILY
0 40 80 120 160 200 240 2800
40
80
120
160
200
240
280MONTHLY
Gauge (mm) Gauge (mm) Gauge (mm)
MPE
(m
m)
MPE
(m
m)
Hourly Daily Monthly Hourly Daily Monthly
Relative Bias -0.07 0.01 0.02 -0.04 0.00 0.00
Relative RMSE 1.42 0.81 0.28 1.16 0.71 0.19
Correlation Coefficient 0.82 0.90 0.91 0.93 0.94 0.95
2004 2006
-0.4-0.2
00.20.40.60.8
11.21.4
0.13-0.30
0.30-0.50
0.50-0.80
0.80-1.25
1.25-2.0
2-4 4-7 7-10 > 10
Gauge Intensity (mm/h)
Rel
ativ
e B
ias
2004 2005 2006
0
1
2
34
5
6
7
8
0.13-0.30
0.30-0.50
0.50-0.80
0.80-1.25
1.25-2.0
2-4 4-7 7-10 > 10
Gauge Intensity (mm/h)
Rel
ativ
e R
MS
E
2004 2005 2006
Bias
RMSE
X (mm)
Pro
babi
lity
Rat
ioof
Mis
sed
Rai
nfal
l
0 1 2 3 4 5 6 7 80
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
0
0.02
0.04
0.06
0.08
200420052006
Categorical metricsCategorical metrics
2004 2005 2006
0.01 0.01 0.02
2004 2005 2006
6% (7.86 cm) 3% (7.93 cm) 4% (11.98 cm)
Total Rainfall Falsely Detected
False Alarm Ratio (FAR)
Probability of Detection
Volume of Missed Rain
SpatioSpatio--Temporal StructureTemporal Structure
Month
Cor
rela
tion
1 2 3 4 5 6 7 8 9 10 11 120
0.2
0.4
0.6
0.8
1
GaugeMPE
2004-2006
Month
Cor
rela
tion
1 2 3 4 5 6 7 8 9 10 11 12
0
0.2
0.4
0.6
GaugeMPE
2004-2006
Spatial Correlation: 4-km lag
Temporal Correlation:1-hour lag
Effect of subEffect of sub--pixel variabilitypixel variability*
**
* **
* **
**
** *
***
**
*
**
*
*
**
**
*
**
*
*
***
***
** *
**
* ** *
*
*
*
**
**
*
*
***
*
*
**
*
*
** *
*
**
** **
**
Distance (km)
Cor
rela
tion
1 2 3 4 5 6 7 8 90
0.2
0.4
0.6
0.8
1
CV <= 0.20.2 < CV <= 0.05CV > 0.5
*
Var(MPE-Gi)
Var(MPE-True)
Var(Gi-True)
2Cov[(MPE-True), (Gi-True)]
=
+
-
Var(MPE-G)/Var(MPE-T)
0
50
100
150
200
250
300
350
1 2 3 4 5 6 7 8Gauge
Varia
nce
Rat
io (%
)
CV<0.2 CV<0.2<0.5
CV>0.5 ALL
Var(MPE-G) / Var(MPE-True)
Hydrologic Application:Hydrologic Application:
724-200
724-201
723-201
Rain Gauge StationDischarge Gauge Station
723-200Vincent Rd.
Gulf South
LaNeuville Rd.
Carriage Light Loop
Millcreek Rd.
STM
FenstermakerCommission Blvd
Lafayette Vineyard
Covenant Church
722-201
•• GSSHA: PhysicallyGSSHA: Physically--based fully distributed based fully distributed hydrologic modeling hydrologic modeling system (Ogden and system (Ogden and Downer, 2002)Downer, 2002)
•• Model Setup: Model Setup: –– 22--d diffusive wave for overland flowd diffusive wave for overland flow–– 11--d explicit diffusive wave for channel flowd explicit diffusive wave for channel flow–– PenmanPenman--MonteithMonteith equation for ET equation for ET –– Green and Green and AmptAmpt infiltration with redistributioninfiltration with redistribution
Calibration/ValidationCalibration/Validation
Time in days since 10/1/2004 00:00
Dis
char
ge(c
ms)
25 30 35 40 45 5005
1015202530354045
MeasuredSimulated
20.7 Km
5.1 Km
0
5
10
15
20
25
30
35
40
45
2004.316 2004.317 2004.318 2004.319 2004.32 2004.321 2004.322 2004.323 2004.324 2004.325
Year
Q (C
MS)
Full Gauge NetworkObserved
0
5
10
15
20
25
30
35
40
45
2004.316 2004.317 2004.318 2004.319 2004.32 2004.321 2004.322 2004.323 2004.324 2004.325
Year
Q (C
MS)
Full Gauge NetworkObservedMPE
0
5
10
15
20
25
30
35
40
45
2004.316 2004.317 2004.318 2004.319 2004.32 2004.321 2004.322 2004.323 2004.324 2004.325
Year
Q (C
MS)
Full Gauge NetworkObservedMPEClose Gauge
0
5
10
15
20
25
30
35
40
45
2004.316 2004.317 2004.318 2004.319 2004.32 2004.321 2004.322 2004.323 2004.324 2004.325
Year
Q (C
MS)
Full Gauge NetworkFar GaugeObservedMPEClose Gauge
Period 1
Period 2
0
5
10
15
20
25
30
35
2004.332 2004.333 2004.334 2004.335 2004.336 2004.337 2004.338 2004.339 2004.34 2004.341
Year
Q (C
MS)
Full Gauge NetworkObserved
0
5
10
15
20
25
30
35
2004.332 2004.333 2004.334 2004.335 2004.336 2004.337 2004.338 2004.339 2004.34 2004.341
Year
Q (C
MS)
Full Gauge NetworkObservedMPE
0
5
10
15
20
25
30
35
2004.332 2004.333 2004.334 2004.335 2004.336 2004.337 2004.338 2004.339 2004.34 2004.341
Year
Q (C
MS)
Full Gauge NetworkObservedMPEClose Gauge
0
5
10
15
20
25
30
35
2004.332 2004.333 2004.334 2004.335 2004.336 2004.337 2004.338 2004.339 2004.34 2004.341
Year
Q (C
MS)
Full Gauge NetworkFar GaugeObservedMPEClose Gauge
Conclusions & Future WorkConclusions & Future Work
• Overall bias is minimal (2-7% at hourly scale; almost zero at daily/monthly scales)
• MPE has conditional bias: overestimation at low rainfall and underestimation at high rainfall
• MPE has conditional variance: variance decreases with increase of intensity
• high probability of detection (> 90%) except at very small intensities
• Low probability of false detection (1-2%) – results in 3-5% falsely detected rain
Conclusions & Future WorkConclusions & Future Work
• Relying on a single gauge for validation can cause overestimation of MPE errors by ~150%-200%
• For runoff purposes, MPE have noticeable value over typical rain gauge availability situations.
• Future work: use the same network and other similar research networks to look at other QPE products
AcknowledgementsAcknowledgements
•• Funding provided by:Funding provided by:
–– UCAR under sponsorship of NOAA/DOC as part of the UCAR under sponsorship of NOAA/DOC as part of the COMET Outreach Program.COMET Outreach Program.
–– Louisiana Board of Regents, Louisiana Board of Regents, BoRSFBoRSF, agreement , agreement NASA/LEQSF (2005NASA/LEQSF (2005--2010)2010)--LaSPACE and LaSPACE and NASA/NASA/LaSPACELaSPACE, grant NNG05GH22H, grant NNG05GH22H
–– University of Louisiana at LafayetteUniversity of Louisiana at Lafayette
Thank You!Thank You!
Continuous StatisticsContinuous Statistics
( )
RG
n
iRGiMPEi
Rn
RR
B
⎟⎠
⎞⎜⎝
⎛−
=
∑=1 ( ) ( )
RG
n
iMPERGRGiMPEi
R
RRRRnRMSE∑=
⎟⎠⎞⎜
⎝⎛ −−−
= 1
221
( )( )))((
),(2222
MPEMPERGRG
MPERGMPERGMPERG
RRRR
RRRRRR
−−
−=ρ
( )
( )2
1
2
1
2
11
⎟⎟⎠
⎞⎜⎜⎝
⎛−
⎟⎠
⎞⎜⎝
⎛−
−=
∑
∑
=
=
n
iRGRGi
n
iRGiMPEi
RRn
n
RR
E
Pearson Correlation
Normalized Root Mean Square Error
Efficiency
Normalized Bias
Conditional Validation based on Rainfall Conditional Validation based on Rainfall Magnitude and SeasonMagnitude and Season
•• Rainfall Magnitude conditioningRainfall Magnitude conditioning–– Define a relationship between gauge and MPE as Define a relationship between gauge and MPE as
a function of rainfall intensity. a function of rainfall intensity. –– 9 intensity intervals used (0.13 mm/hr 9 intensity intervals used (0.13 mm/hr →→ >10 >10
mm/hr)mm/hr)
•• Seasonal conditioningSeasonal conditioning–– Define any seasonal relationship between gauge Define any seasonal relationship between gauge
and MPE.and MPE.–– Hourly statistics plotted on a monthly basis.Hourly statistics plotted on a monthly basis.
Season-based Statistics
Average Gauge
Single Gauge
Average Gauge Single Gauge Average
GaugeSingle Gauge
Gauge Mean (mm)Normalized Bias 0.06 0.06 -0.04 -0.04 -0.20 -0.20Normalized RMSE 1.01 1.13 1.21 1.34 1.07 1.33Correlation Coefficient 0.92 0.90 0.85 0.81 0.87 0.81Efficiency 0.84 0.80 0.72 0.65 0.74 0.63
Non-uniform (Warm Months)
2.21 2.931.90
Highly Uniform (Cold Months)
Less Uniform (Transitional Months)
Analyzing SubAnalyzing Sub--pixel Rainfall Variabilitypixel Rainfall Variability
Single Gauge or Average GaugeSingle Gauge or Average Gauge
–– Stratified based on CVStratified based on CV•• CV CV ≤≤ 0.20.2•• 0.2 < CV 0.2 < CV ≤≤ 0.50.5•• CV > 0.5CV > 0.5
–– Stratified based on seasonStratified based on season•• Cold months (Dec Cold months (Dec –– Feb)Feb)•• Transitional months (Mar Transitional months (Mar –– May, Sep May, Sep -- Nov)Nov)•• Warm months (Jun Warm months (Jun –– Aug)Aug)
SelfSelf--correlation Statisticscorrelation Statistics
All selfAll self--correlation performed at hourly time scale.correlation performed at hourly time scale.
•• Spatial Spatial (G(G11 vs. Gvs. G22 / MPE/ MPE11 vs. MPEvs. MPE22))
–– Two 4Two 4--gauge pixelsgauge pixels–– Representative of 4 km.Representative of 4 km.
•• Temporal Temporal (G/MPE)(G/MPE)
–– One 4One 4--gauge pixelgauge pixel–– 1 hr time shift1 hr time shift
MPE Development at River Forecast MPE Development at River Forecast Center (RFC)Center (RFC)
--Precipitation Processing System (PPS)Precipitation Processing System (PPS)–– Reflectivity preprocessing (QC)Reflectivity preprocessing (QC)–– Conversion of (Z) to (R). Z=250RConversion of (Z) to (R). Z=250R1.21.2 , Z=300R, Z=300R1.41.4
--STAGE ISTAGE I ~ Digital Precipitation Array (DPA)~ Digital Precipitation Array (DPA)-- Hourly radarHourly radar--only product over (HRAP) gridonly product over (HRAP) grid
--Mean Field Bias CorrectionMean Field Bias Correction-- Rain gauges to correct multiplicative biasRain gauges to correct multiplicative bias
--STAGE IISTAGE II ~ mean field bias correcting of DPA~ mean field bias correcting of DPA
--STAGE IIISTAGE III ~ mosaicking Stage II over each RFC domain. ~ mosaicking Stage II over each RFC domain. (Replaced in 2002 (Replaced in 2002 with with MPEMPE))
--STAGE IVSTAGE IV ~ ~ creation of quilted national product of radar-based estimates by The National Center for Environmental Prediction (NCEP)
Cor
rela
tion
1 2 3 4 5 6 7 8 9 10 11 120.6
0.7
0.8
0.9
1
(d)
Nor
mal
ized
Bia
s1 2 3 4 5 6 7 8 9 10 11 12
-1
-0.5
0
0.5
1
1.5
2
(b)
Nor
mal
ized
RM
SE
1 2 3 4 5 6 7 8 9 10 11 12
1
2
3
4
5
6
(c)
Mea
nR
ainf
all(
mm
)
1 2 3 4 5 6 7 8 9 10 11 120
1
2
3
4
5
6200420052006
(a)
Month
Effi
cien
cy
1 2 3 4 5 6 7 8 9 10 11 12-1
-0.5
0
0.5
1
(e)
In next slides I break up plots to increase visibility
Nor
mal
ized
Bia
s
1 2 3 4 5 6 7 8 9 10 11 12-1
-0.5
0
0.5
1
1.5
2
Mea
nR
ainf
all(
mm
)
1 2 3 4 5 6 7 8 9 10 11 120
1
2
3
4
5
6200420052006
Month
Nor
mal
ized
RM
SE
1 2 3 4 5 6 7 8 9 10 11 12
1
2
3
4
5
6
Month
Effic
ienc
y
1 2 3 4 5 6 7 8 9 10 11 12-1
-0.5
0
0.5
1
Cor
rela
tion
1 2 3 4 5 6 7 8 9 10 11 120.6
0.7
0.8
0.9
1
200420052006
Var(MPE-T)'/Var(MPE-T)
0
20
40
60
80
100
120
140
160
180
200
1 2 3 4 5 6 7 8
Gauge
Varia
nce
Rat
io (%
)
CV<0.2 CV<0.2<0.5
CV>0.5 ALL
Var(MPE-True)’ / Var(MPE-True)
Period 4
0
5
10
15
20
25
30
2005.087 2005.088 2005.089 2005.09 2005.091 2005.092 2005.093 2005.094 2005.095 2005.096
Year
Q (
CM
S)
Full Gauge NetworkObserved
0
5
10
15
20
25
30
2005.087 2005.088 2005.089 2005.09 2005.091 2005.092 2005.093 2005.094 2005.095 2005.096
Year
Q (
CM
S)
Full Gauge NetworkObservedMPE
0
5
10
15
20
25
30
2005.087 2005.088 2005.089 2005.09 2005.091 2005.092 2005.093 2005.094 2005.095 2005.096
Year
Q (
CM
S)
Full Gauge NetworkObservedMPEClose Gauge
0
5
10
15
20
25
30
2005.087 2005.088 2005.089 2005.09 2005.091 2005.092 2005.093 2005.094 2005.095 2005.096
Year
Q (
CM
S)
Full Gauge NetworkFar GaugeObservedMPEClose Gauge
Validation Metrics Validation Metrics
•• PointPoint-- or gridor grid--based methodsbased methods
• Continuous metrics
• Categorical metrics
• Distribution-oriented metrics
• Spatial/Temporal Structure- metrics
• Intensity stratification
Rainfall (mm)
Cum
ulat
ive
Pro
babi
lity
10-1 100 101 1020
0.2
0.4
0.6
0.8
1
DA
ILY
Rainfall (mm)10-1 100 101 1020
0.2
0.4
0.6
0.8
1
Rainfall (mm)10-1 100 101 1020
0.2
0.4
0.6
0.8
1Rainfall (mm)
Cum
ulat
ive
Pro
babi
lity
10-1 100 1010
0.2
0.4
0.6
0.8
1
GaugeMPE
2004
HO
UR
LY
Rainfall (mm)10-1 100 1010
0.2
0.4
0.6
0.8
12005
Rainfall (mm)10-1 100 1010
0.2
0.4
0.6
0.8
12006
Cumulative Distribution Function
Var(G-T)/Var(MPE-G)
0
25
50
75
100
1 2 3 4 5 6 7 8Gauge
Varia
nce
Rat
io (%
)
CV<0.2 CV<0.2<0.5
CV>0.5 ALL
Var(G-True) / Var(MPE-G)