Post on 19-Jan-2016
transcript
Overview of the RHtests_dlyPrcp software package for
homogenization of daily precipitation
Xiaolan L. Wang and Y. Feng
EMS 2015, Sofia, Bulgaria, 7-11 September 2015
Climate Research Division, Science & Technology Branch, Environment Canada
FindU is for detecting unknown mean shifts
- in constant trend series with Gaussian IID or AR(1) errors (red noise);
- can be used without reference series
FindUD and StepSize are for testing the significance of documented shifts without a ref. series
Its sister package for Gaussian data – RHtestsV5 (Poster P75) - consists of six major functions:
FindU.wRef is for detecting unknown mean shifts
- in zero-trend series with Gaussian IID or AR(1) errors (red noise);
- for use with reference series
FindUD.wRef and StepSize.wRef are for testing the significance of documented shifts with a ref. series
These and other existing methods cannot be directly used to homogenize
daily precipitation data, because daily precipitation data are - non-continuous (precipitation does not occur every day at a fixed location) - non-negative (no negative values) - non-normally distributed, and - highly variable spatially, which makes it hardly possible to find/use a reference series
The amounts and occurrence frequencies
must be homogenized, separately.
It is wrong to adjust zero daily precip.
amounts for days of no precipitation
occurrence!
3
The RHtests_dlyPrcp software package is specifically designed for, and is the only
software that is suitable for homogenization of daily precipitation data:
- It finds the best Box-Cox transformation for the non-zero precipitation amounts in question, and
applies the transformation to bring the data close to a normal distribution.
- It applies the PMFred algorithm to the transformed data to test for unknown shifts (i.e., transPMFred),
and the F test, to test the significance of documented shifts
- It provides Quantile-Matching (QM) adjustments and IBC adjustments
for the determined shifts in the daily precipitation amounts
Details in Wang et al. 2010 (J. Appl. Meteor. Climatol., 49, 2416-2436)
“mean” adjustments based on
Inverse Box-Cox transformation
The Box-Cox transformation improves the detection power and lowers the FAR (Each case: generated 1000 series of N=1000 from a Log-Normal or Gamma distribution; see Wang et al. 2010)
Original Trans’d Original Trans’d
TPR3 0.078 0.051 0.067 0.044
PMF 0.070 0.055 0.071 0.047
False Alarm Rates (FARs): (nominal significance: 0.05)
Hit rates: ]10,10[ˆ ccc
Log-Normal data
transPMFred
transTPR3
transPMFred
transTPR3
transPMFredtransTPR3
PMFredTPR3
PMFTPR3
PMFredTPR3
TPR3 – a maximum F test used on a common trend two-phase regression model
for larger shifts
for small shifts
Power increases for Gamma distributed data
Log-Normal Gamma
Power increases for Log-Normal distributed data
The software RHtests_dlyPrcp consists of three functions; all of them available in GUI mode:
Parameters usedcurrently
must be changed to the code used for missing values in your data! You can change these default values to the values you want to use. Namely, you can choose (1) the significance level to conduct the test, (2) the segment to which to adjust the series, (3) the number of categories/points you want to use to estimate the probability distribution, (4) to use all or part of the data in a segment to estimate the QM adjustments
You can also choose to test onlyprecip. values that are greater thana chosen threshold, say 0.5 mm
Click FindU button to choose the precip. seriesto be tested. Then, click Ok to run the test. This will find significant Type-1 (unknown) changepoints, i.e., those that are significant even without metadata support
Click FindUD button to find potential Type-0 changepoints, namely,those that are significant only if they are supported by metadata. Skip this step if you don’t have metadata or only want to focus on Type-1 shifts
to adjust the data to the latest seg.(better to adjust to the highest seg.)
Click StepSize button to re-estimate the size and significance of shifts after you make any change in the list of changepoints, for example, add a documented shift, or change the date to a nearby documented date of change, or decide not to adjust a statistically detected shift…
Different stages of further adjustments four different daily precip. data series:
1. Not incl. trace amounts, no adjustment for joining (noT_naJ)
2. Not incl. trace amounts, adjusted for joining (noT_aJ)
3. Incl. trace amounts, no adjustment for joining (wT_naJ)
4. Incl. trace amounts, adjusted for joining (wT_aJ)
by Vincent & Mekis (2009), using
one rainfall ratio & one snowfall ratio
for all data in a segment
Daily precipitation recorded at The Pas (Manitoba, Canada) for Jun 1st, 1910 to Dec 31st, 2007
- snowfall water equivalent; rainfall adjusted for wetting loss and gauge undercatch
(Mekis & Hogg 1999; and updates by E. Mekis)
- joining of two stns at the end of 1945 (5052864 for up to 31 Dec. 1945, 5052880 1 Jan 1946 to 31 Dec. 2007)
Next, I’ll show you:
same three changepoints detectedSame two changepoints detected
Examples of application
All four series have a very significant
changepoint near the time of joining!
The ratio-based adjustments failed
to homogenize the data series!
Results for the two series not including trace amounts(noT series):
Type Date Documented date of change(s) 1 4 Jul 1938 9 Oct 1937 to 8 Aug 1938: changes in gauge type, rim
height, observing frequency; poor gauge condition reported on 9 Oct 1937
1 24 Oct 1946 31 Dec 1945: joining of two nearby stations (5052864 + 5052880)
1 4 Oct 1976 16 Oct 1975 to 18 Oct 1977: gauge type change (standard at 12” rim height to Type B at 16” rim height)
Reflect changes in the min. measurable amount (precision, unit) 1976-77
1945-46joining
1937-38
The transPMFred detected the same 3 changepoints.All of them are supported by metadata:
noT_naJ
-0.76 mm
1. noT_naJ (closest to original measurements):
2. noT_aJ (aJ changed the mean shift size from -0.76 mm to -0.73 mm)
The ratio-based adjustments for station joining failed to homogenize the series, because …
joining
The discontinuities are mainly in the measurements of small precipitation (P ≤ 3 mm), especially in the frequency of measured small precipitation: Series of daily P > 3 mm is homogeneous!
noT_naJ > 3mm
noT_naJ > 3mm
No P < 0.3 mmbefore 1977
(precision changed)
noT_naJ
Much fewer0.3 ~ 0.5 mmbefore 1946-joining point
0.21 mm from SWE
Much fewer< 3.0 mmbefore 1938
Good news for studying extreme precipitation
Any ratio-based adjustments for joining are not good in this case, because larger P are adjusted more than smaller P when they should not be adjusted at all!
noT_aJ
The above freq. discontinuities largely remain:
Type Date Documented date of change(s) 1 29 Jun 1931 early 1930s: MSC gauge intro’d; (new) 9 Oct 1931: noticed the need to relocate the gauge in order to collect a correct rainfall1 19 Nov 1945 31 Dec 1945: joining of two nearby
stations (5052864 + 5052880)
wT_naJ
Inclusion of trace amountsmakes these shifts disappeared!
transPMFred detected the same 2 changepoints, consistent with metadata:
3. wT_naJ 4. wT_aJ
Results for the two series including trace amounts
1976-771937-38
joining
joining
joining
An example of frequency discontinuity
The frequency of reported trace occurrence at The Pas is not homogeneous!
Adding a trace amount for T-flagged days is not good enough
Need to address the issue of frequency discontinuity!
Otherwise, adjustments could make the data deviate more from the truth!
noT_naJ
1945-46
1955-56
No trend
How to address the issue of frequency discontinuity?
- can use FindU or FindU.wRef in the RHtestsV5 package (poster P75) to test
the annual or monthly frequency series of some event, e.g., trace occurrence:
noT_naJ
1945-46
1955-56
FindU, or FindU.wRef with the long-term mean frequency (a constant value)as the reference series
Concluding remarks
should be chosen to reflect changes in measurement precision/unit
- Homogenization of daily precipitation data is very challenging Recommend: test series of dailyP > Pthr with different Pthr values (e.g., 0.0, 0.3 mm, 0.5 mm, 1.0 mm)
Shall aim to get better insight into the cause (metadata) and characteristics of discontinuity(e.g., freq.) before any attempt to adjust daily precipitation data – a non-continuous variable!
also check the frequency series of small P, becausesmall P are harder to measure with accuracy than larger P (larger %error) – discontinuities often exist in freq series of small P (e.g. P < 0.5 mm)
In the presence of frequency discontinuity, any adjustment derived from the measured daily P is not good. (e.g., ratio-based, IBC, Quantile-Matching) One must address the issue of freq. discontinuity first!
Thank you very much for your attention!
Questions/comments?
The package is available at http://etccdi.pacificclimate.org/software.shtml