WGSIP12 Miami - January 2009 Impacts verification - Andy Morse Preliminary report on verification scores for malaria and the climate drivers DEMETER to ENSEMBLES Andy Morse, Cyril Caminade and Anne Jones Department of Geography, University of Liverpool, Liverpool, United Kingdom WGSIP12 Miami - January 2009 Impacts verification - Andy Morse Running Order Introduction Malaria verification ENSEMBLES stream II limited area Climate verification ENSEMBLES stream II - all Africa Summary and Way Forward WGSIP12 Miami - January 2009 Impacts verification - Andy Morse Introduction Normally use BSS and ROCSS to work with probabilistic data Running RMSE, ACC and MSSS deterministically on ensemble mean Developed/modified scripts for skill scores in NCAR Command Language (NCL) share & cross validate? Work not complete will report via ENSEMBLES website WGSIP12 Miami - January 2009 Impacts verification - Andy Morse Introduction ROCSS malaria modelling Tier-2 malaria runs - ROC Skill Scores Above Median Event DEMETER driven LMM. Areas of high interannual variability were selected and persisted forecast skill was removed from the scores. Jones, A. and Morse, A. (2007) CLIVAR Exchanges, 43 May 4-6 JAS Nov 4-6 FMA WGSIP12 Miami - January 2009 Impacts verification - Andy Morse Validation of Liverpool Malaria Model (LMM) runs forced by ENSEMBLES stream2 seasonal forecasts WGSIP12 Miami - January 2009 Impacts verification - Andy Morse ENSEMBLES stream 2 LMM dynamic malaria mode forced with daily Tx and pptn. The domain: Botswana: [27.5S-17.5S] [17.5E-30E] Table of experiments: Expid nameProjectInstitution/ModelPeriod Start date (lead months) Ensemble members ecmfENSEMBLES ECMWF IFS- HOPE Nov 1 (14)9 egrrENSEMBLES UK Met-Office HadGem Nov 1 (14)9 lfpwENSEMBLES Meteo-France ARPEGE4/OPA Nov 1 (14)9 ingvENSEMBLES INGV ECHAM5/OPA Nov 1 (7)9 ifmkENSEMBLESKiel ECHAM5/OM Nov 1 (14)9 WGSIP12 Miami - January 2009 Impacts verification - Andy Morse ENSEMBLES stream 2 Scores: ACC: anomaly correlation coefficients Addresses the question: How well did the forecast anomalies correspond to the observed anomalies? RMSE: Root mean square errors Answers the question: What is the average magnitude of the forecast errors? MSSS: Mean square skill score with WGSIP12 Miami - January 2009 Impacts verification - Andy Morse ENSEMBLES stream 2 Results NDJF: RMSE for Malaria Incidence over Botswana (NDJF ). The reference dataset is a companion run forced by ERA40 reanalysis, and the analysis is applied to the ensemble means. WGSIP12 Miami - January 2009 Impacts verification - Andy Morse ENSEMBLES stream 2 Results NDJF: ACC for Malaria Incidence over Bostwana (NDJF ). The reference dataset is a companion run forced by ERA40 reanalysis, and the analysis is applied to the ensemble means. The dotted area display significant correlations at the 95% confidence interval WGSIP12 Miami - January 2009 Impacts verification - Andy Morse ENSEMBLES stream 2 Results NDJF: MSSS for Malaria Incidence over Botswana (NDJF ). The reference dataset is a companion run forced by ERA40 reanalysis, and the analysis is applied to the ensemble means. WGSIP12 Miami - January 2009 Impacts verification - Andy Morse ENSEMBLES stream 2 Results NDJF: RMSE for Malaria Incidence over Botswana (NDJF ). The reference dataset is a companion run forced by ERA40 reanalysis, and the analysis is applied to the ensemble means. WGSIP12 Miami - January 2009 Impacts verification - Andy Morse ENSEMBLES stream 2 Results NDJF: ACC for Malaria Incidence over Botswana (NDJF ). The reference dataset is a companion run forced by ERA40 reanalysis, and the analysis is applied to the ensemble means. The dotted area display significant correlations at the 95% confidence interval WGSIP12 Miami - January 2009 Impacts verification - Andy Morse ENSEMBLES stream 2 Results NDJF: MSSS for Malaria Incidence over Bostwana (NDJF ). The reference dataset is a companion run forced by ERA40 reanalysis, and the analysis is applied to the ensemble means. WGSIP12 Miami - January 2009 Impacts verification - Andy Morse Validation climate drivers ENSEMBLES stream seasonal forecasts WGSIP12 Miami - January 2009 Impacts verification - Andy Morse ENSEMBLES stream 2 Results NDJF: RMSE for tasmax (NDJF ). The reference dataset is a NCEP reanalysis, and the analysis is applied to the ensemble means. WGSIP12 Miami - January 2009 Impacts verification - Andy Morse ENSEMBLES stream 2 Results NDJF: ACC for tasmax (NDJF ). The reference dataset is a NCEP reanalysis, and the analysis is applied to the ensemble means. WGSIP12 Miami - January 2009 Impacts verification - Andy Morse ENSEMBLES stream 2 Results NDJF: MSSS for tasmax (NDJF ). The reference dataset is a NCEP reanalysis, and the analysis is applied to the ensemble means. WGSIP12 Miami - January 2009 Impacts verification - Andy Morse ENSEMBLES stream 2 Results NDJF: RMSE for prlr (NDJF ). The reference dataset is a GPCP reanalysis, and the analysis is applied to the ensemble means. WGSIP12 Miami - January 2009 Impacts verification - Andy Morse ENSEMBLES stream 2 Results NDJF: ACC for prlr (NDJF ). The reference dataset is a GPCP reanalysis, and the analysis is applied to the ensemble means. WGSIP12 Miami - January 2009 Impacts verification - Andy Morse ENSEMBLES stream 2 Results DJF: MSSS for prlr (NDJF ). The reference dataset is a GPCP reanalysis, and the analysis is applied to the ensemble means. WGSIP12 Miami - January 2009 Impacts verification - Andy Morse Summary & Way Forward? Experience in BSS and ROCSS (probabilistic) Question non probabilistic nature/use of RMSE, ACC and MSSS for EPS verification All data should come with sets of verification plots e.g. ECMWF DEMETERLibrary of scripts? Or portal to where they exsist WGSIP12 Miami - January 2009 Impacts verification - Andy Morse Questions? WGSIP12 Miami - January 2009 Impacts verification - Andy Morse Validation climate drivers ENSEMBLES stream seasonal forecasts also have for DEMETER equivalent years WGSIP12 Miami - January 2009 Impacts verification - Andy Morse ENSEMBLES stream 1 Results DJF: RMSE for tas (DJF ). The reference dataset is a ERA40 reanalysis, and the analysis is applied to the ensemble means. WGSIP12 Miami - January 2009 Impacts verification - Andy Morse ENSEMBLES stream 1 Results DJF: RMSE for tas (DJF ). The reference dataset is a NCEP reanalysis, and the analysis is applied to the ensemble means. WGSIP12 Miami - January 2009 Impacts verification - Andy Morse ENSEMBLES stream 1 Results DJF: ACC for tas (DJF ). The reference dataset is a ERA40 reanalysis, and the analysis is applied to the ensemble means. WGSIP12 Miami - January 2009 Impacts verification - Andy Morse ENSEMBLES stream 1 Results DJF: ACC for tas (DJF ). The reference dataset is a NCEP reanalysis, and the analysis is applied to the ensemble means. WGSIP12 Miami - January 2009 Impacts verification - Andy Morse ENSEMBLES stream 1 Results DJF: MSSS for tas (DJF ). The reference dataset is a ERA40 reanalysis, and the analysis is applied to the ensemble means. WGSIP12 Miami - January 2009 Impacts verification - Andy Morse ENSEMBLES stream 1 Results DJF: MSSS for tas (DJF ). The reference dataset is a NCEP reanalysis, and the analysis is applied to the ensemble means. WGSIP12 Miami - January 2009 Impacts verification - Andy Morse ENSEMBLES stream 1 Results DJF: RMSE for prlr (DJF ). The reference dataset is a GPCP reanalysis, and the analysis is applied to the ensemble means. WGSIP12 Miami - January 2009 Impacts verification - Andy Morse ENSEMBLES stream 1 Results DJF: ACC for prlr (DJF ). The reference dataset is a GPCP reanalysis, and the analysis is applied to the ensemble means. WGSIP12 Miami - January 2009 Impacts verification - Andy Morse ENSEMBLES stream 1 Results DJF: MSSS for prlr (DJF ). The reference dataset is a GPCP reanalysis, and the analysis is applied to the ensemble means. WGSIP12 Miami - January 2009 Impacts verification - Andy Morse ENSEMBLES stream 1 Results JAS: RMSE for prlr (JAS ). The reference dataset is a GPCP reanalysis, and the analysis is applied to the ensemble means. WGSIP12 Miami - January 2009 Impacts verification - Andy Morse ENSEMBLES stream 1 Results JAS: ACC for prlr (JAS ). The reference dataset is a GPCP reanalysis, and the analysis is applied to the ensemble means. WGSIP12 Miami - January 2009 Impacts verification - Andy Morse ENSEMBLES stream 1 Results JAS: MSSS for prlr (JAS ). The reference dataset is a GPCP reanalysis, and the analysis is applied to the ensemble means. WGSIP12 Miami - January 2009 Impacts verification - Andy Morse