Slide 1

XXII Olympic Winter GamesFeb - Mar 2014Pertti Nurmi

WMO WWRP JWGFVR aka Joint Working Group on Forecast Verification ResearchAcknowledgements : FROST-2014 Expert Team + Evgeny Atlaskin, Sigbritt Nsmanpertti.nurmi@fmi.fiForecast Verification Activities ofSochi 2014 Winter Olympics( FROST-2014 ~ Forecast and Research in the Olympic Sochi Testbed )

#FROST-2014 Verification activitiespertti.nurmi@fmi.fiFinnish Meteorological Institute

XXII Olympic Winter GamesFeb - Mar 2014

Finnish Meteorological Institute

SochiFinnish Meteorological Institute#FROST-2014 Verification activitiespertti.nurmi@fmi.fiFinnish Meteorological Institute

XXII Olympic Winter GamesFeb - Mar 2014

Finnish Meteorological Institute

Finnish Meteorological Institute#FROST-2014 Verification activitiespertti.nurmi@fmi.fiFinnish Meteorological Institute

XXII Olympic Winter GamesFeb - Mar 2014

Finnish Meteorological Institute

Finnish Meteorological Institute#FROST-2014 Verification activitiespertti.nurmi@fmi.fiFinnish Meteorological Institute

XXII Olympic Winter GamesFeb - Mar 2014

Finnish Meteorological Institute

Observation stationsRoshydromet - 6 Avalanche-protection service - 5Sport venues - 12Olympic sport venues

LauraRusSki GorkiRosa KhutorSankiFinnish Meteorological Institute#FROST-2014 Verification activitiespertti.nurmi@fmi.fiFinnish Meteorological InstituteTHE issue on FROST-2014 verification

localpointregional( global )Spatial scaleForecast aggregation timeminuteshoursdaysNWPnowcastsveryshortrangeDeterministic + EPSWide range of scales, models *, methods, variables, observations Difficult to define a general verification setup to cover everything, systematically Including end-products to customers athletes Impacts

XXII Olympic Winter GamesFeb - Mar 2014

Finnish Meteorological Institute* models aka forecast contributorsFinnish Meteorological Institute#FROST-2014 Verification activitiespertti.nurmi@fmi.fiFinnish Meteorological Institute

Issues to consider / take into account #1All FROST-2014 contributors verify their own output General practice ! BUT can be biased NOT sufficient / adequateTherefore, also centralized (HMC Russia) AND external verification to validate verification resultsTwo complementary approaches (like Vancouver Olympics SNOW-V10)High temporal resolution Point Verification User needs : FDP*Users require thresholds for their decisions Sports activities and events take place at, and along, pointsOfficial (final) End-Products as well as objective direct model outputResearch-oriented verification, Detailed model comparisons : RDP*XXII Olympic Winter GamesFeb - Mar 2014* FDP = Forecast Demonstration Project / RDP = Research Demonstration Project (WMO terminology)#FROST-2014 Verification activitiespertti.nurmi@fmi.fiFinnish Meteorological Institute

Issues to consider / take into account #2Details of all available models/systems + observationsForecast lead times and projectionsEnhanced observation network with N (?) stationsHi-fi site-specific (point) observations highly importantOn-line forecast production from all contributorsCommon data base structure - ftp site - for storageFreeze the FDP system at certain stageXXII Olympic Winter GamesFeb - Mar 2014Need to freeze?When to freeze? No changes are allowed in the last 3-4 months before the Olympics Standardized formats for fc & observation storage !(Close to) Real-time verification soon as data(sets) become availableRetrospective verification also Comprehensive post-event verification possible only after all fc/obs archived and QCd Two differing results (?)Analyses used as additional truth ? Pitfalls, independence issueUsefulness applicability - availability of model independent analyses( e.g. INCA )#FROST-2014 Verification activitiespertti.nurmi@fmi.fiFinnish Meteorological Institute

Issues to consider / take into account #3Pointwise verification of all forecasts, tuned at all forecast = decision pointsBaseline quality from a coarse model as referenceVarious hi-res models from contributors at various (time/space) resolutionsStatistical adaptations, if there are any (?)Official End-Products by the Olympics forecaster team !Inter-comparisons between, and added value, (a) to (d) IMPACT analysisDefine each sports site/event relevant observing station(s) and forecast variables and thresholds Underway !Use of verification software packages R, MET, etc MET, adapted for FROST-2014 (COSMO-RU forecasts)Roshydromet : Internal RDP verification (A. Muravev)FMI : run own tuned verification system locally (more on this later)Input data (fc + obs) via common ftp siteXXII Olympic Winter GamesFeb - Mar 2014#FROST-2014 Verification activitiespertti.nurmi@fmi.fiFinnish Meteorological Institute

Issues to consider / take into account #4Formal evaluation period : 15 January 15 March (2 months)High-impact What is hi-impact ? ThresholdsSample size issue (statistical inference; confidence intervals)Timing (error) of events onset, duration, cessationStratification vs. Aggregation Need both !New categorical verification measures : SEDS, SEDI Spatial verification (CRA, SAL, MODE, FSS ) probably NOTApplicability/availability of remotely sensed data Prob. too big efforts !Case studies Hard to generalize results fall under RDPSpeciality : FMI Road Weather forecast verification Next slideUnder externally funded CoMoSeF projectXXII Olympic Winter GamesFeb - Mar 2014#FROST-2014 Verification activitiespertti.nurmi@fmi.fiFinnish Meteorological Institute

Sochi RWM : First model run 24.8.13

#FROST-2014 Verification activitiespertti.nurmi@fmi.fiFinnish Meteorological Institute11

FROST-2014 : Weather parameter categories & thresholdsFROST-2014 Concept paperFinal definition of events vs. categoriesAssure availability of deterministic products, contributorsAssure availability of EPS, prob. products, contributorsIdentification of users, decision-makersIdentification of FDP vs. RDP verification activitiesIdentification of verification measures / techniques !Finalization underwayXXII Olympic Winter GamesFeb - Mar 2014#FROST-2014 Verification activitiespertti.nurmi@fmi.fiFinnish Meteorological Institute

Participants

Deterministic modelsEPSVerificationCommentHMC, RussiaCOSMO-RU 7 km & 2 kmCOSMO-RU ~ 1 kmCOSMO-RU 2 km+ officialend-productsIRAM, Russia

+ 4 hr nowcastsARPA SIMC, ItalyCOSMO-S-14 7 km

Env. CanadaGEM 25 km downscaled to 1 km & 250 mZAMG, AustriaINCA 1-hourly analysesAladin LAEFNOAA, USANMMB 1 kmNMMB 7 kmHIRLAMconsortiumGLAMEPSHarmon EPSKMA, S-KoreaGlobal 25 km downscaledHigh resolution 1 kmFMI, FinlandHARMONIE : 1 kmRoad Weather Model on selected Sochi roads (CoMoSeF project) Close-to-real-time and retrospective objective point verification Model output at common FROST ftp server in agreed formatOfficial final forecasts by Olympics forecastersRequires online availability of all / most forecastsand local observationsFDP: in blue RDP: in greenXXII Olympic Winter GamesFeb - Mar 2014FROST-2014: Contributors - Models

FMI verification softwarepackage

+ several Nowcasting systems#FROST-2014 Verification activitiespertti.nurmi@fmi.fiFinnish Meteorological Institute

Operational Verification Software@ FMIa 2-minute intro to

Finnish Meteorological Institute#FROST-2014 Verification activitiespertti.nurmi@fmi.fiFinnish Meteorological Institute14Some Properties & FeaturesWeb-based; Interactive; Menu-driven; Real-time; 2 languagesDaily forecast vs. observation comparisonTime-averaged (monthly, seasonal, annual) statisticsAll basic - and some novelty - verification measuresComparisons : Forecasters E-P vs. model; model vs. other modelsDatabase coverage : Gridded data + stations over Finland and Sochi

Finnish Meteorological Institute#FROST-2014 Verification activitiespertti.nurmi@fmi.fiFinnish Meteorological Institute15

Result presentation formatZoomMaster selection menuOptionsProducerParameterAnalysis hourLead timePeriod typeVerif. MeasurePresentationStation groupUser interface:Web browser default page#FROST-2014 Verification activitiespertti.nurmi@fmi.fiFinnish Meteorological Institute

http://dev.meteor.fmi.fi/verif/

Idea to adjust to FROST-2014#FROST-2014 Verification activitiespertti.nurmi@fmi.fiFinnish Meteorological Institute

Master selection menu

FROST-2014Forecasters official FCsCOSMO - 2.2 kmCOSMO 7 kmGEM-Downscaled 1 kmHARMONIEGLAMEPSHARMONEPSALADIN-LAEFetc#FROST-2014 Verification activitiespertti.nurmi@fmi.fiFinnish Meteorological Institute

Summarizing some issues #1Local vs. external verification Contributors do their own thing, anyway(Close to) Real-time verification (RTFV) Visualization of fc-obs time series and inter-comparison on-the-flyStart aggregating verification statistics already before the beginning of the Games late 2013Guarantee observation quality, quantity, homogeneity and availabilityMake best use of new observation typesObservation uncertaintyAvoid rejection of good observations (in DA)Event-based verification Definition of the event - ThresholdsIndividual events vs. statisticsUser-focused (FDP) vs. Model-focused (RDP) verificationXXII Olympic Winter GamesFeb - Mar 2014#FROST-2014 Verification activitiespertti.nurmi@fmi.fiFinnish Meteorological Institute

Summarizing some issues #2Wide(r) use of diagnostic verificationUse of latest/new scores and measuresVerification of high resolution EPSUse of verification packagesIntuitive, easy-to-understand user-focused verification products and proceduresSocietal aspects THE IMPACT IssueVerification training : self-learning tools No time for face-to-face tutorial !Improved verification reporting Joint papers by FROST-2014 groupFinalize timeline of actions Time is short, as always Results to be presented at the next ECAM XXII Olympic Winter GamesFeb - Mar 2014

#FROST-2014 Verification activitiespertti.nurmi@fmi.fiFinnish Meteorological Institute

XXII Olympic Winter GamesFeb - Mar 2014

Finnish Meteorological Institute

Thank You(Sochi, 2013; Photos Nikolai Bocharnikov)Other FROST-2014 papers :D. Kiktev : EMS2013-400, Tue 08:45T. Bazlova : EMS2013-110, poster, Mon-TueA. Montani : EMS2013-166, Fri 15:15

Finnish Meteorological Institute#FROST-2014 Verification activitiespertti.nurmi@fmi.fiFinnish Meteorological Institute6th International Verification Methods Workshop 13-19 March 2013, New Delhi, IndiaTutorial Session: March 13-15, Scientific Workshop: March 17-19Highlights recent advances in the theory and practice of verification of weather and climate forecasts and warningsTargets participants from research, operational, and forecast user communities Includes a 3-day WMO verification tutorial session and a 3-day scientific program.

Potential topics:Verification of high impact weather forecasts and warnings including heavy rainVerification of ensembles and probability forecasts Spatial forecast verificationClimate projection evaluationIntra-seasonal and seasonal forecast verificationTropical cyclone verificationAviation forecast verificationPropagation of uncertaintyUser issues including communicating verification to decision makersVerification tools

Sheet1TABLE 1 (From Doyle's Threshold Charts)ParameterCategory 1Category 2Category 3Category 4Category 5Category 6Category 7Category 8Temperature< -25C 0C>4CRH< 30%>65%Winds>3 4m/s>5m/s>7m/s>13 17m/sGusts14- 17 m/sWind DirectionVisibility< 30m>50 15 30 cm6 h Snow>15 cm2 h Snow> 2cm6 h Rain>15 mmTable 2 (From Bailey's CAWW Talk)VariableCategory 1Category 2Category 3Category 4Category 5Category 6Category 7Category 8Winds< 5 kts5 w < 10 kts10 w < 15 kts15 w < 20 kts20 w < 25 ktsw 25 kts--Wind Directiond 339 & d < 24 (N)24 d < 69 (NE)69 d < 114 (E)114 d < 159 (SE)159 d < 204 (S)204 d < 249 (SW)249 d < 294 (W)294 d < 339 (NW)Visibilityv < 1/4 SM1/4 v < 1/2 SM1/2 v < 3 SM3 v < 6 SMv 6 SM---Ceilingc < 150 ft150 c< 400 ft400 c< 1000 ft1000 c< 2500 ft2500 c< 10000 ftc 10000 ft--Precip Rater = 0 mm/hr (None)0 < r 0.2 mm/hr (Trace)0.2 < r 2.5 mm/hr (Light)2.5 < r 7.5 mm/hr (Moderate)r > 7.5 mm/hr (Heavy)---Precip TypeNo PrecipLiquidFreezingFrozenMixed (w/Liquid)Unknown--Table 3 (Suggestion for SNOW-V10 Verification)VariableCategory 1Category 2Category 3Category 4Category 5Category 6Category 7Category 8Temperature< -25C 0C>4CRH< 30%>65%Winds< 3 m/s3 w < 4 m/s4 w < 5 m/s5 w < 7 m/s7 w < 13 m/s13 w < 15 m/s 15 m/s 17m/sWind Directiond 339 & d < 24 (N)24 d < 69 (NE)69 d < 114 (E)114 d < 159 (SE)159 d < 204 (S)204 d < 249 (SW)249 d < 294 (W)294 d < 339 (NW)Visibilityv < 30m30 v < 50m50 v < 200m200 v < 300m300 v < 500m 500m-Ceilingc < 50m50 c< 120 m120 c< 300 m300 c< 750 m750 c< 3000 mc 3000 m--Precip Rater = 0 mm/hr (None)0 < r 0.2 mm/hr (Trace)0.2 < r 2.5 mm/hr (Light)2.5 < r 7.5 mm/hr (Moderate)r > 7.5 mm/hr (Heavy)---Precip TypeNo PrecipLiquidFreezingFrozenMixed (w/Liquid)Unknown--Table 4 (Revised Suggestion for SNOW-V10 Verification)VariableCategory 1Category 2Category 3Category 4Category 5Category 6Category 7Category 8Temperature-25 C