Ensemble Task
Isidora Jankov
Collaborators:NOAA’s Environmental Modeling CenterNOAA’s Earth System Research Laboratory
1
Ensemble Task AOP12 Activities Activity Status
HWRF ensemble for GSI-Hybrid Applications methods
Moved to the DA task
Establish a functionally similar environment of the NCEP SREF
Accomplished
HMT Winter Experiment Delayed
Evaluate ensemble products using CAPS, AFWA, and NCEP algorithms
On hold
Ensemble Code Repository Accomplished
Co-host semi-annual DTC Ensemble Task Workshop
Accomplished
Maintain ensemble group wiki page and web site
On-goingWork done late in AOP11, DTC contribution to SREF post-processing
2
Work performed late in AOP11NAEFS Bias Correction and Downscaling effort
High-resolution version of SREF (SREFx) was recently implemented operationally at EMC.
SREFx at 16km (NMM, NMMB)/17km (ARW) resolution
Significant resolution upgrade, but still missing small scale variability needed for NWS forecast grids
Downscaling to the resolution of the NDFD (~5km) should allow greater use of NCEP SREF output in the production of forecasts
Courtesy of Brian Etherton, Jun Du and Tara Jensen
3
Bias Correction and Downscaling
Bias Correction accomplished by taking the mean forecast of each model core (ARW, NMM, etc.) sub-ensemble of the NCEP SREF and comparing it to the NAM analysis valid at the same time.
Downscale by comparing the RTMA analysis (GRID 259, like GRID 197, 1073x689, 5.079km) with the NAM analysis valid at the same time (10m wind,2m temperature and humidity).
ARW
NMM
NMMB
NAM RTMA
ANALYSESFORECASTS
BC
DSApproach used in NAEFS
4Courtesy of Brian Etherton, Jun Du and Tara Jensen
Bias Correction and Downscaling
Recursive averaging method is used (~30 day mean)Bias-Correction (in operations) and Downscaling (DTC testing)
from NAEFSBias corrected and downscaled fields for each memberAlso: downscaled forecasts of ensemble min, max, mean,
spread, 10th, 50th, and 90th percentiles.
ARW
NMM
NMMB
NAM RTMA
ANALYSESFORECASTS
BC
DS
5Courtesy of Brian Etherton, Jun Du and Tara Jensen
Implementation of downscaling at EMC Downscaling code
used for NAEFS ported to DTC
Testing/evaluation of code application to existing NCEP SREF output showed forecast improvement
DTC code sent to EMC (Jun Du) for application to the new 16km/17km NCEP SREF
Code implemented by EMC in approximately 2 days, with downscaled forecasts being produced since December (Presentation at AMS Annual Meeting )
6Courtesy of Brian Etherton, Jun Du and Tara Jensen
Raw SREFx vs Bias corrected SREFx (Nov. 10 – 30, 2011, against NDAS)
3 15 27 39 51 63 75 870
0.5
1
1.5
2
2.5
3
SREFX_rawSREFX_BC
3 15 27 39 51 63 75 87-0.050
0.050.1
0.150.2
0.250.3
0.350.4
SREFX_rawSREFX_BCno skill
Ens mean fcst of T2m: RMSE Prob fcst of T2m: RPSS (12km NAM as ref)
Implementation of bias correction at EMC
Courtesy of Jun Du, NOAA/EMC 7
Testing/Evaluation at DTC - Results
As implemented at DTC, the bias correction reduces the RMSE of 2m T forecasts & the downscaling adds further improvement.
ARW and NMM members of SREF 2011 – 0900 UTC InitializationCompared to RTMA Analyses
Courtesy of Brian Etherton, Jun Du and Tara Jensen
8
Ensemble Task AOP12 ActivitiesActivity Status
HWRF ensemble for GSI-Hybrid Applications methods
Moved to the DA task
Establish a functionally similar environment of the NCEP SREF
Accomplished
HMT Winter Experiment Delayed
Evaluate ensemble products using CAPS, AFWA, and NCEP algorithms
On hold
Ensemble Code Repository Accomplished
Co-host semi-annual DTC Ensemble Task Workshop
Accomplished
Maintain ensemble group wiki page and web site
On-going
9
NCEP SREF Functionally Similar EnvironmentActivity description: Define & establish scalable, flexible , highly portable heterogeneous computational Functionally Similar Environment, (FSE) to NCEP SREF operations.Objectives and Accomplishments:1.Develop FSE acceptance evaluation criteria in
collaboration w/ NCEP 2.Establish portable integration of NCEP SREF
components & DTC platforms 3.Participate in updates of NCEP NEMS code, SREF
scripts & data cross management4.Continually support research, DET T&E workflow &
future R2O integrations5.Analyze the differences & integrate NCEP operative &
DET research workflows6.Work closely w/ EMC to deploy computational &
research innovations.
Courtesy of Eugene Mirvis10
Major Accomplishment:• NCEP SREF v6.0.0 adapted
by DTC• DTC now has functionally
similar environment for 4-5 of 6 SREF modules
11
New Generation of The Ensemble Testbed
5. MERGING OPS AND RESEARCH WORKFLOWS
3. PARTICIPATE IN UPDATES,
OPS CODE & DATA MANAGEMENT
6. ACADEMIA SUPPORT & INNOVATIONS
IBM CRAY XSEDE SGI LINUX TBD
4. SUPPORT RESEARCH, T&E
WORKFLOW
2. PORTABLE INTEGRATION of OPS &
Research Computing Platforms
1. DEVELOPMENT OF THE FSE ACCEPTANCE CRITERIA DTC/
DET
~ 60 UNIVERSITIES
Conventions & Criteria
Of FunctionalSimilarity
OPSENVIRO
N-MENT
TESTBED
ENVIRON-
MENT
NOAA ESRL
1212
Shared Physics
NCEP NEMS – DTC Ensemble Workflows Merging Analysis
External dataprocessing
Alternative
Obs. data
Standard• Obs. data
NEMS
GSI
NEMS/WRF
TerrestrialDATA
Gridded Data:
NAM,GFS,RUC, snow,
ice, SST
PostVisualiz
e& Verify
RIP4
Grid-vs-Obs
VSDB/FVS
EnsembleInitializatio
n
GrADS
GEMPAK
NCL
Grid-vs-Grid
VSDB/FVS
- - not used yet at NCEP
MET
IC/LBC
perturbLand
surface
perturb
Coldstart
GLOBAL
• GFS/STD• GFS /SL• FIM
MESO
• NMMB• ARW
USER
• T.B.D.
NEMS Apps Layer
NEMS I/O
Stochastic Forcing Perturbation Component
NEMS entry
Ensemble
Post
UPP Ensemble
post and product
generation
BIASCorrect
&
downscale
GLOB
?
NPS
Land surface init
MESO
?
YES
NO
13
Ensemble Task AOP12 ActivitiesActivity Status
HWRF ensemble for GSI-Hybrid Applications methods
Moved to the DA task
Establish a functionally similar environment of the NCEP SREF
Accomplished
HMT Winter Experiment Delayed
Evaluate ensemble products using CAPS, AFWA, and NCEP algorithms
On hold
Ensemble Code Repository Accomplished
Co-host semi-annual DTC Ensemble Task Workshop
Accomplished
Maintain ensemble group wiki page and web site
On-going
14
HMT Winter Experiment Activity description:
Use HMT Winter Experiment real-time system to test the impacts of alternate initial perturbation techniques (including the dynamical downscaling of global perturbations)
Test alternate ensemble perturbation techniques (cloud microphysical parameterizations
Presentation at NWA Annual.Accomplishments:
Plans developedNWA presentation delivered
Issues:This task has been affected by changes in staffing which resulted in a significant delay.
15
HMT Activities AOP12
2012 Tasks necessary to be completed for 2013 activities.
HMT real time
DTC 1 DTC 2
Microphysics
HMT SREF (ARW)
HMT
Initial Perturbatio
ns
HMT HMT SREF (ARW)
Grid Spacing
9 9 91
2
1. Evaluate the impact of changes in microphysics2. Evaluate the impact of initial perturbation changes
16
Impact of Microphysics on 2010-2011 HMT
90% Intensity show over forecast of precip for ARW-Fer and ARW-MY members especially at higher thresholds – which means when it rains it pours in these members
> 25.4
Diff
ere
nce (
Fcst-
Ob
s)
in N
ear
Peak
In
ten
sit
y in
MO
DE O
bj (m
m)
Object Threshold (mm)
NMM ARW
HMTNAMGFS
Using Attributes from MODE Objects
Color groupsDifferentMicrophysics
FerrierMilbrant-Yau
Thompson
Intense Cores
Optimal
Courtesy of Tara Jensen17
00Z 06Z
LAM Forecast
Global Model Forecast
Global Model Analysis interpolated on LAM grid
PerturbationLegend
Forecast Time
Initial Perturbations Approach
XaL = Xf
L + (XaG – Xf
G)18
Proposed Activities for 2013
19
Ensembles AOP 2013 Proposed Activities
ID Activity
HMTFour tests related to implementation of the HMT innovations into SREF
EN3
Evaluate options for optimal combination of reference dynamic cores and corresponding analyses in SREF (currently beyond 100%)
EN4
Evaluate the merits of using 2 vs. 3 dynamic cores in SREF – using ARW subset available from HMT activity (currently beyond 100%)
20
Ensemble Proposed AOP 2013 Activities
1. Evaluate impact of changes in initial conditions in context of SREF2. Evaluate impact of HMT microphysics in context of SREF3. Evaluate impact of increased resolution (16 vs. 9 km SREF) 4. Evaluate the impact of mixed microphysics by optimally combining HMT and SREF runs.
HMTReal time
DTC 1(AOP12)
DTC 2(AOP12)
DTC 3(AOP13)
NCEPOperatio
ns
Microphysics
HMT SREF (ARW)
HMT SREF (ARW)
SREF
Initial Perturbation
s
HMT HMT SREF (ARW)
SREF (ARW)
SREF
Grid Spacing
9 9 9 9 161
2&4 3
HMT CONTEXT SREF CONTEXT
21
Tressa L. Fowler
Verification
22
Verification AOP 2012 ActivitiesAOP 2012 Activity Status
MET releases in July and January
Done / Slightly Delayed
Enhancement of software testing
Done, more in progress
New verification tools for T&E support
Some done, more in progress
METViewer GUI enhancements
In progress
Initial release of MET TC Slated for 4.1 release
LSM verification Delayed
Tutorial Feb 4-5
Community presentations Many completed
MET help ongoing
23
GoalsPromote easy, consistent, correct
verification across the NWP community, especially in operations where there is the greatest potential for impact.
Create a cohesive team of scientists and software engineers engaged in challenging, interesting, rewarding work.
24
AccomplishmentsSoftware Development
MET TCMADIS data supportEnsemble spread skillGRIB2Series analysis tool
Testing and Evaluation supportHMT verificationMMET cases
Community support
25
Series Analysis Tool
Frequency Bias
Gilbert Skill Score
Happy Users
26
Verification Support of HMTCapability was added to METViewer:
User can constrain aggregation using Base Rate (aka, observed relative frequency).
This feature helps users focus on model skill for meaningful events without having to constrain on an event-by-event basis
Increases analysis speed and relevance
No constraint
Base Rate > 0.02
HMT Ensemble Mean
27
Confidence TestsBox-and-whiskers now on most FVS plots (thanks to Keith Brill of HPC)
Verification Implementations
28
EMC Verification Website (intranet only):
http://emc-ls-sand02.ncep.noaa.gov/EMC_VSDB_verif/index.cgi
Slide excerpted from Geoff DiMego
Verification AOP 2013 Activities
ID Activity DescriptionVX2 Verification support for T&E activitiesVX1 Community Support: Release, tutorial,
helpVX3 MET Development – External usersVX4 MET Development – Internal usersVX5 MET-TC Development (currently
beyond 100%)
29
Next YearSoftware Development – Internal
Enhancements to MET TC in support of HWRF testingSeries analysis tool testing and enhancementsMETViewer GUI redesign
Software Development – ExternalSatellite data and cloud verificationComplete MODE GUILand Surface Model evaluation capability
Testing and Evaluation supportHMT EvaluationUse of MET TC for HWRF evaluationsAdditional MMET casesRadiation scheme evaluation 30
Community SupportYearly tutorial, with HWRF tutorial if
fundedUpdated online tutorialMET HelpVerification lectures at conferences and
workshops
31
MET TC Development
Community TC development
Hurricane T&E development
32
MODE GUICapability developed outside DTC but will be leveraged for inclusion in community software!
33
Track Error Rank of Candidate Model vs. 3 Operational Models
34
Leveraging SURFRADDevelopment of SURFRAD ingest
underwayUseful for radiation scheme evaluationsLand surface model verificationSolar forecast evaluation for DOE project
35
ConclusionsSome work will carry over to 2013, due to
staffing changes.2013 will be a year of rebuilding.We accomplished much in 2012.Our 2013 planned work reaches into new
areasMore opportunities to assist with R2OMore learning opportunities for our staff
36
Louisa Nance
Director’s Office & Visitor Program
37
Director’s Office AOP 2013 Activities
AOP 2012 AOP 2013
Director’s Office Staff Director’s Office Staff
Non-salary expenses: travel, external meetings, etc…
Non-salary expenses: travel, external meetings, etc… (reduced budget by 20 K)
Director’s Reserve (65 K) Director’s Reserve (65 K) – 85-100%
DTC Visitor Program DTC Visitor Program – 85-100%
External communications (status: newsletter is in progress)
External communications – below 100%
CO1: Investigate requirements ECMWF-like environment) – 85-100%
CO2: Post-doctoral Scientist Program – 85-100%
38
Director’s Office StaffAOP 2012National Director (Kuo) 0.5FTEAssistant Director (Nance)
0.55 FTE JNT
Management (Brown) 0.1 FTEBudget support (Jensen) 0.25
FTESupport staff for community
activities (Johnson) 0.5 FTEGSD
Management (Toth) 0.1 FTEBudget support
(Etherton/Bernardet & Admin Asst) 0.25 FTE
AOP 2013National Director (Kuo) 0.5FTEAssistant Director (Nance)
0.55 FTE JNT
Management (Brown) 0.1 FTEBudget support (Jensen) 0.25
FTESupport staff for community
activities (Zarlingo) 0.3 FTEGSD
Management (Toth) 0.3 FTEBudget support
(Etherton/Bernardet & Admin Asst) 0.30 FTE
39
External MeetingsAOP 20123 Workshops3 Tutorials4 Governance
meetings
AOP 20131 Workshop1 Tutorial4 Governance
meetings
40
DTC Visitor Program
Framework for engaging the research community in projects directed at improving NWP, with an eye towards eventually improving operational forecasts
41
DTC Visitor Program
42
Supports visitors to work w/ the DTC to test new forecasting & verification techniques, models & model components for NWP
PI project – up to 2 months salary & travel & per diem
Graduate student project - up to 1 yr temp living per diem stipend & travel expenses for student to work w/ DTC &/or one of its partners + travel & per diem for up to 2 2-wk visits to the location of the student by project PI
Current DTC VisitorsPI Institution Project Title
Jonathan Vigh
NCAR
Development of an HWRF Diagnostics Module to Evaluate Intensity & Structure Using Synthetic Flight Paths Through Tropical Cyclones
Adam Clark U. of OklahomaDevelopment & Application of 3-Dimensional Object Algorithms to High Resolution Forecasts
Delia Arnold &
Don Morton
Technical U. of Catalonia / U.
of Alaska –Fairbanks
Moving The NEMS & NMM-B Into a Broader Community Resource Environment
Robert Fovell / Travis Wilson
UCLAImprovements to modeling persistent surface cold pools in WRF
Sai Ravela MITDeploying the MIT Field Alignment System & Test-bed (FAST) in DTC
43
Future DTC Visitors
PI Institutio
n Project Title
Thomas Galarneau NCAR
Diagnosing Tropical Cyclone Motion Forecast Errors in HWRF
Robert Fovell / Huang
UCLAImproving HWRF Track and Intensity Forecasts Via Model Physics Evaluation and Tuning
Man Zhang CSU
Impact Assessment of Cloud-Affected AMSU-A Radiance Assimilation in TC inner-Core Region using Hybrid Data Assimilation Approaches
Marion Mittermaie
rUKMET
Incorporating Observations Uncertainty to a Spatial Probabilistic Verification Framework for km-scale Models
Adam ClarkU. of
Oklahoma
Object-based Time-Domain Diagnostics for High-resolution Ensemble Forecasting and Evaluation in NOAA/HWT Spring Forecasting Experiments
44
AOP 2012 Visitor Funds
45
EC chair expressed concern over allocating all of the AOP 2012 funds budgeted for visitor program due to AOP 2013 funding uncertainties
NSF funds cannot be reprogrammed to cover other DTC activities, so DTC moved forward with awarding 5 new visitor projectsEstimate new projects will require only ~15-20 K from the
NOAA AOP 2012 fundingFor continued NSF support, will be important to show
NOAA matching, so plan to pursue off-cycle visitor proposals consistent with DTC priorities once we have a better picture of the AOP 2013 funding
Demonstrating NOAA matching for visitor program will be important for future proposals to NSF for both the visitor program and potential Post-doctoral Scientist Program
Potential new activities
Following new activities currently fall in the 85-100 % scenario
46
ECMWF-like Visitor Tools ($128K)Background: ECMWF has an interactive meteorological software application that allows researchers to easily prepare & execute research experiments.Scope: Investigate current capabilities & consider options for building & expanding capabilities to easily prepare & run research experiments using NCEP’s models.
Requirements analysis:Identify user community
(skills, needs)Identify system
components adaptable to these needs
Identify hardware, software & data needs & resources
Identify documentation & training needs
Preliminary system design
Planning requires EMC staff participation in this info gathering stage!47
Post-doctoral Scientist Program ($151K)Motivation: Mechanism for engaging next generation scientists in research focused on improving operational forecastsProposed format:• Gather information from EMC on key areas for
which they are interested in contributions from the research community
• Applicants submit proposals for projects focused on an identified key area
• Successful applicant would reside at EMC in order to work closely with EMC staff and gain better understanding of operational requirements
48
Summary• Activities that engage the NWP community
and keep the community informed about DTC activities are important to full-filling the DTC mission
• Need to make sure we don’t overlook the importance of these activities and determine which types are most effective
49