Ensemble Task

Isidora Jankov

Collaborators:NOAA’s Environmental Modeling CenterNOAA’s Earth System Research Laboratory

1

Ensemble Task AOP12 Activities Activity Status

HWRF ensemble for GSI-Hybrid Applications methods

Moved to the DA task

Establish a functionally similar environment of the NCEP SREF

Accomplished

HMT Winter Experiment DelayedEvaluate ensemble products using CAPS, AFWA, and NCEP algorithms

On hold

Ensemble Code Repository AccomplishedCo-host semi-annual DTC Ensemble Task Workshop

Accomplished

Maintain ensemble group wiki page and web site

On-goingWork done late in AOP11, DTC contribution to SREF post-processing2

Work performed late in AOP11NAEFS Bias Correction and Downscaling effortHigh-resolution version of SREF (SREFx) was

recently implemented operationally at EMC.SREFx at 16km (NMM, NMMB)/17km (ARW)

resolution Significant resolution upgrade, but still

missing small scale variability needed for NWS forecast grids

Downscaling to the resolution of the NDFD (~5km) should allow greater use of NCEP SREF output in the production of forecasts

Courtesy of Brian Etherton, Jun Du and Tara Jensen

3

Bias Correction and Downscaling

Bias Correction accomplished by taking the mean forecast of each model core (ARW, NMM, etc.) sub-ensemble of the NCEP SREF and comparing it to the NAM analysis valid at the same time.

Downscale by comparing the RTMA analysis (GRID 259, like GRID 197, 1073x689, 5.079km) with the NAM analysis valid at the same time (10m wind,2m temperature and humidity).

ARW

NMM

NMMB

NAM RTMA

ANALYSESFORECASTS

BC

DSApproach used in NAEFS

4Courtesy of Brian Etherton, Jun Du and Tara Jensen

Bias Correction and Downscaling

Recursive averaging method is used (~30 day mean)Bias-Correction (in operations) and Downscaling (DTC testing)

from NAEFSBias corrected and downscaled fields for each memberAlso: downscaled forecasts of ensemble min, max, mean, spread,

10th, 50th, and 90th percentiles.

ARW

NMM

NMMB

NAM RTMA

ANALYSESFORECASTS

BC

DS

5Courtesy of Brian Etherton, Jun Du and Tara Jensen

Implementation of downscaling at EMC Downscaling code

used for NAEFS ported to DTC

Testing/evaluation of code application to existing NCEP SREF output showed forecast improvement

DTC code sent to EMC (Jun Du) for application to the new 16km/17km NCEP SREF

Code implemented by EMC in approximately 2 days, with downscaled forecasts being produced since December (Presentation at AMS Annual Meeting )

6Courtesy of Brian Etherton, Jun Du and Tara Jensen

Raw SREFx vs Bias corrected SREFx (Nov. 10 – 30, 2011, against NDAS)

3 15 27 39 51 63 75 870

0.5

1

1.5

2

2.5

3

SREFX_rawSREFX_BC

3 15 27 39 51 63 75 87-0.050

0.050.1

0.150.2

0.250.3

0.350.4

SREFX_rawSREFX_BCno skill

Ens mean fcst of T2m: RMSE Prob fcst of T2m: RPSS (12km NAM as ref)

Implementation of bias correction at EMC

Courtesy of Jun Du, NOAA/EMC 7

Testing/Evaluation at DTC - ResultsAs implemented at DTC, the bias correction reduces the RMSE of 2m T forecasts & the downscaling adds further improvement.

ARW and NMM members of SREF 2011 – 0900 UTC InitializationCompared to RTMA Analyses

Courtesy of Brian Etherton, Jun Du and Tara Jensen

8

Ensemble Task AOP12 ActivitiesActivity Status

HWRF ensemble for GSI-Hybrid Applications methods

Moved to the DA task

Establish a functionally similar environment of the NCEP SREF

Accomplished

HMT Winter Experiment DelayedEvaluate ensemble products using CAPS, AFWA, and NCEP algorithms

On hold

Ensemble Code Repository AccomplishedCo-host semi-annual DTC Ensemble Task Workshop

Accomplished

Maintain ensemble group wiki page and web site

On-going9

NCEP SREF Functionally Similar EnvironmentActivity description: Define & establish scalable, flexible , highly portable heterogeneous computational Functionally Similar Environment, (FSE) to NCEP SREF operations.Objectives and Accomplishments:1.Develop FSE acceptance evaluation criteria in

collaboration w/ NCEP 2.Establish portable integration of NCEP SREF

components & DTC platforms 3.Participate in updates of NCEP NEMS code, SREF

scripts & data cross management4.Continually support research, DET T&E workflow &

future R2O integrations5.Analyze the differences & integrate NCEP operative &

DET research workflows6.Work closely w/ EMC to deploy computational &

research innovations.Courtesy of Eugene Mirvis10

Major Accomplishment:• NCEP SREF v6.0.0 adapted

by DTC• DTC now has functionally

similar environment for 4-5 of 6 SREF modules

11

New Generation of The Ensemble Testbed

5. MERGING OPS AND RESEARCH WORKFLOWS

3. PARTICIPATE IN UPDATES,

OPS CODE & DATA MANAGEMENT

6. ACADEMIA SUPPORT & INNOVATIONS

IBM CRAY XSEDE SGI LINUX TBD

4. SUPPORT RESEARCH, T&E

WORKFLOW

2. PORTABLE INTEGRATION of OPS &

Research Computing Platforms

1. DEVELOPMENT OF THE FSE ACCEPTANCE CRITERIA DTC/

DET

~ 60 UNIVERSITIES

Conventions & Criteria

Of FunctionalSimilarity

OPSENVIRO

N-MENT

TESTBED

ENVIRON-

MENT

NOAA ESRL

1212

Shared Physics

NCEP NEMS – DTC Ensemble Workflows Merging Analysis

External dataprocessing

Alternative

Obs. data

Standard• Obs. data

NEMS

GSI

NEMS/WRF

TerrestrialDATA

Gridded Data:

NAM,GFS,RUC, snow,

ice, SST

PostVisualiz

e& Verify

RIP4

Grid-vs-Obs

VSDB/FVS

EnsembleInitializatio

n

GrADS

GEMPAK

NCL

Grid-vs-Grid

VSDB/FVS

- - not used yet at NCEP

MET

IC/LBC

perturbLand

surface

perturb

Coldstart

GLOBAL

• GFS/STD• GFS /SL• FIM

MESO• NMMB• ARW

USER • T.B.D.

NEMS Apps Layer

NEMS I/O

Stochastic Forcing Perturbation Component

NEMS entry

Ensemble

Post

UPP Ensemble

post and product

generation

BIASCorrect

&

downscale

GLOB ?

NPS

Land surface init

MESO?

YES

NO

13

Ensemble Task AOP12 ActivitiesActivity Status

HWRF ensemble for GSI-Hybrid Applications methods

Moved to the DA task

Establish a functionally similar environment of the NCEP SREF

Accomplished

HMT Winter Experiment DelayedEvaluate ensemble products using CAPS, AFWA, and NCEP algorithms

On hold

Ensemble Code Repository AccomplishedCo-host semi-annual DTC Ensemble Task Workshop

Accomplished

Maintain ensemble group wiki page and web site

On-going14

HMT Winter Experiment Activity description:

Use HMT Winter Experiment real-time system to test the impacts of alternate initial perturbation techniques (including the dynamical downscaling of global perturbations)

Test alternate ensemble perturbation techniques (cloud microphysical parameterizations

Presentation at NWA Annual.Accomplishments:

Plans developedNWA presentation delivered

Issues:This task has been affected by changes in staffing which resulted in a significant delay. 15

HMT Activities AOP12

2012 Tasks necessary to be completed for 2013 activities.

HMT real time

DTC 1 DTC 2

Microphysics

HMT SREF (ARW)

HMT

Initial Perturbatio

ns

HMT HMT SREF (ARW)

Grid Spacing

9 9 91

2

1. Evaluate the impact of changes in microphysics2. Evaluate the impact of initial perturbation changes

16

Impact of Microphysics on 2010-2011 HMT

90% Intensity show over forecast of precip for ARW-Fer and ARW-MY members especially at higher thresholds – which means when it rains it pours in these members

> 25.4

Diff

eren

ce (

Fcst

-Obs

) in

Nea

r Pe

ak In

tens

ity

in M

OD

E O

bj (

mm

)

Object Threshold (mm)

NMM ARW

HMTNAMGFS

Using Attributes from MODE Objects

Color groupsDifferentMicrophysics

FerrierMilbrant-YauThompson

Intense Cores

Optimal

Courtesy of Tara Jensen17

00Z 06Z

LAM Forecast

Global Model Forecast

Global Model Analysis interpolated on LAM grid

PerturbationLegend

Forecast Time

Initial Perturbations Approach

XaL = Xf

L + (XaG – Xf

G)18

Proposed Activities for 2013

19

Ensembles AOP 2013 Proposed Activities

ID Activity

HMT Four tests related to implementation of the HMT innovations into SREF

EN3Evaluate options for optimal combination of reference dynamic cores and corresponding analyses in SREF (currently beyond 100%)

EN4Evaluate the merits of using 2 vs. 3 dynamic cores in SREF – using ARW subset available from HMT activity (currently beyond 100%)

20

Ensemble Proposed AOP 2013 Activities

1. Evaluate impact of changes in initial conditions in context of SREF2. Evaluate impact of HMT microphysics in context of SREF3. Evaluate impact of increased resolution (16 vs. 9 km SREF) 4. Evaluate the impact of mixed microphysics by optimally combining HMT and SREF runs.

HMTReal time

DTC 1(AOP12)

DTC 2(AOP12)

DTC 3(AOP13)

NCEPOperatio

nsMicrophysic

sHMT SREF

(ARW)HMT SREF

(ARW)SREF

Initial Perturbation

s

HMT HMT SREF (ARW)

SREF (ARW)

SREF

Grid Spacing

9 9 9 9 1612&4 3

HMT CONTEXT SREF CONTEXT

21

Tressa L. Fowler

Verification

22

Verification AOP 2012 ActivitiesAOP 2012 Activity Status

MET releases in July and January

Done / Slightly Delayed

Enhancement of software testing

Done, more in progress

New verification tools for T&E support

Some done, more in progress

METViewer GUI enhancements

In progress

Initial release of MET TC Slated for 4.1 release

LSM verification DelayedTutorial Feb 4-5Community presentations Many completedMET help ongoing

23

GoalsPromote easy, consistent, correct

verification across the NWP community, especially in operations where there is the greatest potential for impact.

Create a cohesive team of scientists and software engineers engaged in challenging, interesting, rewarding work.

24

AccomplishmentsSoftware Development

MET TCMADIS data supportEnsemble spread skillGRIB2Series analysis tool

Testing and Evaluation supportHMT verificationMMET cases

Community support

25

Series Analysis Tool

Frequency Bias

Gilbert Skill Score

Happy Users

26

Verification Support of HMTCapability was added to METViewer:User can constrain aggregation using Base Rate (aka, observed relative frequency). This feature helps users focus on model skill for meaningful events without having to constrain on an event-by-event basisIncreases analysis speed and relevance

No constraintBase Rate > 0.02

HMT Ensemble Mean

27

Confidence TestsBox-and-whiskers now on most FVS plots (thanks to Keith Brill of HPC)

Verification Implementations

28

EMC Verification Website (intranet only):

http://emc-ls-sand02.ncep.noaa.gov/EMC_VSDB_verif/index.cgi

Slide excerpted from Geoff DiMego

Verification AOP 2013 ActivitiesID Activity Description

VX2 Verification support for T&E activitiesVX1 Community Support: Release, tutorial,

helpVX3 MET Development – External usersVX4 MET Development – Internal usersVX5 MET-TC Development (currently

beyond 100%)

29

Next YearSoftware Development – Internal

Enhancements to MET TC in support of HWRF testingSeries analysis tool testing and enhancementsMETViewer GUI redesign

Software Development – ExternalSatellite data and cloud verificationComplete MODE GUILand Surface Model evaluation capability

Testing and Evaluation supportHMT EvaluationUse of MET TC for HWRF evaluationsAdditional MMET casesRadiation scheme evaluation 30

Community SupportYearly tutorial, with HWRF tutorial if

fundedUpdated online tutorialMET HelpVerification lectures at conferences and

workshops

31

MET TC Development

Community TC development

Hurricane T&E development

32

MODE GUICapability developed outside DTC but will be leveraged for inclusion in community software!

33

Track Error Rank of Candidate Model vs. 3 Operational Models

34

Leveraging SURFRADDevelopment of SURFRAD ingest

underwayUseful for radiation scheme evaluationsLand surface model verificationSolar forecast evaluation for DOE project

35

ConclusionsSome work will carry over to 2013, due to

staffing changes.2013 will be a year of rebuilding.We accomplished much in 2012.Our 2013 planned work reaches into new

areasMore opportunities to assist with R2OMore learning opportunities for our staff

36

Louisa Nance

Director’s Office & Visitor Program

37

Director’s Office AOP 2013 Activities

AOP 2012 AOP 2013Director’s Office Staff Director’s Office StaffNon-salary expenses: travel, external meetings, etc…

Non-salary expenses: travel, external meetings, etc… (reduced budget by 20 K)

Director’s Reserve (65 K) Director’s Reserve (65 K) – 85-100%

DTC Visitor Program DTC Visitor Program – 85-100%

External communications (status: newsletter is in progress)

External communications – below 100%

CO1: Investigate requirements ECMWF-like environment) – 85-100%CO2: Post-doctoral Scientist Program – 85-100%

38

Director’s Office StaffAOP 2012National Director (Kuo) 0.5FTEAssistant Director (Nance)

0.55 FTE JNT

Management (Brown) 0.1 FTEBudget support (Jensen) 0.25

FTESupport staff for community

activities (Johnson) 0.5 FTEGSD

Management (Toth) 0.1 FTEBudget support

(Etherton/Bernardet & Admin Asst) 0.25 FTE

AOP 2013National Director (Kuo) 0.5FTEAssistant Director (Nance)

0.55 FTE JNT

Management (Brown) 0.1 FTEBudget support (Jensen) 0.25

FTESupport staff for community

activities (Zarlingo) 0.3 FTEGSD

Management (Toth) 0.3 FTEBudget support

(Etherton/Bernardet & Admin Asst) 0.30 FTE

39

External MeetingsAOP 20123 Workshops3 Tutorials4 Governance

meetings

AOP 20131 Workshop1 Tutorial4 Governance

meetings

40

DTC Visitor ProgramFramework for engaging the research community in projects directed at improving NWP, with an eye towards eventually improving operational forecasts

41

DTC Visitor Program

42

Supports visitors to work w/ the DTC to test new forecasting & verification techniques, models & model components for NWP

PI project – up to 2 months salary & travel & per diem

Graduate student project - up to 1 yr temp living per diem stipend & travel expenses for student to work w/ DTC &/or one of its partners + travel & per diem for up to 2 2-wk visits to the location of the student by project PI

Current DTC VisitorsPI Institution Project Title

Jonathan Vigh NCAR

Development of an HWRF Diagnostics Module to Evaluate Intensity & Structure Using Synthetic Flight Paths Through Tropical Cyclones

Adam Clark U. of OklahomaDevelopment & Application of 3-Dimensional Object Algorithms to High Resolution Forecasts

Delia Arnold &

Don Morton

Technical U. of Catalonia / U.

of Alaska –Fairbanks

Moving The NEMS & NMM-B Into a Broader Community Resource Environment

Robert Fovell / Travis Wilson

UCLAImprovements to modeling persistent surface cold pools in WRF

Sai Ravela MIT Deploying the MIT Field Alignment System & Test-bed (FAST) in DTC

43

Future DTC VisitorsPI Institutio

n Project Title

Thomas Galarneau NCAR Diagnosing Tropical Cyclone Motion

Forecast Errors in HWRFRobert Fovell / Huang

UCLAImproving HWRF Track and Intensity Forecasts Via Model Physics Evaluation and Tuning

Man Zhang CSUImpact Assessment of Cloud-Affected AMSU-A Radiance Assimilation in TC inner-Core Region using Hybrid Data Assimilation Approaches

Marion Mittermaie

rUKMET

Incorporating Observations Uncertainty to a Spatial Probabilistic Verification Framework for km-scale Models

Adam Clark U. of Oklahoma

Object-based Time-Domain Diagnostics for High-resolution Ensemble Forecasting and Evaluation in NOAA/HWT Spring Forecasting Experiments

44

AOP 2012 Visitor Funds

45

EC chair expressed concern over allocating all of the AOP 2012 funds budgeted for visitor program due to AOP 2013 funding uncertainties

NSF funds cannot be reprogrammed to cover other DTC activities, so DTC moved forward with awarding 5 new visitor projectsEstimate new projects will require only ~15-20 K from the

NOAA AOP 2012 fundingFor continued NSF support, will be important to show NOAA

matching, so plan to pursue off-cycle visitor proposals consistent with DTC priorities once we have a better picture of the AOP 2013 funding

Demonstrating NOAA matching for visitor program will be important for future proposals to NSF for both the visitor program and potential Post-doctoral Scientist Program

Potential new activitiesFollowing new activities currently fall in the 85-100 % scenario

46

ECMWF-like Visitor Tools ($128K)Background: ECMWF has an interactive meteorological software application that allows researchers to easily prepare & execute research experiments.Scope: Investigate current capabilities & consider options for building & expanding capabilities to easily prepare & run research experiments using NCEP’s models.

Requirements analysis:Identify user community

(skills, needs)Identify system

components adaptable to these needs

Identify hardware, software & data needs & resources

Identify documentation & training needs

Preliminary system designPlanning requires EMC staff participation in this info gathering stage!

47

Post-doctoral Scientist Program ($151K)Motivation: Mechanism for engaging next generation scientists in research focused on improving operational forecastsProposed format:• Gather information from EMC on key areas for

which they are interested in contributions from the research community

• Applicants submit proposals for projects focused on an identified key area

• Successful applicant would reside at EMC in order to work closely with EMC staff and gain better understanding of operational requirements

48

Summary• Activities that engage the NWP community

and keep the community informed about DTC activities are important to full-filling the DTC mission

• Need to make sure we don’t overlook the importance of these activities and determine which types are most effective

49