TheNationalCenterforAtmosphericResearchissponsoredbytheNationalScienceFoundation.Anyopinions,findingsandconclusionsorrecommendationsexpressedinthispublicationarethoseoftheauthor(s)anddonotnecessarilyreflecttheviewsoftheNationalScienceFoundation.
©UCAR
DARTTutorialSection18: LostinPhaseSpace:TheChallengeofNotKnowingtheTruth.
RealityStrikes
Inrealapplications,thetruthisunknown.
Allthatwehaveareobservations.
Havingthetruthavailablehasbeenconvenient,butalsomisleading.
Muchlessinformationisavailablefromtheobservations.Theyaregenerallyfunctionsofthestatevariables.Theyarealwayscontaminatedwithobservationalerrors.
DARTTutorialSection18:Slide2
Whattoexpect…
−4 −2 0 2 40
0.2
0.4
0.6
0.8
Prob
abilit
y
Prior PDF Obs. Likelihood
S.D.Inflated S.D. Expected Separation
Actual 3.698 SDs
RecallthatExpected(prior_mean– observation)= σ prior
2 +σ obs2
ErrorisdominatedbyobservationalnoiseifSuppose= 1.0, = 0.1, thenE(RMS) = 1.005.Halvingto0.05 => E(RMS) = 1.001;onlya0.4% reduction!
σ obs2 ≫σ prior
2
σ prior
σ obs σ prior
DARTTutorialSection18:Slide3
&quality_control_nmlinput_qc_threshold = 3.0outlier_threshold = -1.0
/
FirstObservation-spacediagnostics:WhetherornottoassimilateorrejectobservationsbasedontheirExpectedSeparationiscontrolledduringfilter basedonnamelistsettingsininput.nml.
If > outlier_threshold
Theprogramobs_diag post-processesobs_seq.final,calculatesmetricslikeRMSE,bias,ensemblespread,totalspread,#ofobservationsusedorrejected…Startwiththelorenz_96model.
yp − yo σ prior2 +σ obs
2 Observationrejected!(DARTQC==7)
DARTTutorialSection18:Slide4
Observation-spacediagnosticsTheobservationsequencefileisnotinaparticularlyuser-friendlyformat.Toaidintheevaluationandinterpretation,aprogramnamedobs_diagmustberuntoproduceanetCDFfilewithresultsthatcanbeplottedinamannerofyourchoosing.DARThasMatlabfunctions/scriptsthatcreatehigh-qualitygraphics.Forup-to-dateinformationonthelatest,greatestdiagnostics,goto:http://www.image.ucar.edu/DAReS/DART/DART2_Diagnostics.php#obs_diagnostics
&obs_diag_nmlobs_sequence_name = 'obs_seq.final',bin_width_days = -1,bin_width_seconds = -1,init_skip_days = 0,init_skip_seconds = 0,Nregions = 3,trusted_obs = 'null',lonlim1 = 0.00, 0.00, 0.50, -1.0lonlim2 = 1.01, 0.50, 1.01, -1.0reg_names = 'whole’, ‘yin’, ‘yang’, ‘bogus,create_rank_histogram = .true.,outliers_in_histogram = .true.,use_zero_error_obs = .false.,verbose = .false./
DARTTutorialSection18:Slide5
Observation-spacediagnosticsTheobservationsequencefileisnotinaparticularlyuser-friendlyformat.Toaidintheevaluationandinterpretation,aprogramnamedobs_diagmustberuntoproduceanetCDFfilewithresultsthatcanbeplottedinamannerofyourchoosing.DARThasMatlabfunctions/scriptsthatcreatehigh-qualitygraphics.Forup-to-dateinformationonthelatest,greatestdiagnostics,goto:http://www.image.ucar.edu/DAReS/DART/DART2_Diagnostics.php#obs_diagnostics
TheseworkwithANYobs_seq.final fromANYexperimentwithANYmodel!
HereareafewoftheMatlabfunctionsavailableindiagnostics/matlab
• plot_rank_histogram.m• plot_evolution.m• plot_rmse_xxx_evolution.m• two_experiments_evolution.m• plot_profile.m• plot_bias_xxx_profile.m• plot_rmse_xxx_profile.m• two_experiments_profile.m
DARTTutorialSection18:Slide6
Lorenz96observationdiagnosticexample
month/day - Jan.01,1601 01:00:00 start01/01 01/06 01/11 01/16 01/21 01/26 01/31 02/05 02/10 02/15
rmse
0
1
2
3
4
5
6
7
yangRAW_STATE_VARIABLE
forecast: mean=2.9555 analysis: mean=2.8093
forecastanalysis
data file: /Users/thoar/svn/DART/clean_lanai/models/lorenz_96/work/obs_diag_output.nc
# of
obs
: o=
poss
ible
, ∗=a
ssim
ilate
d
23
24
25
26
27
28
29
30
outlier_threshold=-1.0
DARTTutorialSection18:Slide7
&filter_nml…obs_sequence_in_name = "obs_seq.out”obs_sequence_out_name = "obs_seq.final”…/
&quality_control_nmlinput_qc_threshold = 3.0,outlier_threshold = -1.0,/
FirstObservation-spacediagnostics:Trysettingtherejectionthresholdtoasmallpositivenumberandrerunningfilter,andthenrerunningobs_diag onthenewoutputfile.
Don’tforgettorerunfilter!Don’tforgettorerunobs_diag!Don’tforgettousetherightfilenameinobs_diag_nml!
Thisispotentially ,butuseful.Rejecting‘good’observationscanleadtoinflatedestimateofquality.
DARTTutorialSection18:Slide8
FirstObservation-spacediagnostics:
month/day - Jan.01,1601 01:00:00 start01/01 01/06 01/11 01/16 01/21 01/26 01/31 02/05 02/10 02/15
rmse
0
0.5
1
1.5
2
2.5
3
yangRAW_STATE_VARIABLE
forecast: mean=1.6213 analysis: mean=1.5243
forecastanalysis
data file: /Users/thoar/svn/DART/clean_lanai/models/lorenz_96/work/obs_diag_output.nc
# of
obs
: o=
poss
ible
, ∗=a
ssim
ilate
d
0
5
10
15
20
25
30
Observationsbeingrejected!
LowerRMSEthanbefore!$1,000,000question:Why?
outlier_threshold=3.0
DARTTutorialSection18:Slide9
Lorenz_96exercises:
• Pickacasethatworksrelativelywellandlookatobservation-spacediagnostics.
• Pickacasethatissimilar,butclearlydifferent,withphysical-spacediagnostics.
• Seeifyoucandetectthedifferencewithobservation-spacediagnostics.
• Rerunobs_diag withdifferentbin_widths.
DARTTutorialSection18:Slide10
Observation-spacediagnostics:rankhistograms
0 5 10 15 20 25 30 35 40 45 500
10
20
30
40
50
60
70
2297 obs possible, 2269 obs binned
Observation Rank (among ensemble members)co
unt
MPEX
0 5 10 15 20 25 30 35 40 45 500
50
100
150
200
250
300
350
12263 obs possible, 12106 obs binned
Observation Rank (among ensemble members)
coun
t
RADIOSONDE_TEMPERATURE @ 500 hPaFull Domain
May.16,2015 21:00:01 −− May.24,2015 03:00:00data file: /Users/thoar/svn/DART/clean_lanai/models/wrf/work/obs_diag_output.nc
>> fname = ‘obs_diag_output.nc’;>> timeindex = -1;>> varname = ‘RADIOSONDE_TEMPERATURE’;>> plot_rank_histogram(fname, timeindex, varname);
ResultsfromWRFreal-timeforecasting.
DARTTutorialSection18:Slide11
Observation-spacediagnostics:timeevolution(bylevel)
08/01 08/06 08/11 08/160
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
2
month/day − Aug.01,2005 06:00:00 start
rmse
and
tota
lspr
ead
Northern Hemisphere (20−80)RADIOSONDE_TEMPERATURE @ 500 hPa
rmse pr=1.1971, po=0.98162 totalspread pr=0.91985, po=0.81559
rmsetotalspread
data file: /glade/u/home/thoar/svn/DART/trunk/models/cam/work/obs_diag_output.nc
0
200
400
600
800
1000
1200
1400
1600
1800
2000
# of
obs
: o=
poss
, �=u
sed
Initiallytinyspreadandlargeobservationrejection– systemnotperformingwell– yet!
plot_rmse_xxx_evolution.mplot_evolution.m
Totalspreadisthesqrtofthepooledvarianceoftheobservationerrorandtheensemblevariance.
MuchBetter!Veryfewobservationsbeingrejected.
DARTTutorialSection18:Slide12
Observation-spacediagnostics:time-averagedprofilesplot_profile.m plot_bias_xxx_profile.mplot_rmse_xxx_profile.m
bias (model − observation) and totalspread16−May−2015 21:00:01 through 24−May−2015 03:00:00
hPa
−1 0 1 2 3
100150200250300
400
500
700
850
925
1000
bias pr=0.26525bias po=0.02529totalspread pr=1.563totalspread po=0.99202
0 7 14 21 28# of obs (o=possible, ∗=assimilated) x100
MPEXRADIOSONDE_TEMPERATURE
data file: /Users/thoar/svn/DART/clean_lanai/models/wrf/work/obs_diag_output.nc
bias (model − observation) and totalspread16−May−2015 21:00:01 through 24−May−2015 03:00:00
hPa
−1 0 1 2 3
100150200250300
400
500
700
850
925
1000
bias pr=−0.028017bias po=0.0027291totalspread pr=1.5112totalspread po=1.0003
0 4 8 12 16# of obs (o=possible, ∗=assimilated) x1000
Full DomainRADIOSONDE_TEMPERATURE
data file: /Users/thoar/svn/DART/clean_lanai/models/wrf/work/obs_diag_output.nc
Note:Thesearemuchmoreinformativeformodelswithlevels!(i.e.the1Dmodelsarenotveryinterestingthisway)
DARTTutorialSection18:Slide13
&obs_diag_nmlobs_sequence_name = ‘’obs_sequence_list = ’file_list.txt’first_bin_center = 2005, 8, 1, 6, 0, 0last_bin_center = 2005, 8,26, 0, 0, 0bin_separation = 0, 0, 0, 6, 0, 0bin_width = 0, 0, 0, 6, 0, 0time_to_skip = 0, 0,10, 0, 0, 0max_num_bins = 1000trusted_obs = 'null’…
/
Awordofwarning…
08/01 08/06 08/11 08/16 08/21 08/260
0.5
1
1.5
2
2.5
3
month/day − Aug.01,2005 06:00:00 start
rmse
and
tota
lspre
ad
Northern HemisphereRADIOSONDE_TEMPERATURE @ 500 hPa
rmse pr=1.1176, po=0.91808 totalspread pr=0.91241, po=0.8187
rmsetotalspread
data file: /glade/scratch/raeder/SE30r4_Katrina/Diag_NoSoTrCarib_2005_8_1−23/obs_diag_output.nc
0
400
800
1200
1600
2000
2400
# of
obs
: o=
poss
, +=u
sed
obs_diag ‘time_to_skip’settingwillallowyoutoignorethespinupbeforestartingthetime-averagingfortheverticalprofileswhilestillcalculatingmetricsfortheentireperiodofrecordforthetime-evolutionproducts.
NOTE:The&obs_diag_nml isdifferentforlow-ordermodelsandrealisticmodels.Checkcarefullywhenusing.
DARTTutorialSection18:Slide14
Observation-spacediagnostics:comparingexperiments
08/01 08/06 08/11 08/16 08/21 08/26 08/31−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
31−Jul−2005 18:00:01 through 31−Aug−2005 06:00:00
fore
cast
bia
s (m
odel
− o
bser
vatio
n)
0
30
60
90
120
150
180
210
240
270
# of
obs
(o=p
ossi
ble,
∗=a
ssim
ilate
d)
Southern HemisphereRADIOSONDE_U_WIND_COMPONENT @ 500 hPa
Identical Twin PriorFraternal Twin Prior
data file: /Users/thoar/svn/DART/clean_lanai/models/cam/work/obs_diag_itwin.ncdata file: /Users/thoar/svn/DART/clean_lanai/models/cam/work/obs_diag_ftwin.nc
two_experiments_evolution.mtwo_experiments_profile.m
Thisisusefulforquickcomparisons.Really‘fair’comparisonsrequiremoreprocessingtocomparethesamesetofobservationsacrossexperiments.
assimilation_code/programsobs_common_subset.htmlobs_seq_coverage.htmlobs_selection.htmlobs_seq_verify.html
FYI:‘Identical’meansthemodelthatwasusedtogeneratetheobservationsisalsousedfortheassimilation.‘Fraternal’meanstheobservationscamefromadifferentmodel.
DARTTutorialSection18:Slide15
Observation-spacediagnostics:netCDF
SOMEoftheinformationintheobservationsequencefilescanbeconvertedtonetCDFandeasilyplotted.Aprogramnamedobs_seq_to_netcdfmustberuntoproducethenetCDF.
HereareafewoftheMatlabfunctionsavailableindiagnostics/matlab.
• link_obs.m• plot_obs_netcdf.m• plot_obs_netcdf_diffs.m• plot_coverage.m
DARTTutorialSection18:Slide16
&schedule_nmlcalendar = 'Gregorian’first_bin_start = 2005, 8, 13, 21, 0, 0first_bin_end = 2005, 8, 14, 03, 0, 0last_bin_end = 2005, 8, 14, 03, 0, 0bin_interval_days = 10000bin_interval_seconds = 0max_num_bins = 1000print_table = .true.
/&obs_seq_to_netcdf_nml
obs_sequence_name = ‘cam_obs_seq.2005-08-14-00000.final’obs_sequence_list = ‘’lonlim1 = 160.lonlim2 = 40.latlim1 = 10.latlim2 = 65.
/
Theprogramobs_seq_to_netcdf convertsmuchoftheinformationinanobservationsequencefiletoanetCDFfile.Fornow,we’regoingtoexploreapre-computedfileavailableat:
www.image.ucar.edu/pub/DART/Tutorial_Datasets/obs_epoch_SE30r4_Katrina.nc
Itwasgeneratedwiththefollowinginput:
Complicatedobservation-spacediagnostics.
DARTTutorialSection18:Slide17
MatlabHands-On: link_obsexploringobservations
Thisenablesrotationwiththemouse.
‘paintbrush’allowsyoutoselectobservationsfor‘brushing’
Trydifferentobstypes,Trytolocaterejectedobs,Whyweretheyrejected?Tryplot_obs_netcdf.m
DARTTutorialSection18:Slide18
1. FilteringForaOneVariableSystem2. TheDARTDirectoryTree3. DARTRuntimeControlandDocumentation4. Howshouldobservationsofastatevariableimpactanunobservedstatevariable?
Multivariateassimilation.5. ComprehensiveFilteringTheory:Non-IdentityObservationsandtheJointPhaseSpace6. OtherUpdatesforAnObservedVariable7. SomeAdditionalLow-OrderModels8. DealingwithSamplingError9. MoreonDealingwithError;Inflation10. RegressionandNonlinearEffects11. CreatingDARTExecutables12. AdaptiveInflation13. HierarchicalGroupFiltersandLocalization14. QualityControl15. DARTExperiments:ControlandDesign16. DiagnosticOutput17. CreatingObservationSequences18. LostinPhaseSpace:TheChallengeofNotKnowingtheTruth19. DART-CompliantModelsandMakingModelsCompliant20. ModelParameterEstimation21. ObservationTypesandObservingSystemDesign22. ParallelAlgorithmImplementation23. Locationmoduledesign(notavailable)24. Fixedlagsmoother(notavailable)25. Asimple1Dadvectionmodel:TracerDataAssimilation
DARTTutorialIndextoSections
DARTTutorialSection18:Slide19