Post on 30-Mar-2015
transcript
Generation of Pareto Optimal Ensembles of Calibrated
Parameter Sets for Climate Models
Keith Dalbey, Ph.D.Sandia National Labs, Dept 1441, Optimization and Uncertainty Quantification
Michael Levy, Ph.D.Sandia National Labs, Dept 1442, Numerical Analysis and Applications
Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy’s National Nuclear
Security Administration under Contract DE-AC04-94AL85000.
December 12-17, 2010
Outline• Motivation• Approach: Pareto Ensemble• What Does “Pareto Optimal” Mean?• Finding a “Pareto Optimal” Ensemble• Results of Tuning Climate Model• Summary & Future Work
ReferencesJackson et al, “Error reduction and convergence in climate prediction,”
Journal of Climate, 2008.
Eddy & Lewis, “Effective generation of pareto sets using genetic programming,” Proc. of ASME Design Engineering Technical Conference, 2001.
Dalbey & Karystinos, “Fast generation of space-filling latin hypercube sample designs,” Proc. of 13th AIAA/ISSMO Multidisciplinary Analysis and Optimization Conference, 2010.
Motivation Calibrating (tuning) climate models • choosing values of model parameters to predict well
Is difficult because• They have many inputs and outputs• Diverse parameters sets can match observations similarly well• Errors can compensate: “2 wrongs can make a right” under
historical conditions• Climate change (new conditions) might expose a previously
hidden mis-calibration, so…
History matching is necessary but not sufficient for good predictions.
• The future is uncertain, but we can quantify the uncertainty (estimate statistics) for possible future climates.
Approach: Pareto EnsembleHow can we make good statistical predictions?
Use a diverse ensemble of “good” parameter sets to determine the range/spread of possible future climates
QUESTION: What’s the definition of a “good” parameter set? There are multiple outputs and what’s good for one output can be bad for another.
(AN) ANSWER: It’s Pareto optimal. A point (parameter set) is Pareto optimal if there is no other point that is as good or better than it in ALL outputs.
What does the “Pareto” mean?It’s just the name of the person who discovered it…Vilfredo Federico Damaso Pareto was an Italian engineer, sociologist, economist, and philosopher.
What Does “Pareto Optimal” Mean?
2D Pareto front schematics
What Does “Pareto Optimal” Mean?•Usually, the current approx. of the true Pareto front.
•The Pareto front defines the “zero sum game” of all optimal compromises you could make.
•Unlike a weighted combination of objective functions, it lets you choose a specific compromise/ combination AFTER the optimization is complete.
• It does NOT say which compromise/combination is best, just what all the “optimal” choices are.
• It says “Don’t choose anything Pareto non-optimal because there’s something better in all criteria.”
Finding a “Pareto Optimal” Ensemble•Used the Multi Objective Genetic Algorithm (MOGA) in DAKOTA’s (Design Analysis Kit for Optimization and Terascale
Applications) JEGA (John Eddy’s Genetic Algorithm) sub-package
•GA’s typically need 1000’s of simulations, I could only afford 1000…
•Used test problem (find surface of radius=1 6D hyper-sphere in input space, 10 outputs) to tune MOGA settings and initial population (space-filling, specifically Binning
Optimal, Symmetric Latin Hypercube Sampling, or BOSLHS), for:• Large Pareto Ensemble
• Mean radius close to 1
• Uniform spread
• Small radius variance
Finding a “Pareto Optimal” Ensemble
Use DAKOTA’s MOGA on a test problem with 6 inputs and 10 outputs; true solution is a radius 1 hypersphere
Default Monte Carlo seed
PDF’s of the Pareto Ensemble’s
1. # of points
2. Point spread
3. Mean radius
4. Standard deviation of radius
1 2
3 4
Finding a “Pareto Optimal” Ensemble
Use DAKOTA’s Multi Objective Genetic Algorithm on a test problem with 6 inputs and 10 outputs
true solution is a radius 1 hypersphere
BOSLHS seedDefault Monte Carlo seed
Results of Tuning Climate Model
Summary & Future Work•Climate model parameters that match history well might not predict well (climate change might expose a previously hidden mis-calibration of parameters).
•Plan: Use a diverse ensemble of “good” (Pareto optimal) parameter sets to determine the range/spread of possible future climates.
•Used MOGA to find a (very large) Pareto optimal ensemble of calibrated parameter sets.
•Next steps: –down select Pareto optimal ensemble, and–simulate smaller ensemble out to 2100.
Some “Good” Parameter SetsInputs % change in output mismatch relative to CCSM4 defaultRHMINL RHMINH ALFA TAU [hrs]C0 [m -̂1] KE [(m^2s/kg) 0̂.5/s]TREFHT T U PS RELHUM LHFLX LWCF SWCF PRECT RADBAL
0.9348 0.7941 0.5527 4.426 3.445E-3 6.883E-6 10.5869 1.33 6.515 0.64 -9.13728 38.049 -33.32 -20.5 50.546 -45.9170.9348 0.8789 0.3379 3.020 4.711E-3 7.867E-6 4.73702 0.42 9.581 0.75 23.3127 31.136 -26.88 -31.94 49.582 -65.1220.9393 0.6727 0.2348 2.316 5.086E-3 4.148E-6 -0.309 3.88 11.42 1.44 -15.2271 6.6722 3.5036 14.204 13.99 -99.3760.9496 0.7055 0.2365 6.652 5.836E-3 7.211E-6 14.4249 3.39 8.135 0.99 -29.4557 45.088 -31.76 -29.09 49.752 -73.4570.9277 0.8039 0.3483 3.254 4.383E-3 3.383E-6 1.38947 0.47 1.366 1.22 -10.3006 30.064 -27.13 -16.6 46.626 -65.0480.9316 0.7992 0.0715 2.151 5.133E-3 8.195E-6 -4.1146 1.59 14.62 1.12 44.5008 -2.331 0.5983 24.616 4.9049 -99.9920.9348 0.6586 0.2090 2.316 5.273E-3 3.383E-6 -0.2539 3.38 7.114 0.62 -23.2472 5.3236 2.0664 20.214 11.527 -61.240.9348 0.8610 0.3379 3.020 4.711E-3 7.867E-6 1.34654 0.67 8.201 1.2 19.6143 24.863 -31.78 -26.22 40.619 -85.4330.9393 0.6982 0.2348 2.316 5.086E-3 4.148E-6 3.17856 3.13 15.23 1.61 -11.5716 5.8576 -1.338 6.613 14.649 -90.4690.9418 0.7371 0.2365 6.652 5.836E-3 7.211E-6 13.0888 2.74 8.034 0.39 -26.0691 51.814 -30.86 -25.73 49.191 -84.6720.9316 0.7795 0.0715 2.151 5.133E-3 8.195E-6 -5.7248 2.6 19.24 0.81 35.4784 -1.642 4.6604 31.289 1.5978 -78.5710.9324 0.7362 0.0973 0.910 4.570E-3 9.617E-6 -1.7177 2.77 27.21 0.37 27.8201 -15.75 30.85 45.384 -7.412 -99.8090.9324 0.8320 0.1171 1.848 4.570E-3 8.523E-6 -1.021 2.58 27.97 0.45 58.6333 0.156 -6.863 13.883 5.2161 -92.776
CCSM4 default 0.91 0.8 0.1 1 3.500E-3 1.000E-6 0 0 0 0 0 0 0 0 0 0
Range0.95 0.9 0.6 8 6.000E-3 1.000E-50.8 0.6 0.05 0.5 3.000E-3 3.000E-6