C L

EarthCARE Level 2 DocumentationEarthCARE Level 2 Documentation

ACM-CAP Cloud, Aerosol andACM-CAP Cloud, Aerosol and Precipitation “Best Estimate”Precipitation “Best Estimate”

C L

Final ReportFinal Report

VARSY ProjectVARSY Project

C L

Code: L2b-ACM-CAP-FRIssue: 01Date: 12/09/2013Reference: University of Reading

Name Function Signature

Prepared by Robin Hogan Project Scientists

C L

Reviewed by Pavlos Kollias Project Scientist

Approved by Pavlos Kollias Project Manager

Signatures and approvals on original

C L

C L

This page intentionally left blank

C L

Document InformationDocument InformationContract Data

Contract Number: 4000104528/11/NL/CTContract Issuer: ESA-ESTEC

C L

Internal DistributionName Unit Copies

Robin Hogan University of Reading 1Internal Confidentiality Level

Unclassified Restricted Confidential

External Distribution

C L

Name Organisation Copies

Tobias Wehr ESA-ESTEC 1 Michael Eisinger ESA-ESTEC 1Dulce Lajas ESA-ESTEC 1Pavlos Kollias McGill University 1Julien Delanoë LATMOS 1Gerd-Jan van Zadelhof KNMI 1

C L

David Donovan KNMI 1Alessandro Battaglia University of Leicester 1

Archiving

Word Processor: MS Word 2003File Name: VARSY-READING-FR-001

C L

Document Status LogDocument Status Log

Issue Change description Date Approved

01 First version delivered to ESTEC 12/09/2013

C L

C L

Table of ContentsTable of Contents1. PURPOSE AND SCOPE____________________________________________________________7

2. DOCUMENTS____________________________________________________________________82.1. Applicable Documents______________________________________________________________8

2.2. Reference Documents_______________________________________________________________9

2.3. List of Abbreviations_______________________________________________________________10

C L

3. INTRODUCTION________________________________________________________________11

4. ACHIEVEMENTS WITHIN THE PROJECT_________________________________________12

5. Current Status_____________________________________________________________________13

C L

List of TablesList of Tables

Table 1: Applicable Documents..........................................................................................................................8

C L

Table 2: Reference Documents...........................................................................................................................9

Table 3: List of abbreviations...........................................................................................................................10

C L

1. 1. PURPOSE AND SCOPEPURPOSE AND SCOPE

This document briefly summaries the work performed at Reading within VARSY project. The

specific work that has been performed revolved around the development of the software for the ACM-

CAP product.

C L

C L

2. 2. DOCUMENTSDOCUMENTS

2.1. 2.1. Applicable DocumentsApplicable Documents

The following table specifies the applicable documents that shall be complied during the project

development.

C L

Table 1: Applicable Documents

C L

Reference

Code Title Issue

[SOW] EC-SW-ESA-SY-0310 Statement of Work: VARSY - 1-Dimensional VARiational Retrieval of SYnergistic EarthCARE Products

1.0

[CC] Appendix 2 to AO/1-6823/11/NL/CT

Draft Contract (attachment to SOW) 1.0

[AD 1] EC-SW-ESA-SY-0152 EarthCARE Level 2 Processor Development

General Requirements Baseline

1.0

C L

Reference

Code Title Issue

[AD 2] EC.ICD.ASD.SY.00004 EarthCARE Product Definitions. Vol. 0:

Introduction

[AD 3] EC.ICD.ASD.SY.00005 EarthCARE Product Definitions. Vol. 1:

Common Product Definitions

1.0

[AD 4] EC.ICD.ASD.ATL.00021 EarthCARE Product Definitions. Vol. 2b: 1.0

C L

Reference

Code Title Issue

ATLID level 1

C L

Reference

Code Title Issue

[AD 5] EC.ICD.ASD.BBR.00022 EarthCARE Product Definitions. Vol. 3b:

BBR level 1

1.0

[AD 6] EC.ICD.ASD.MSI.00023 EarthCARE Product Definitions. Vol. 4b:

MSI level 1

1.0

[AD 7] ECSIM-DMS-TEC-ICD01-R ECSIM Simulator Interface Control Document

C L

Reference

Code Title Issue

[AD 8] PE-TN-ESA-GS-0001 Ground Segment: File Format Standard 1.0

[AD 9] EC-TN-ESA-GS-0218 Tailoring of the Earth Explorer File Format Standard for the EarthCARE Ground Segment

2.0

C L

2.2. 2.2. Reference DocumentsReference Documents

The following table specifies the reference documents that shall be taken into account during the

project development.

C L

Table 2: Reference Documents

Reference

Code Title Issue

[RD1] ECSIM-DMS-TEC-SUM-01-R

ECSIM System User Manual

[RD2] ECSIM-KNMI-MAD01-R ECSIM Model and Algorithms Document

[RD3] EE-MA-DMS-GS-0001 Earth Explorer Mission CFI Software:

C L

Reference

Code Title Issue

General Software User Manual

[RD4] EOP-SM/1567/TW EarthCARE Mission Requirements Document

C L

Reference

Code Title Issue

[ATLAS-FR] EC-FR-KNMI-ATL-027 ATLAS Final report 1.0

[ATLAS-ACM-TC]

EC-TN-KNMI-ATL-ACM-TC-024 L2b Classification ATBD

1.213/03/08

[ATLAS-EBD]

EC-TN-KNMI-ATL-ATBD-A-EBD-021

L2a ATLID Extinction, Backscatter and Depolarization algorithm ATBD

1.1 27/04/09

C L

Reference

Code Title Issue

[ATLAS-FM]

EC-TN-KNMI-ATL-ATBD-A-FM-010

L2a ATLID Feature mask ATBD 2.2

[RATEC-FR]

RATEC-FR-READING-1 RATEC Final Report 1.0, April 2011

C L

2.3. 2.3. List of AbbreviationsList of Abbreviations

Table 3: List of abbreviations

Abbreviation Name

C L

1D-VAR RS 1-dimensional variational retrieval scheme

ATLID Atmospheric Lidar (The EarthCARE lidar)

CASPER Cloud and Aerosol Synergetic Products from EarthCARE retrievals

CPR Cloud Precipitation Radar (The EarthCARE radar)

EarthCARE The Earth Clouds, Aerosols and Radiation Explorer

ECSIM EarthCARE Simulator

C L

HSRL High-Spectral Resolution Lidar

MSI Multi-spectral Imager (The EarthCARE imager)

C L

3. 3. INTRODUCTIONINTRODUCTION This document summarizes the work carried out at the University of Reading during the VARSY activity in the development of the “Cloud, Aerosol and Precipitation Best Estimate” product. It should be read in combination with the Product and Algorithm Requirements Document [PARD], which contains an extensive review of the previous relevant synergy retrieval work, the Product Definition Document [PDD], describing the product, and the Algorithm Theoretical Basis Document [ATBD], which describes in detail how the algorithm works. Given the high level of detail in these other (longer) documents, section 4 contains a briefer overview of the work and several short accounts of contributing studies that assisted in the ongoing development of the Best Estimate algorithm. Section 5 then

C L

summarises our progress in the development of the algorithm since the end of the previous RATEC project, along with the remaining work to be done.

C L

4. 4. ACHIEVEMENTS WITHIN THE PROJECTACHIEVEMENTS WITHIN THE PROJECT

4.1. 4.1. One-sided gradient constraint for liquid cloudsOne-sided gradient constraint for liquid cloudsAn important physical constraint for liquid clouds is that the gradient of liquid water content with height should not exceed the adiabatic value, which is a function of temperature and pressure. In the [ATBD] it is explained how a “one-sided gradient constraint” has been added, which adds a term to the cost function if this gradient is exceeded. Here we evaluate the liquid-cloud part of the retrieval by comparing it to independent data.

C L

C L

C L

C L

Figure 1. (from top) Calipso lidar backscatter observed for a stratocumulus cloud observed over ocean; the forward-modelled values at the final

iteration of a retrieval using ACM-CAP making use only of the lidar; the

C L

corresponding retrieved liquid water content; the corresponding atmospheric optical depth at the lidar wavelength.

Figure 1 depicts a retrieval of the liquid water content using purely the Calipso lidar return. A one-sided gradient constraint has been used. It can be seen that by forward modelling the multiple scattering signal of the lidar, optical depths to 60 are retrieved; if it were not for multiple scattering, retrievals above an optical depth of around 5 would not be possible.

Independent verification of these retrievals is provided in Figure 2a, which shows a very good agreement between observed and simulated radar path-integrated attenuation; this variable is

C L

proportional to liquid water path. The one location where the agreement is not so good is in the region of highest PIA; presumably this is where the optical depth is so high that even with multiple scattering, there is not enough information in the multiply scattered returned light to infer the full optical depth of the cloud. Figure 2b demonstrates that this may be overcome by assimilating also the PIA in the retrieval.

C L

C L

Figure 2. Comparison of the observed and forward modelled radar-path integrated attenuation (which is proportional to liquid water path) for the

C L

retrieval shown in Figure 1. The top panel shows a comparison for the retrieval using only the lidar, while the bottom panel shows a retrieval

where the path-integrated attenuation was assimilated too.

These results are for the Calipso lidar, which has a field of view of around 90 m. The EarthCARE lidar has a much narrower field of view, which reduces the strength of the multiple scattering effect that is providing most of the information on optical depths above around 5. We have performed simulations and find that there is a critical field of view of around 50 m, below which it is no longer possible to use multiple scattering to infer the optical depth of thick clouds. Unfortunately, EarthCARE has a smaller value than this. Therefore, the retrieval will need to rely more on the other information available, such

C L

as the solar radiances (in the day only), the path integrated attenuation (useful over oceans only), the HSRL capability of the lidar and of course the gradient constraint.

4.2. 4.2. Automatic differentiation library “Adept”Automatic differentiation library “Adept”An outstanding question at the end of the RATEC project and the start of the VARSY project was the problem of coding up by hand all the adjoints of the various components of the retrieval algorithm, given how time consuming and error prone this activity is. As solution has been found in that we have developed a new automatic-differentiation software library “Adept”, which is easy to incorporate into an existing C++ project and can compute both adjoints and Jacobian matrices only slightly slower than it would take a hand-coded function (although without the long and difficult process of doing the hand-

C L

coding). Note that other automatic-differentiation libraries are available that build on the same operator-overloading idea (e.g. ADOL-C, CppAD and Sacado), but they are all considerably slower than Adept. This work is in the process of being published (Hogan 2013, submitted to ACM Trans. Math. Softw.), and the software has been released at http://www.met.reading.ac.uk/clouds/adept/.

Very recently we have been working on a parallelized version of the Jacobian computation in Adept, using OpenMP; this exploits the multi-core architecture of many modern workstations. This will speed up the retrieval substantially. It will also make it more robust, as it would be coupled with a move to the Levenberg-Marquardt minimization scheme, which uses the curvature of the cost function in addition to its gradient which makes it easier to find the minimum of the cost function.

C L

4.3. 4.3. Storage of error descriptorsStorage of error descriptorsIn previous projects there was considerable discussion about how to compress the information on errors, which for N retrieved variables is in the form of an NxN error covariance matrix and an NxN averaging kernel matrix. We now routinely report

(a) 1-sigma errors; these are the square-root of the diagonal of the error covariance matrix

(b) Error correlations between variables at the same height; these are derived by normalizing specific off-diagonal elements of the error covariance matrix

C L

(c) The area of the averaging kernel for a particular variable; this is found by integrating a row of the averaging kernel and indicates what fraction of the information for a particular retrieved variable is from the observations, as opposed to the a-priori constraints

(d) The spatial width of the peak in the averaging kernel; this indicates the extent to which the nature of the observations have smoothed the retrieval at each height.

An example for a simulated ice cloud profile retrieved by radar and lidar is shown in Figure 4. In this example, the radar and lidar both sampled the cloud down to 4 km, below which it was sampled by radar alone. It can be seen that the error rises rapidly in the radar-only region, particularly for the number concentration parameter N0’ which transitions back to the a-priori value when only one instrument is available. The errors in extinction and ice water content are found to be well correlated, but less so between extinction and either particle size or number concentration. The right panel shows

C L

that down to 4 km, the averaging kernel area is close to 1, indicating that most of the information comes from the observations, but below this height, where the lidar no longer detects the cloud, the retrieval of number concentration parameter N0’ in particular is forced to rely more on the a-priori estimate so the area of the averaging kernel falls.

C L

Figure 3. Example of error descriptors for a retrieved ice cloud profile: (left)

C L

vertical profile of percentage errors in various variables; (centre) error correlation between various pairs of variables; (right) area of averaging

kernel for two variables.

4.4. 4.4. New model for radar scattering by snowflakesNew model for radar scattering by snowflakesAt the start of the VARSY project, the radar scattering model by ice particles was oblate spheroids, based on the findings of Hogan et al. (2012, J. Appl. Meteorol. Soc.). However, this model works only

C L

up to sizes equal to the wavelength. For larger particles, the internal structure of the ice particle, which is ignored in the oblate spheroid model, becomes important. This presents a problem because all ice particles are different and it is very time consuming to perform “discrete dipole approximation” calculations on a large number of representative ice particle shapes.

We have made a breakthrough in characterizing ice particle shape that allows a formula to be derived for the mean backscatter cross section of an ensemble of particles of the same size but different internal morphologies. This makes use of the Rayleigh-Gans scattering approximation, which recent work shows to be valid for low-density snowflakes at millimetre wavelengths, where the shape of the particle observed by a vertically pointing radar only needs to be described by a function A(z), which is the area of ice in a horizontal plane through the particle at height z.

C L

The parameters of the model have been derived by examining 50 ice aggregates generated by the Westbrook aggregation model. One such aggregate is depicted in Figure 4a. The solid black line in Figure 4b shows the mean A(z) function, and a simple parametric fit by the dashed line. The smaller-scale features in the A(z) function have been found to follow a power law that may be characterized by two coefficients. This enables an analytic expression to be found for the backscatter cross section.

C L

C L

Figure 4. (left) Example ice aggregate from the Westbrook aggregation model, viewed with its longest axis horizontal (representing the ordinary fall mode of ice particles), where the shading is proportional to how much mass is present integrated along the third dimension. (right) The area of ice A(z) intersected by a horizontal plane through ice aggregates as a function of

vertical distance z, where the mean area of 50 aggregates is shown by the solid black line, and a parametric fit is shown by the dashed line. The dotted

line shows the specific case of the particle shown in the left panel.

Figure 5 depicts the mean backscatter cross section from both the ensemble of aggregates (using the Rayleigh Gans approximation) and the new equation. It can be seen that the new equation generally fits

C L

very well, except at the far right of the plot corresponding to very high frequencies. At this point, the wavelength is smaller than the individual crystals that make up the aggregate, so the power law breaks down. This situation does not occur for ice aggregates at 94 GHz, so the new formula is suitable to use for both CloudSat and EarthCARE. To illustrate the importance of the effect of internal structure in the particles, consider a 1cm particle at 94 GHz, which corresponds to kDz=20 in Figure 5. It can be seen that here the backscatter of the aggregates is around 2 orders of magnitude larger than the backscatter of a homogeneous spheroid with the same mass and size.

This new scattering model has been implemented into the retrieval algorithm and it has been found that it approximately halves the ice water content and extinction coefficient retrieved from the radar alone in regions of high radar reflectivity factor, such as in the first few kilometres above the melting layer.

C L

C L

Figure 5. Mean backscatter cross section versus normalized wavenumber (where k=2/ and is the wavelength) for the 50 aggregates (solid black

line) and the new equation (dashed line).

4.5. 4.5. Retrieval of riming factorRetrieval of riming factorIn principle, the EarthCARE Doppler velocity provides information on the density of ice particles, since particles with the same ice mass (and hence radar reflectivity) will fall faster if that mass is concentrated

C L

in a smaller volume. The density of ice particles is largely controlled by riming, i.e. accretion of supercooled water droplets in mixed-phase clouds. With reasonable observational support, the algorithm currently assumes the Brown & Francis mass-size relationship for ice clouds, applicable when riming is not present. Figure 6 depicts fall speed as a function of mass and size, and it can be seen that even for large particles, the Brown & Francis relationship corresponds to fall speeds no higher than around 1 m/s.

We have added the capability to retrieve a “riming factor”, which scales the exponent of the mass-size relationship such that a riming factor of 0 corresponds to Brown & Francis while a value of 1 corresponds to solid ice. It can be seen that for larger particles the fall speed, and hence Doppler velocity, increases monotonically with riming factor, indicating that riming factor can be retrieved if Doppler velocity is available.

C L

Figure 7 shows a simulated retrieval of riming factor, where the top row shows the “truth”, the bottom row shows the simulated observations and the red dashed lines compare the retrieval to the truth and the simulated observations. In this case the riming factor of up to 0.5 is successfully retrieved. Work is still required to evaluate this approach, and in particular its applicability in that often riming occurs when there are significant vertical motions present, in which case the Doppler velocity could not be interpreted unambiguously in terms of terminal fall speed.

C L

C L

Figure 6. The colours show the terminal fall speed of ice particles according to the Heymsfield and Westbrook model, as a function of their diameter and

their ice fraction (the fraction of a circumscribing sphere if this diameter that contains ice). The black line shows the default Brown & Francis mass-

size assumption. The blue lines show a parameterization for how ice fraction is allowed to vary in a retrieval representing riming, where the number

shown is the “riming factor”.

C L

C L

C L

Figure 7. Simulated retrieval of riming factor. The blue lines in the top row show the “truth”. These have been used to forward model the simulated EarthCARE observations on the bottom row, with noise. The retrieval has

been run, with the dashed red lines on the bottom panels showing the forward modelled observations at the final iteration, and the dashed red

lines on the top panel showing the retrieved cloud/rain variables. The grey lines depict the individual iterations of the algorithm.

C L

4.6. 4.6. Retrieval of aerosol properties using a KalmanRetrieval of aerosol properties using a Kalman smoothersmootherThe HSRL offers the potential to unambiguously retrieve the vertical profile of extinction coefficient, without any assumptions on the nature of the particles. However, in the case of aerosol which has a very low optical depth, it is challenging to extract the signal from the noise. The approach taken in the lidar-only L2a algorithms is to average the ATLID data first, but for the ACM-CAP retrieval, this is not really compatible with the retrieval of the other atmospheric constituents which can be retrieved independently on a profile-by-profile basis. Therefore, the capability has been added to use a “Kalman smoother” for any retrieved variable; currently this is forward-only, meaning that each profile uses the retrieved values at the previous profile as an additional constraint, which has a smoothing forward-in-time only. Soon a

C L

forward-reverse Kalman smoother will be added, in which the retrieval is repeated in the reverse direction, at which time the retrieval is constrained by both the previous and subsequent profiles, leading to smoothing that is symmetric in time.

Figure 8 demonstrates the performance of the Kalman smoother on Calipso data in comparison to retrievals with no smoothing and only vertical smoothing. It can be seen that it successfully removes the noise arising from the noisy backscatter measurements. Further work would obviously be required to test whether it can successfully do the same in HSRL lidar observations.

C L

C L

C L

C L

Figure 8. Demonstration of the Kalman smoother for aerosol retrievals. (from top) Observed Calipso backscatter; retrieved aerosol mass content

with each pixel retrieved separately; retrieved aerosol mass content with a vertical smoothness constraint; retrieved aerosol mass content with a

C L

vertical smoothness constraint and a forward-in-time Kalman smoother.

C L

5. 5. CURRENT STATUSCURRENT STATUSConsiderable effort has been put into determining the best structure for the “Best Estimate” algorithm, to ensure that despite its size and complexity it is still flexible, efficient and elegant. The framework is as modular as possible, with a matrix library “Marvel” providing general functionality in linear algebra and an automatic-differentiation library “Adept” to enable gradients to be computed, on top of which is built a library providing the functionality for performing 1D variational retrievals with multiple instruments. The forward models are all in the form of separate libraries.

C L

For full details of the workings of the code, see [ATBD]. Here we provide a summary of the status of each component of the unified retrieval algorithm, describing not only what has been implemented but also what work remains to be done. The typefaces refer to:

Features implemented since the end of the RATEC project. Essential features to be added. Desirable features.

5.1. 5.1. Core codeCore code Automatic differentiation using the “Adept” replaces the original numerical differentiation,

resulting in considerable speed-up.

C L

The L-BFGS minimization method now uses a free liblbfgs library (incorporated into package), which is faster than the implementation in the GNU scientific library.

Error estimates and averaging kernel information are now computed and stored in the output files.

Optimized calculation of the prior contribution to the cost function by exploiting the sparseness of the inverse of the prior error covariance matrix, B-1, and the Twomey-Tikhonov matrix, T.

Implemented Levenberg-Marquardt minimization (a more robust version of Gauss-Newton). Add support for ESA-compliant XML input files and HDF output files (in addition to the

formats already supported).

C L

Implement parallelized Adept computation of Jacobian matrices using OpenMP, which will considerably speed up the Levenberg-Marquardt method and make it more attractive than L-BFGS.

5.2. 5.2. Scattering lookup tablesScattering lookup tables Applied automatic differentiation to the scattering look-up procedure. Consider cubic interpolation; the current linear interpolation may be the cause of the failure to

formally converge in significant fraction of profiles.

C L

5.3. 5.3. Lidar forward modelLidar forward model Debugged the adjoint for the small-angle multiple scattering model, but automatic

differentiation is now used instead. HSRL signals can be assimilated, including the effects of multiple scattering. Incorporate lidar depolarization forward model.

5.4. 5.4. Radar forward modelRadar forward model Verified the adjoint of the single scattering forward model, but automatic differentiation now

used instead.

C L

Enhance the multiple scattering models to predict the multiple scattering enhancement of the surface return, for making path-integrated attenuation estimates.

Incorporate multiple-scattering effects in the Doppler forward model (but note that in these situations the vertical wind will be significant so perhaps the Doppler velocity will not be usable to infer microphysical properties)

5.5. 5.5. Infrared radiance forward modelInfrared radiance forward model Implemented the Delanoe and Hogan radiance model. Implemented automatic differentiation of this model.

C L

5.6. 5.6. Shortwave radiance forward modelShortwave radiance forward model “scatter” program now can compute full Legendre expansion of phase function, which can be

treated by scattering look-up table module. Interface LIDORT radiance model to the algorithm.

5.7. 5.7. Ice retrievalIce retrieval Incorporated aspects of Delanoe and Hogan algorithm: retrieve number concentration parameter

with suitable a-priori, allow vertically varying lidar ratio to be retrieved, and included oblate ice scattering model for radar.

C L

Added capability to retrieve an ice-density multiplication factor in deep convection and other riming clouds to simulate riming ice; this uses information from the Doppler velocity.

Now calculate and report snow flux (kg m-2 s-1) for all ice clouds. Compare results to Delanoe and Hogan algorithm in ice clouds.

5.8. 5.8. Liquid cloud retrievalLiquid cloud retrieval Implemented and verified the one-sided gradient-constraint approach. Add number concentration retrieval (infrastructure already written) when solar radiance

information is available.

C L

5.9. 5.9. Rain retrievalRain retrieval Currently retrieve rain rate using Beard (1976) raindrop fall-speed model. Path integrated attenuation (PIA) is forward modelled but not yet assimilated. Added appropriate smoothness constraint to ensure retrieval does not jump to a-priori in ground

clutter. Retrieve number concentration parameter when PIA available for assimilation

(infrastructure has been written). Cope with deep convection where the rain signal is strongly affected by multiple scattering

and it is best to assume that the rain properties are constant with height.

C L

5.10. 5.10. Melting layer retrievalMelting layer retrieval Implemented very simple melting layer model: a rain-rate dependent radar attenuation. It is possible to retrieve a multiplying factor for the thickness of the melting layer, to account for

the fact that subtleties in the thermodynamic profile can change the rate of melting. Added term to cost function penalizing difference between snow flux above and rain flux

below the melting layer, but it is not working properly – fix.

5.11. 5.11. Aerosol retrievalAerosol retrieval New formulation: retrieve number concentration with size approximately fixed.

C L

Implemented a forward Kalman smoother, but need to modify this to work in both directions so that smoothing is symmetric in time.

Test impact of solar radiances. Forward model the lidar ocean-surface return as an optical depth constraint.