The Aerospace Corporation (Aerospace)
Prelaunch Assessment of the Northrop Grumman VIIRS Cloud Mask
Thomas Kopp, The Aerospace CorporationKeith Hutchison, Northrop GrummanAndrew Heidinger, NOAA/STARRichard Frey, University of Wisconsin
IGARSS25 July 2011
[email protected] Satellite Systems
Outline
• Definitions of VIIRS Cloud Mask (VCM) contents and validation conditions
• High level review of the VCM logic• Global results with the pre-launch VCM without any tuning• Quantitative Improvements Using the Northrop Grumman (NG)
tuning tool• Methods for evaluating individual granules during Intensive Cal/Val
(ICV) of the VCM
[email protected] Satellite Systems
VCM Contents
• The VCM itself determines one of four cloud cover conditions for each pixel– Confidently Cloudy– Probably Cloudy– Probably Clear– Confidently Clear
• All downstream EDR products, except for imagery, require the VCM as an input
• Downstream products will use either the confidently cloudy or confidently clear condition– The probably clear/cloudy cases account for pixels that are not completely
cloud covered but due either to the difficulty of the scene or partial clouds such as cumulus, are not sufficiently clear to reliably determine the conditions at the surface
[email protected] Satellite Systems
VCM Performance Metrics
•Probability of Correct Typing (PCT)− PCT = (1 - Binary Cloud Mask Error) = {1 – [(VCM = conf. clear) & (Truth = conf.
cloudy) OR ((VCM = conf. cloudy) & (Truth = conf. clear)]/[total #pixels in each geographic class – PCPC]
•Cloud Leakage (CL) − CL = [(VCM = conf. clear) & (Truth = conf. cloudy)]/total #pixels in the geographic
class•False Alarm Rate (FA)
− FA = [(VCM = conf. cloudy) & (Truth = conf. clear)]/total #pixels in the geographic class
•Fraction of Pixels Classified as Probably Clear/Cloudy (FPCPC)– FPCPC = [(VCM = prob. clear) or (VCM = prob. cloudy)]/total #pixels in the
geographic class
[email protected] Satellite Systems
Overview of VCM Approach
• There are five possible processing paths in the VCM algorithm for the analysis of SDR data collected in daytime conditions
Cloud Spectral Tests Used to Determine Cloud Confidence Water Land Desert Coast Snow
1. M9 (1.38 µm) Reflectance Test X X X (if TPW > 0.25
cm)
X X
2. M15-M16 (10.76 – 12.01 µm) Brightness Temperature Difference (BTD)
X X X X
3. Tri-Spectral M14, M15, M16 (8.55, 10.76, 12.01 µm) BTD Test
X
4. M15-M12 (10.76-3.70 µm) BTD Test
X(if no sun
glint)
X(if TOC NDVI >
0.2)
X(if Lat > 60 or < -
60)
X(if no sun glint
and if TOC NDVI > 0.2)
X
5. M12-M13 (3.70-4.05 µm) BTD Test
X (if –60 < Lat < 60) and no sun glint
X(if –60 < Lat < 60) and TOC
NDVI > 0.2
X(if –60 < Lat < 60)
6. M1 (0.412 µm) Reflectance Test X(if –60 < Lat <
60) 7. M5 (0.672 µm),
M1 (0.412 µm) Reflectance TestsX
(M5 if TOC NDVI ≥ 0.2;
M1 otherwise)
X(M5 if TOC NDVI ≥ 0.2;
M1 otherwise)
8. M7 (0.865 µm) Reflectance Test X
9. M7/M5 (0.865 / 0.672 µm) Reflectance Ratio Test
X X(if RefM5 ≥
LD_M5_Gemi Thresh)
Cloud Spatial Tests Used to Modify the Final Cloud Confidence Classification Water Land Desert Coast Snow
10. I5 (11.45 µm) Spatial Test X
11. I2 (0.865 µm) Reflectance Test X
[email protected] Satellite Systems
Pre-launch, Pre-tuned Global VCM Results
•VCM version used 1.5.0.48 from 2009•The initial thresholds were used, the VCM for this testing was not tuned•Comparisons made with collocated MOD35 C6 cloud mask and CALIOP matchups for comparison
– Cloudy for the VCM in this case included probably cloudy pixels– Clear for the VCM in this case included probably clear pixels
• Compared only 1-km CALIOP segments with either 0% or 100% cloud cover– Resulted in approximately 15 million collocations per month
[email protected] Satellite Systems
Caveats and Notes to Global VCM Results
•Results intended to show “where we were” in late 2009•Neither of the two I-band tests could be simulated using the proxy data, a significant source of error that will not be quantified until the post-launch validation of the VCM•Thin cirrus has a major impact on the results•Analysis limited to near-nadir views (MODIS viewing zenith angle of +/- 20 degrees)•Hit rate = (# agree cloud + # agree clear) / total #• Hanssen-Kuiper Skill Source (HKSS) = (# agree cloud * # agree clear) –
(# disagree cloud * # disagree clear) / (# agree clear + # disagree clear) * (# agree cloud + # disagree cloud) •Results follow on the next few slides
[email protected] Satellite Systems
VCM Designed to Exploit VIIRS 1.38-µm Data
MODIS vs VIIRS RSRs MODIS vs VIIRS TOA Radiances
VIIRS OOB Response is
orders of magnitude
less
MODIS OOBResponse is as large as the in-band response
Thin cirrus clouds will be more readily detected with VIIRS data than in MODIS
[email protected] Satellite Systems
VCM Versus Heritage Performance, COT > 1.0
VCM and heritage performance are comparable when thin cirrus clouds are eliminated from the results
[email protected] Satellite Systems
One Year Means of Hit Rates and Skill
July 2007 – June 2008, Comparison with CALIOP
Scene Type Mean VCM Hit Rate
Mean MOD35 C6 Hit Rate
Mean VCM HKSS
Mean MOD35 C6 HKSS
Global 79.0 86.9 62.0 72.51
60S-60N 84.1 89.9 74.0 78.1
Global day 83.4 89.0 66.1 76.3
Global night 75.1 85.1 59.6 69.1
60S-60N Water day 87.9 90.9 77.7 79.8
60S-60N Water night 83.2 90.5 76.4 76.0
60S-60N Land day 80.7 88.2 63.8 76.6
60S-60N Land night 80.1 87.4 65.0 74.7
[email protected] Satellite Systems
Pre-Launch Tuning Approach
• Pre-launch tuning is based on 14 granules which employed Global Synthetic Data (GSD)– Of these 14, 11 contained land backgrounds
•These granules covered each VCM geographic type and ranged from straightforward to difficult scenes • GSD provides unique data to set the mid-point thresholds
– Typical methods of tuning, using on-orbit sensor data, rely upon 100% cloudy and 100% cloud free distributions
– GSD alone allows cloud distributions to be evaluated at the mid-point (50% cloudy) condition
•GSD allows setting thresholds and then minimize the distance between the confidently cloudy and confidently clear thresholds
[email protected] Satellite Systems
Advantages Using Global Synthetic Data (GSD)
• GSD allows testing of the tuning process with proxy data and then apply the procedure to VIIRS-unique data– Tuning process is validated by :
–(1) tuning with GSD truth data developed with MODIS Relative Spectral Responses (RSR)
–(2) tested in the VCM using MODIS granules –(3) quantitatively evaluated using manually-generated cloud
data of the MODIS data– Tuning for VIIRS data is then completed by examining changes
in cloud distribution for each test in GSD truth data using the VIIRS RSR
[email protected] Satellite Systems
Initial VCM Results Showed Following Needs
• Reduce the number of probably clear and probably cloudy (PCPC) classifications by adjusting the overall cloud confidence threshold
• Identify tests that generated the highest percentage of false alarms for each VCM background condition and tune the mid-point thresholds (i.e. 50% cloud cover condition) accordingly – only possible with GSD
• Further reduce the number of PCPC classifications, as necessary, by adjusting the distance between the mid-point thresholds of a given individual cloud test and the low and/or high threshold using cloud distributions in the GSD.
[email protected] Satellite Systems
Initial Untuned VCM Performance - Land
Granule ID2001152_1600
2001196_1755
2001213_1210
2001213_1220
2002001_0340
2002032_1745
2002032_1750
2002118_0215
2003194_0905
2003299_1835
2003299_1840 Summary
land 628336 1127120 45030 54432 1367600 689449 1022770 673794 937571 1249880 108676
nPoorQual 0 0 0 0 0 0 0 0 0 0 0
nCldTruth 594640. 446056 14489 26133 382203 415431 393467 414653 88968 171321 28760
nClrTruth 33696 681069 30541 28299 985394 274018 629301 259141 848603 1078560 79916
nConfCldy 590182 466534 13811 32590 486446 458338 461027 539672 43910 241635 28341
nConfClr 6351 222995 3182 543 156780 49701 170838 65656 450710 352476 34965
nPrbCldy 2660 8692 1037 1032 54349 36563 23768 14685 3290 22254 1251
nPrbClr 29143 428904 27000 20267 670022 144847 367135 53781 439661 633517 44119
FalseAlarms 11974 46230 1316 7723 131286 82537 93687 125723 1378 89767 2630
Leakage 472 1346 28 38 1313 538 672 64 13895 1315 111
BinaryError 12446 47576 1344 7761 132599 83075 94359 125787 15273 91082 2741
FPCPC 0.05 0.39 0.62 0.39 0.53 0.26 0.38 0.10 0.47 0.52 0.42 0.38
pFalseAlarms 0.02 0.07 0.08 0.23 0.20 0.16 0.15 0.21 0.00 0.15 0.04 0.12
pLeakage 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.03 0.00 0.00 0.00
pBinaryError 0.02 0.07 0.08 0.23 0.21 0.16 0.15 0.21 0.03 0.15 0.04
PCT 0.98 0.93 0.92 0.77 0.79 0.84 0.85 0.79 0.97 0.85 0.96 0.87
[email protected] Satellite Systems
Overview of the Pre-Launch Tuning Process
• Identify the tests causing the largest number of errors• Use GSD with MODIS RSRs to generate cloud cover
distributions for the cloud detection tests identified above– Generate distributions for 0%, 50%, and 100% cloud cover – Set key mid-point threshold using the 50% cloud cover, then
minimize low- and high thresholds• Update VCM using these thresholds• Execute the updated algorithm on the set of MODIS
granules• Evaluate the performance using the manually generated
cloud masks• Assess the changes in performance
[email protected] Satellite Systems
Example for a Case With Too Many PCPC PixelsMODA.2001.196.1755 Manually-Generated Mask
Qthresh = 99%
Land - Pre
FPCPC 0.39pFalseAlarms 0.07pLeakage 0.00pBinaryError 0.07PCT 0.93
Qthresh = 90%
Land - PostFPCPC 0.14pFalseAlarms 0.05pLeakage 0.01pBinaryError 0.06PCT 0.94
[email protected] Satellite Systems
Specific Cloud Detection Case, GEMI Test (Land)
Changed from 1.95 to 1.87 Changed from 1.90 to 1.82 Changed from 1.85 to 1.78
[email protected] Satellite Systems
Quantitative Impacts – GEMI ResultsM7/M5 Land628336 1127120 45030 54432 1367600 689449 1022770 673794 937571 1249880 108676 Summary
nPoorQual 0 0 0 0 0 0 0 0 0 0 0nCldTruth 594640 446056 14489 26133 382203 415431 393467 414653 88968 171321 28760nClrTruth 33696 681069 30541 28299 985394 274018 629301 259141 848603 1078560 79916nConfCldy 581516 408380 11854 20537 438229 404075 433527 466103 39542 211063 24899nConfClr 18824 650863 29320 21885 804817 183687 505091 134608 885731 960178 78081nPrbCldy 3268 8259 283 903 17190 23936 11252 12251 1490 11933 426nPrbClr 24728 59623 3573 11107 107361 77751 72898 60832 10808 66708 5270FalseAlarms 6113 11129 149 677 85209 55293 70980 53987 212 77525 984Leakage 2116 16834 1544 808 12483 15123 10395 298 39374 12827 2674BinaryError 8229 27963 1693 1485 97692 70416 81375 54285 39586 90352 3658FPCPC 0.04 0.06 0.09 0.22 0.09 0.15 0.08 0.11 0.01 0.06 0.05 0.07pFalseAlarms 0.01 0.01 0.00 0.01 0.06 0.08 0.07 0.08 0.00 0.06 0.01 0.05pLeakage 0.00 0.01 0.03 0.01 0.01 0.02 0.01 0.00 0.04 0.01 0.02 0.01pBinaryError 0.01 0.03 0.04 0.04 0.08 0.12 0.09 0.09 0.04 0.08 0.04 PCT 0.99 0.97 0.96 0.96 0.92 0.88 0.91 0.91 0.96 0.92 0.96 0.93
Previous untuned results
FPCPC 0.04 0.06 0.46 0.22 0.37 0.22 0.18 0.08 0.12 0.19 0.150.17
pFalseAlarms 0.01 0.01 0.01 0.01 0.08 0.11 0.08 0.19 0.00 0.06 0.010.06
pLeakage 0.00 0.01 0.00 0.01 0.00 0.00 0.00 0.00 0.04 0.00 0.010.01
pBinaryError 0.01 0.03 0.02 0.03 0.14 0.15 0.11 0.20 0.04 0.08 0.03
PCT 0.99 0.97 0.98 0.97 0.86 0.85 0.89 0.80 0.96 0.92 0.970.91
[email protected] Satellite Systems
Tests Improved by the Pre-Launch Tuning Effort
• Reflectance test over desert (M1)• Reflectance test over land (M5)• Reflectance test over water (M7)• Ratio test over land (GEMI)• Ratio test over water (M7/M5)• Mid-Wave minus long wave infrared over snow (M12 – M15)• Mid-Wave infrared difference over snow (M12 – M13)
[email protected] Satellite Systems
Performance After Daytime Tuning CompletedPerformance
MeasureLand Ocean Desert Snow
PCPC
SysSpec
38.3
15%
19.2
15%
14.7
15%
58.0
15%
False Alarms
SysSpec
12.4
7.0
6.0
5.0 10.5 22.5
Leakage (%)
SysSpec
0.5
3.0
0.3
1.0 2.3 0.02
PCT (%)
SysSpec
87.1
90.0
93.7
94.0 87.2 77.4
Performance Measure
Land Ocean Desert Snow
PCPC
SysSpec7.5
15%
22.5
15%
3.5
15%
5.1
15%
False Alarms
SysSpec4.6
7.0
1.4
5.0 2.1 3.7
Leakage (%)
SysSpec1.4
3.0
0.6
1.0 3.7 0.9
PCT (%)
SysSpec93.4
90.0
97.5
94.0 93.9 95.1
Untuned VCM: March 2010 Tuned VCM: November 2010
[email protected] Satellite Systems
Tool for Visualization of the VCM
• The previous analyses reveal quantitative aspects of the VCM, but lack context
• Historically the capability to visualize the output from each individual cloud detection test has been used operationally at the Air Force Weather Agency
• Key to a useful visualization are two fundamental factors– It must overlay each test on applicable imagery– It must contain the reflectance/brightness temperatures used within the
cloud mask• This reveals if any bands have bad or saturated values
• The visualization should also note if any degraded conditions of note exist in the scene– These include aerosols, sun glint, and shadows
• The following pair of slides show this capability
[email protected] Satellite Systems
Conclusion
• Pre-launch validation of the VCM uses three different approaches to verify the VCM will meet expectations– Large scale quantitative analysis– Small scale quantitative analysis via GSD– Visualization of individual granules with each component cloud
detection test• Results show promise that the VCM will meet or exceed its
requirements• Each of these methods will be employed in some form
post-launch, though we will no longer need GSD as actual VIIERS data will be available