Emerging Technologies and Methods in Earth Observation for
Agricultural MonitoringFeb. 13, 2018
National Agricultural Library
USDA Foreign Agricultural ServiceBob Tetrault: Deputy Director, International
Production Assessment
State of the Practice, USDA-FAS Perspective 1
Agenda1) USDA Foreign Agricultural Service, Office
of Global Analysis, International Production Assessment Division (IPAD)
Introduction and decision-making process
2) IPAD’s use of Earth Observation products State of the practice and EO product transition
3) Products in a decision-support portfolio arranged by ARL and product type
State of the Practice, USDA-FAS Perspective 2
USDA’s Economic Information System, Official U.S. government estimates of World Agricultural Production• 18 Commodities,166 Countries,• 1,209 Country-Crop Pairs (e.g. Australia-Wheat) • Crop Analysis:
Where is the crop grown? When is the crop harvested? How is the crop doing? How big is the crop?(area, yield, & production)
What does it mean to food
supply?What does it
mean to agricultural
prices?
State of the Practice, USDA-FAS Perspective 3
USDA-FAS-IPAD uses a well-defined decision-making process
Foreign Agricultural ServiceOffice of Global Analysis
IPA Division
4
Analysis of data sources and
data products
Presentation and discussion of data products
In FY2017, FAS Office of Global Analysis reviewed the business process for USDA’s Interagency Commodity Estimates Committee (ICEC).
Source: MODIS NDVI 8-day & SPAM-IIASA 2005 Soybean Mask,NASA/GSFC/GIMMS, USDA/FAS/IPAD
Southern Rio Grande do Sul
Northern Rio Grande do Sul
Product Example: MODIS NDVI 8-day for Soybean-Producing Areas of Rio Grande do Sul, Brazil
Southern Rio Grande do Sul(2% of Brazil soybean
production)
• NDVI indicates crop growth is delayed and vegetation condition is below last year.
• Crops planted in Nov/Dec had poor germination/low, soil moisture. Development was slow and plants had low vigor (PR SEAB-DERAL). Rain at the end of the January was beneficial.
CLOUD-REDUCEDDATA SAMPLE
Foreign Agricultural ServiceOffice of Global AnalysisIPA Division
Northern Rio Grande do Sul(14% of Brazil soybean
production)
NDVI indicates crop growth is delayed and vegetation condition is below last year.
CLOUD-REDUCEDDATA SAMPLE
5
Earth Observation Products function when they are in context of other products.This example incorporates several support products including: crop statistics crop distribution maps croplands mask precipitation & soil moisture
Crop condition product (EO NDVI) includes several contextual elements including: NDVI for this year & last year minimum, maximum & mean
NDVI count of observations vs.
expected
Critical: Crop analyst’s input and experience
Foreign Agricultural ServiceOffice of Global Analysis
IPA Division
IPAD Products: World Agricultural Production--production briefs, Commodity Intelligence Reports, Lockup presentations to World Agricultural Outlook Board
6
Research to operations—from an operational user’s perspectiveFour issues to be managed during transition
1. Identify / implement funding source Transition from research funds to operational funds
2. Identify / implement IT systems responsibility Identify who is responsible for data generation,
ingest and visualization (RACI: Responsible, Accountable, Consulted, Informed)
3. Product review / Training Blend of scientific review of observations to product
latency to confidence in the product. 4. Data continuity (future satellites)
State of the Practice, USDA-FAS Perspective 7
Table 1 Short Term
Horizon
NASA GIMMS MODIS NDVI
Soil Moisture Palmer Model SMOS @50km
G-REALM
GDA Yield Forecaster
(MODIS NDVI@250)
croplands mask
SSM/I Yield Forecaster
Budget FAS-IPAD FAS-IPAD FAS-IPAD & NASA-ROSES FAS-IPAD FAS-IPAD
RACI R: NASA-GIMMS A:
IPAD
R: NASA-GIMMSR: Inuteq/ASRC
A: IPAD
R: ESSICR: Inuteq/ASRC
A: IPAD
R: GDA A: IPAD
R: WeatherPredictR: Inuteq/ASRC
A: IPADIT System 2 external
websites; managed by
NASA-GIMMS
ftp managed by NASA-GIMMS;
ingest/visualization managed by Inuteq/ASRC
ftp managed by ESSIC/SGT;
ingest/visualization managed by Inuteq/ASRC
External website; managed by GDA
ftp managed by Weather Predict;
ingest/visualization managed by Inuteq/ASRC
ProductReview DONE DONE DONE DONE DONE
Training DONE To increase use NEEDS To increase use To increase use
DataContinuity
MODIS>>NPP/VIIRS
SMOS>>SMAP
JASON-3>>Sentinel 3a
MODIS>>NPP/VIIRS
SSMI>>tbd
ARL 9.0 8.5 8.0 7.5 7.0
Select USDA-FAS-IPAD Products, by ARL Level—Short Term Horizon
Foreign Agricultural ServiceOffice of Global Analysis
IPA Division
Transition to full integration and repeated use (ARL 9.0) requires identified budget, IT systems, and product review (including training)
8
Table 2 Medium Horizon
Soil Moisture Palmer Model SMAP @36km
NASA GIMMS (VIIRS
NDVI@350)
Precipitation(USAF 557WW
@10km)
Harmonized Sentinel-Landsat
(NDVI@30m)
Evaporative Stress Index
(ESI)
Budget FAS-IPAD FAS-IPAD& tbd
USAF 557 WW & FAS-IPAD NASA NASA-ROSES
RACI(notional)
R: NASA-GIMMSR: Inuteq/ASRC
A: IPAD
R: NASA-GIMMSA: IPAD
R: USAF 557 WW R: Inuteq/ASRC
A: IPADR: NASA-GSFC?
R: NASA-Marshall
R: Inuteq/ASRC A: IPAD
IT System(notional)
ftp managed by NASA-GIMMS;
ingest/visualization managed by Inuteq/ASRC
2 external websites;
managed by NASA-GIMMS
ingest managed by Inuteq/ASRC; IPAD internal database
(CADRE) managed by Inuteq/ASRC
TBD TBD
Product(notional) NEEDS NEEDS NEEDS NEEDS NEEDS
Training NEEDS DONE DONE NEEDS NEEDS
DataContinuity
SMAP>>tbd
NPP/VIIRS>>JPSS/VIIRS
GPM>>tbd
L-8/ S-2a&b>>Landsat-9 (small
satellites?)tbd
ARL 6.0 5.0 5.0 4.0 4.0
Select USDA-FAS-IPAD Products, by ARL Level—Medium Horizon
Foreign Agricultural ServiceOffice of Global Analysis
IPA Division
Transition from prototype requires external budget, focus on IT systems, and product review (observations>>algorithms>>training)
9
• X-Axis: NASA’s ARL, 1.0 to 9.0• Y-Axis: Product typeCrop area and crop yield products (e.g. relative crop
yield forecasting, flooded area)Crop condition products (e.g. NDVI, soil moisture,
CHIRPS, ESI, HLS)Support products (e.g. crop distribution maps, crop
calendar, fieldwork data collection, CropSignal—crop stats, precipitation and temperature)
• Z-Axis: Impact (a qualitative assessment)Crop analysts’ will be surveyed about the products
and converted into a quantitative score. Impact score is from 100 to 1,200 with 1,200 indicating that the product has a strong impact on IPAD.
USDA-FAS-IPAD decision-support system product portfolio
Foreign Agricultural ServiceOffice of Global Analysis
IPA Division
10
USDA-FAS-IPAD decision-support system product portfolio
Fifteen products (right side) are currently “state of practice.”ARL on X-axis; Product type on Y-axis; Impact to program on Z-axis.
Foreign Agricultural ServiceOffice of Global Analysis
IPA Division
11
IPAD products—state of practice (15)Products in development (12)
NASA GIMMS MODIS NDVI (charts & maps)
GDA Yield Forecaster (MODIS NDVI@250)--
croplands mask
Soil Moisture Palmer Model SMAP @36km
Precipitation maps and charts (USAF 557WW
@10km)
0.0 2.0 4.0 6.0 8.0 10.0Application Readiness Level (ARL)
Cro
p A
rea
&
Cro
p Yi
eld
Cro
p C
ondi
tion
Supp
ort
Anticipated products—from an operational user’s perspectiveSix products (ARL 1.0 to 3.0)
– GDA yield forecaster using VIIRS @350m– Flooded area estimation using Sentinel-1(baseline plus
change product)– Yield forecasting using Harmonized Landsat-Sentinel
@30m– Effective field edge boundaries for a global common land
unit, using Sentinel-2a&b @10m (machine learning)– Relative crop area estimation using Landsat-8 &
Sentinel-2a&b (machine learning)– Soil moisture—corrected 2-layer Palmer model using
SMAP @13km
State of the Practice, USDA-FAS Perspective 12
USDA-FAS-IPAD Research Needs
1. Complete transition of research products to operations2. Understand product interactions
Foreign Agricultural ServiceOffice of Global Analysis
IPA Division
13
IPAD products—state of practice (15)Products in development (12)
NASA GIMMS MODIS NDVI (charts & maps)
GDA Yield Forecaster (MODIS NDVI@250)--
croplands mask
Soil Moisture Palmer Model SMAP @36km
Precipitation maps and charts (USAF 557WW
@10km)
0.0 2.0 4.0 6.0 8.0 10.0Application Readiness Level (ARL)
Cro
p A
rea
&
Cro
p Yi
eld
Cro
p C
ondi
tion
Supp
ort
1. How does 10km precip. data interact with soil moisture?
2. How does ESI interact with NDVI?
3. How does NDVI @30m interact with crop models?
4. How does fieldwork interactwith soil moisture, CHIRPS and ESI?
5. How does effective field edge identification interactwith crop area estimation?
Emerging Technologies and Methods in Earth Observation for
Agricultural MonitoringFeb. 15, 2018
National Agricultural Library
USDA Foreign Agricultural ServiceBob Tetrault: Deputy Director, International
Production Assessment
Research to Operations, USDA-FAS Perspective
15
Acknowledge the different cultures, different communication, and different time scales for researchers vs. operational users
– Long-term strategic horizon: for developments that may need years of work, and might need substantial resources. (early stage research ARL 1.0 to ARL 3.0)
– Medium-term horizon: for development work over a 1-4 year time scale. (typically ARL 4.0 to ARL 7.0)
– Short-term horizon: for smaller developments or concludes long- and/or medium-term work; responsive to user feedback; responsive to changes in the computing environment (security upgrades), and/or changes in data products. (typically ARL 7.0 to ARL 9.0)
Research to Operations, USDA-FAS Perspective
Adapted from: The ECMWF research to operations (R2O) process, Buizza et al. 2017
16
Research OperationsGoals Explore untested technologies;
new knowledge; uncertain or distant future application
Robust technologies; continuity of previous data source and methods;
value in practicalityDrivers Quest for new results; publish
or perish; new grantsRepeatable results; system
securityFunding Supports entirely new topics Supports existing productsCosts Research personnel Software and system maintenanceIT System Code standards and
documentation secondary importance;
Code standards essential; documentation essential
Products Research papers; no fixed schedule
Products to support customers; Routine and rigid delivery schedule
User Community
Narrow; highly trained Broad based, often untrained; decision-making process with
multiple users
Differences between researchers and operations
Foreign Agricultural ServiceOffice of Global Analysis
IPA Division
Researchers and operations have different goals, drivers, measurements of success, and end users.
17
Adapted from An Emerging Protocol for Research-to-Operations (R2O) at CPC, Lenic et al. 2011
Issue Research Satellite
Operational Satellite
MODIS example
Data continuity Rarely replaced, no spare satellite
on orbit
Plans for high-priority sensors; spare satellite
Transition to VIIRS
Data format and collection
change over mission’s life
No change MODIS now on Collection 6
Data latency Hours to months Hours or less NRT within 24 hours; NRT to STD 82 hours
Impact of reduced data quality
Reduced science Lower value products
Will VIIRS products be reduced data quality
compared to MODIS?
Time value of data High long-term value
High immediate value
Aqua MYD NRT 7/3/2017 failure; produced on 7/10/2017, but ICEC
meetings finished. Terraproduct on time. (minimal
impact to lockup)
Key Differences between research satellites and operational satellites
Foreign Agricultural ServiceOffice of Global Analysis
IPA Division
Operational agricultural monitoring uses research satellites but need operational products.
18
Adapted from: Satellite Observations of the Earth’s Environment: Accelerating the transition of research to operations (2003) Anthes et al. National Research Council
Research to operations—from an operational user’s perspectiveFour issues to be managed during transition
1. Identify / implement funding source Transition from research funds to operational funds
2. Identify / implement IT systems responsibility Identify who is responsible for data generation,
ingest and visualization (RACI: Responsible, Accountable, Consulted, Informed)
3. Product review / Training Blend of scientific review of observations to product
latency to confidence in the product. 4. Data continuity (future satellites)
State of the Practice, USDA-FAS Perspective 19
ID Product QA Process Steps Prime Second Tertiary
1 Does the algorithm produce the product? Research organization
COR
2 Does the observation or measurements correspond to in situ measurements?
Research organization
COR
3 Can the IT system reliably produce the product in a timely fashion? (product latency)
Research organization
On-site contractor
COR
4 Has the products’ generation, ingest, storage and visualization been assigned RACI?
COR Research organization
On-site contractor
5 Does the new product compare well with a previously used product?
COR Research organization
On-site contractor
6 Do the crop analysts understand the science and the algorithm that underlie the product?
COR Research organization
Crop Analysts
7 Are the crop analysts trained? Can they adequately explain the product to their end users?
COR Crop Analysts Research organization
8 Do the geospatial products adequately represent what is found on the ground?
Crop Analysts COR Research organization
9 Do the crop analysts have confidence in the product? Does the product compare fit within convergence of evidence?
Crop Analysts COR Research organization
10 Is the visualization of the product suitable for the crop analysts? (e.g. units of measurement, legend, color scheme, font size, line width, etc.)
Crop Analysts COR & on-site contractors
Research organization
11 How are the crop analysts using the product? And how much? COR IPAD management
Crop Analysts
12 What prevents the crop analysts from using the product? COR IPAD management
Crop Analysts
Product Quality Assurance—Complicated Hand-off 20
Foreign Agricultural ServiceOffice of Global Analysis
IPA Division
ID Product QA Process Steps Prime Second Tertiary
7 Are the crop analysts trained? Can they adequately explain the product to their end users?
COR Crop Analysts Research organization
8 Do the geospatial products adequately represent what is found on the ground?
Crop Analysts COR Research organization
9 Do the crop analysts have confidence in the product? Does the product compare fit within convergence of evidence?
Crop Analysts COR Research organization
10 Is the visualization of the product suitable for the crop analysts? (e.g. units of measurement, legend, color scheme, font size, line width, etc.)
Crop Analysts COR & on-site contractors
Research organization
Product Quality Assurance—Complicated Hand-off
05
1015202530
11/3/2014
11/17/2014
12/1/2014
12/15/2014
12/29/2014
1/12/2015
1/26/2015
2/9/2015
2/23/2015
3/9/2015
Surface Soil Moisture 536-193 (Fazenda Oilema)
PM-SMOS_surface PM_surface
1. How can we better understand the EO products in the field?
2. Will this improve the confidence in the EO products? Foreign Agricultural Service
Office of Global AnalysisIPA Division
at Fazenda Oilema, Feb. 2015
21
Research to operations—from an operational user’s perspective
Research to Operations, USDA-FAS Perspective
Summary ARL 4.0 to 7.0 ARL 7.0 to 9.0Time scale Medium horizon Short termBudget Mixed sources Operational fundingRACI & IT System
Notional Operational & documented
Product Notional to developed Developed & documentedTraining Focused on understanding
science and algorithmsFocused on increasing use
Data Continuity
Notional Identifiable
“Well trained people are, therefore, one of the most important components of the remote sensing technology transfer process” (NRC, 2001b).
22
• Extra slides
IPAD(on-site
contractor)
ESA/VITO
Proba-VNASA
GIMMS (NDVI)
USGS (Landsat)
NASA MODIS Daily
Deimos (DMC)
UCSB (CHIRPS)
USAF 557th
Weather Wing
WMO
NOAA (CMORPH)
NASA (GPM & TRMM)
NASA/ ARS
(SMOS &
SMAP)
ESSIC (G-
REALM)
Pacific Disaster Center
GDA
Weather Predict (SSM/I)
USGS/ FEWS (Eta)
Storm Data (wind
speeds)
USDA-FAS-IPAD Earth Observation Data Sources
Precipitation
Satellite Imagery
IPAD ingests data or accesses websites from more than 17 data sources.
1. Data Generation (partner)
2. Data Ingest (on-site contractor)
3. Product Generation (partner or on-site contractor)
4. Visualization(partner or on-site contractor)
24