Adam Lewis–SPEDDEXES 2014

Unlocking the Landsat Archive,

the Australian Geoscience Data

Cube & etc

Adam Lewis, Geoscience

Australia

Outline

• Steps toward unlocking GA’s Landsat Archive

• Putting data to use : the Australian Geoscience Data Cube

(AGDC)

• Where the data cube is heading - some future directions

• Discrete Global Gridding System

Business Systems Development NEO Team Brief

National Flood Risk Information Portal

Value layer

Delivery,

storage and

analysis layer

Data acquisition

and preparation

layer

Emergency

management

Water

Private

sector

Carbon

accounting

Climate

and

weather

APS 200 and

FOI reform

Emergency

management

tools

Water

tools

Carbon

accounting

tools

Climate and

weather tools

National

framework

datasets:

Authoritative

Base Image and

Landcover of

Australia

Geometric

correction

Image

generation

Observation

corrections

Analysis of

biophysical dynamics

(Green/brown/water/s

oil fraction and

indices)

Generation of

landcover map

Data acquisition -Public good data (Landsat, MODIS, GMES Sentinels)

-Commercial data (DMCii, SPOT, WV2, Geoeye, aerial)

Scene storage

Projected grid storage

Un-projected grid storage

Virtual compute

Cloud compute

Web services

Security WMS WFS

WCS WCPS

Getting useful information out of data

Traditional remote sensing product process is too

slow

GA Wednesday Seminar 30/10/13 - Datacube

Petabyte heirarchical

archive: Millions of

individual scenes

Tape store accessed by

robot.

Orthorectification

calibration, cloud

Masking, atmospheric

correction, mosaicing

Client

requests

product

Identify footprint

of product in

space or time

Search catalog

order scenes

Product packaging

and delivery

Feature extraction,

algorithm application

spectral unmixing


Value layer

Delivery,

storage and

analysis layer

Data acquisition

and preparation

layer Data acquisition -Public good data (Landsat, MODIS, GMES Sentinels)


Historical focus – collecting data

Historical strength

-1979 Australian Landsat

Station

- circa 600,000 Landsat

scenes

- unique archive over Australia

(now largely repatriated to the

USGS)

- raw data, on tape, ~200 tB

John MacDonald &

Warren Serone, 2012


Data acquisition

and preparation

layer

Image

generation



Automate the production of images

Automated systems ‘Process Management Application’

Baseline: 50 scenes per day, manual (3 fte)

Target: 1000 scenes per day

Actual: 2000+ scenes per day

Driver: International Forest Carbon Initiative


Data acquisition

and preparation

layer

Image

generation



Deal with the storage problem

Automated systems ‘Process Management Application’

Baseline: 50 scenes per day, manual (3 fte)

Target: 1000 scenes per day

Actual: 2000+ scenes per day

Driver: International Forest Carbon Initiative Storing processed images - Earth

Observation Data Store

- Baseline: a few hundred scenes

- Jan 2012: ~500,000 records

- Dec 2012: >2,000,000 records

Driver: International Forest Carbon Initiative


Data acquisition

and preparation

layer

Image

generation

Geometric

correction

Observation

corrections

Analysis of



oil fraction and

indices)

Generation of

landcover map



Calibration to produce a measurement in x,y,z,t

Calibration – Surface reflectance

Baseline: empirical methods

Target: physics-based method

Actual: community acceptance,

published method

Driver: Data stewardship

Correlation betw een Landsat 5 and 7 in Band 1y = 0.9796x

R2 = 0.986

0

1000

2000

3000

4000

5000

6000

7000

0 1000 2000 3000 4000 5000 6000 7000

Landsat 7 (Surface reflectance x 10000)

Landsat 5 (

Surf

ace r

efle

cta

nce x

10000)

Series1

Linear (Series1)


Data acquisition

and preparation

layer

Image

generation

Geometric

correction

Observation

corrections

Analysis of



oil fraction and

indices)

Generation of

landcover map



Quality assessment / filters

Calibration – Pixel quality assessment

Baseline: accepted methods exist

Target: implement an accepted method

Actual: achieved

Driver: Australian Space Research

Program, Unlocking the Landsat Archive



Data acquisition

and preparation

layer

Image

generation

Geometric

correction

Observation

corrections

Analysis of



oil fraction and

indices)

Generation of

landcover map



Algorithms to estimate biophysical parameters

Extracting ‘biophysical primaries’,

i.e,. water (in this case)

Baseline: no accepted methods

Target: published, accepted, available,

rapid, automatic method

Current state: accepted, automated, un-

published, involves commercial

software.

Driver: Emergency management

Density of quality-assured observations (15 years)

Lewis and Thankappan, AOMSUC October 2013

How do you work with this???



Value layer

Delivery,

storage and

analysis layer

Data acquisition

and preparation

layer

Emergency

management

Water

Private

sector

Carbon

accounting

Climate

and

weather

APS 200 and

FOI reform

Emergency

management

tools

Water

tools

Carbon

accounting

tools

Climate and

weather tools

National

framework

datasets:

Authoritative

Base Image and

Landcover of

Australia

Geometric

correction

Image

generation

Observation

corrections

Analysis of



oil fraction and

indices)

Generation of

landcover map



Scene storage



Virtual compute

Cloud compute

Web services

Security WMS WFS

WCS WCPS

Capture, analysis and application of Earth obsvns

Data organised for HPC - time series observations


Data organised for HPC - time series observations


Tile Count

(Currently approx. 4M tiles)

Landsat Scene Count

(Currently approx. 650k

scenes)


Value layer

Delivery,

storage and

analysis layer

Data acquisition

and preparation

layer

Emergency

management

Water

Private

sector

Carbon

accounting

Climate

and

weather

APS 200 and

FOI reform

Emergency

management

tools

Water

tools

Carbon

accounting

tools

Climate and

weather tools

National

framework

datasets:

Authoritative

Base Image and

Landcover of

Australia

Geometric

correction

Image

generation

Observation

corrections

Analysis of



oil fraction and

indices)

Generation of

landcover map



Scene storage



Virtual compute

Cloud compute

Web services

Security WMS WFS

WCS WCPS

Getting useful information out of data


Four-month non-interpolated median NDVI for

entire Murray Darling Basin

• Initial Datacube test area

• 2,112,000,000 pixels (i.e. 2.1 Billion).

• Every observation can be traced back to

its source capture image through

provenance information layers

Normalised 15-year surface water count (25m)



Datacube Overview - 13/09/2013

Area NE of Lake Eyre showing channel bathymetry and porous dunes













Some next steps for the data cube

Steering committee of key stakeholders– GA, NCI, CSIRO

Data reside on the RDSI – NCI node

Exploring the relationship between data and models

More data

• Geology - radiometrics; gravity; ASTER mineral maps

• Topographics – elevation, slope, topographic elements

• Climate surfaces

• Additional EO datasats: Landsat-8, MODIS, Landsat-MSS,

other, future satellites – Sentinel-2; himawari-8/9

• Derived measurements: Fractional cover, Surface Water,

Burnt areas, etc (using nationally accepted algorithms

developed through collaborative efforts)


Discrete Global Grid Systems

A common global grid architecture would allow us to:

• Organise measurements over the globe

• Calculate gradients faithfully

• Compare time-series of globally distributed data

• Make statistically meaningful regional comparisons of global

data

• Compare and combine data from multiple measurements

taken at different resolutions

• Improve operation of numerical models

• Document the precision as well as location of spatial data on

the globe

Towards a Global Discrete Nested Grid

Kimerling/Goodchild Criteria for gridding systems


Criterion

Criteria in Kimerling et al. (1999)

(Goodchild's Numbers given in

parentheses)

Criteria in Goodchild (1994)

1.Domain is globe

Areal cells constitute a complete tiling of

the globe, exhaustively covering the

globe without overlapping. (3,7)

1. Each area contains one point

2. Equal area

Areal cells have equal areas. This

minimizes the confounding effects of

area variation in analysis, and provides

equal probabilities for sampling designs.

(2)

2. Areas are equal in size

3. Same topology

Areal cells have the same topology

(same number of edges and vertices). (9,

14)

3. Areas exhaustively cover the domain

4. Equal shape

Areal cells have the same shape. ideally

a regular spherical polygon with edges

that are great circles. (4)

4. Areas are equal in shape

5. Compactness Areal cells are compact. (10)

5. Points form a hierarchy preserving

some (undefined) property for m < n

points

6. Straight Edges on Projection Edges of cells are straight in a projection.

(8)

6. Areas form a hierarchy preserving

some (undefined) property for m < n

areas

7. Perimeter Bisection

The midpoint of an arc connecting two

adjacent cells coincides with the midpoint

of the edge between the two cells.

7. The domain is the globe (sphere,

spheroid)

Kimerling/Goodchild Criteria for gridding systems


8. Hierarchy The points and areal cells of the various resolution grids which constitute the grid

system form a hierarchy which displays a high degree of regularity. (5,6)

8. Edges of areas

are straight on

some projection

9. Single point A single areal cell contains only one grid reference point.(1)

9. Areas have the

same number of

edges

10. Maximally centred Grid reference points are maximally central within areal cells. (11) 10. Areas are

compact

11. Equidistant Grid reference points are equidistant from their neighbors. (12)

11. Points are

maximally central

within areas

12. Addressing Grid reference points and areal cells display regularities and other properties

which allow them to be addressed in an efficient manner.

12. Points are

equidistant

13. Latitude Longitude The grid system has a simple relationship to latitude and longitude.

13. Edges are

areas of equal

length

14. Arbitrary resolution The grid system contains grids of any arbitrary defined spatial resolution. (5,6)

14. Addresses of

points and areas

are regular and

reflect other

properties

Progressing global gridding systems

rHEALPix from Landcare Research New Zealand

OGC Special Working Group to be proposed in May


What about the modelling layer?


Opening up new possibilities

• Establishing the ability to do things that we don’t

yet know about / are not yet possible

“For example, over 40% of the revenue from IBM

last year came from products and services that

were impossible to do just two years ago.”


Date post:	03-Sep-2014
Category:	Education
Upload:	aceas13tern
View:	169 times
Download:	2 times

Adam Lewis–SPEDDEXES 2014

Education