Developing standards and automated production for Sentinel...

Page 1 of 56

CUSTOMER

DEFRA

Developing standards and automated production for Sentinel-2 Analysis

Ready Data

PROJECT

EODIP CCDP5 Sentinel-2 ARD

DATE

18/10/2017

STATUS

Final

VERSION NUMBER

V0.1

CLASSIFICATION

PREPARED BY OPEN

Tom Jones, Earth Observation Specialist (Catapult)

Dan Wicks, Senior Earth Observation Specialist (Catapult)

Simon Agass, Head of Geospatial Systems & Solutions

(Catapult)

Dr. Pete Bunting, Reader (Aberystwyth University)

REVIEWED BY

Mark Jarman, Head of Earth Observation (Catapult)

Richard Hilton, Head of Technology Solutions (Catapult)

Joana Kamenova, Space Innovation Facilitator (Catapult)

APPROVED BY

Dr Gwawr Jones, Earth Observation Specialist (JNCC) Satellite Applications Catapult Electron Building Fermi Avenue Harwell Didcot Oxfordshire OX11 0QR Tel: +44 (0) 1235 567999 Email: [email protected]

THIS DOCUMENT IS THE PROPERTY OF SATELLITE APPLICATIONS CATAPULT AND MUST NOT BE COPIED OR USED FOR ANY PURPOSE OTHER THAN THAT FOR WHICH IT HAS BEEN SUPPLIED.

mailto:[email protected]

Developing standards and automated production for Sentinel-2 Analysis Ready Data

Status: Final

Version: v0.1

Date: 18/10/2017

Page 2 of 56


Status: Final

Version: v0.1

Date: 18/10/2017

Page 3 of 56

TABLE OF CONTENTS

TABLE OF CONTENTS ........................................................................................................................ 3

GLOSSARY ........................................................................................................................................ 5

FOREWARD ...................................................................................................................................... 8

PART 1: SENTINEL-2 ANALYSIS READY DATA PROJECT ........................................................................ 9

Introduction ............................................................................................................................................. 9

Background .............................................................................................................................................. 9

Sentinel-2 Context ................................................................................................................................. 11

Sentinel-2 ARD Project ........................................................................................................................... 12

Summary ................................................................................................................................................ 13

Development of a UK ARD and Data Cube Strategy........................................................................... 14

Sentinel-2 ARD - next steps for the project and beyond .................................................................... 15

Requirements for UK routine production of Analysis Ready Data ..................................................... 16

PART 2: SUPPORT FOR ESA SENTINEL-2 DATA WITHIN ARCSI ........................................................... 21

Key Points: ............................................................................................................................................. 21

Introduction ........................................................................................................................................... 21

Data Analysis .......................................................................................................................................... 22

Parsing Sentinel-2 Headers................................................................................................................. 22

At-sensor (top of atmosphere) radiance ............................................................................................ 23

At-sensor (top of atmosphere) reflectance ........................................................................................ 23

Surface Reflectance (SREF) ................................................................................................................. 23

Standardised Surface Reflectance ...................................................................................................... 24

Cloud Masking .................................................................................................................................... 25

Clear-Sky Masks .................................................................................................................................. 25

Topographic Shadow Masks ............................................................................................................... 26

Image Band Sharpening ...................................................................................................................... 27

Processing Multiple Granules ............................................................................................................. 27

Access to Sentinel-2 Imagery ............................................................................................................. 28

Results .................................................................................................................................................... 28

Quality of the Cloud Masking ............................................................................................................. 28

Quality of the Surface Reflectance Spectra ........................................................................................ 29

Image Pixel Comparison to ESA ARD Product ........................................................................................ 29


Status: Final

Version: v0.1

Date: 18/10/2017

Page 4 of 56

Standardised Reflectance ................................................................................................................... 34

Imagery Sharpening ............................................................................................................................ 34

ARCSI Recommendations & Conclusions ............................................................................................... 36

Recommendations.............................................................................................................................. 36

Conclusions ......................................................................................................................................... 37

APPENDIX A - SENTINEL-2 CHARACTERISTICS AND PROCESSING CONSIDERATIONS .......................... 38

APPENDIX B - SENTINEL-2 ARD INTERNATIONAL CONTEXT ............................................................... 46

APPENDIX C - A SUMMARY OF STANDARDS .................................................................................... 52

REFERENCES ................................................................................................................................... 54


Status: Final

Version: v0.1

Date: 18/10/2017

Page 5 of 56

GLOSSARY

Term Definition

AC Atmospheric correction

ACIX Atmospheric Correction Inter-Comparison Exercise

AGDC Australian Geoscience Data Cube

AOD Aerosol Optical Depth

AOT Aerosol Optical Thickness

API Application Programming Interface

ARCSI Atmospheric and Radiometric Correction of Satellite Imagery

ARD Analysis Ready Data

AWS Amazon Web Services

BEIS Department for Business, Energy and Industrial Strategy

BoA Bottom of Atmosphere

BRDF Bidirectional Scattering Distribution Function

CARD4L Committee on Earth Observation Satellites Analysis Ready Data for Land

Catapult Satellite Applications Catapult

CEDA Centre for Environmental Data Analysis

CEMS Climate, Environment and Monitoring from Space

CEOS Committee on Earth Observation Satellites

CESBIO Centre for the Study of the Biosphere from Space

CNES French National Centre for Space Studies

DC Data Cube

Defra Department for Environment, Food & Rural Affairs

DEM Digital Elevation Model

DfID Department for International Development

DIAS Data and Information Services


Status: Final

Version: v0.1

Date: 18/10/2017

Page 6 of 56

DOS Dark Object Subtraction

ECMWF European Centre for Medium-Range Weather Forecasts

EO Earth Observation

EO CoE Earth Observation Centre of Excellence

EODIP Earth Observation Data Integration Pilot

ESA European Space Agency

FCO Foreign and Commonwealth Office

FLAASH Fast Line-of sight Atmospheric Analysis of Spectral Hypercubes

GDP Gross Domestic Product

GEO Group on Earth Observations

GEOSS Group on Earth Observations (GEO) and Global Earth Observation System of Systems

GFZ German Research Centre for Geosciences

HMA Heterogeneous Missions Accessibility

HMG Her Majesty's Government

HPC High Performance Computing

IT Information Technology

JASMIN Joint Analysis System

JAXA Japan Aerospace Exploration Agency

LSI-VC Land Surface Imaging Virtual Constellation

MACCS Multi-sensor Atmospheric Correction and Cloud Screening

MOD Ministry of Defence

MODIS Moderate Resolution Imaging Spectroradiometer

MODTRAN MODerate resolution TRANsmission

MPI Message Passing Interface

NCEO National Centre for Earth Observation

NDVI Normalized Difference Vegetation Index

NERC Natural Environment Research Council


Status: Final

Version: v0.1

Date: 18/10/2017

Page 7 of 56

NIR Near Infrared

NSA United States National Security Agency

ODC Open Data Cube

OGC Open Geospatial Consortium

RBINS Royal Belgian Institute of Natural Sciences

SAFE Standard Archive Format for Europe

SEDAS Sentinel Data Access Service

SGP Space Growth Partnership

SREF Surface Reflectance

SSGP Space for Smarter Government Programme

STFC Science and Technology Facilities Council

SWIR Shortwave Infrared

THEIA French Land Data Centre

TIR Thermal Infrared

ToA Top of Atmosphere

UKSA UK Space Agency

USGS United States Geological Survey

UTM Universal Transverse Mercator

WGCV Working Group on Calibration and Validation


Status: Draft

Version: v0.1

Date: 18/10/2017

Page 8 of 56

FOREWARD

This report provides information about using the Atmospheric and Radiometric Correction of Satellite

Imagery (ARCSI) software to produce Sentinel-2 Analysis Ready Data (ARD). It further provides context

regarding the rationale for ARD and wider international activities relating to ARD.

The report consists of:

PART 1: Sentinel-2 ARD project. This section provides a summary of project rationale, an

overview of international ARD activities, commentary on the development of a UK ARD and Data

Cube (DC) strategy and requirements for UK routine production of ARD.

PART 2: Support for ESA Sentinel-2 data within ARCSI. This section demonstrates how the ARCSI

software can be used to produce ARD as a topographically corrected surface reflectance product

for Sentinel-2.

Appendices: This sectional provides supporting materials on Sentinel-2 characteristics and

processing considerations, Sentinel-2 ARD international context and a summary of standards.


Status: Final

Version: v0.1

Date: 18/10/2017

Page 9 of 56

PART 1: SENTINEL-2 ANALYSIS READY DATA PROJECT

Introduction

The launch of the various Sentinel satellites by the European Space Agency (ESA) as part of the European

Commission’s Copernicus Programme1 is generating ever-increasing interest and therefore user

requirements across government and different market sectors. However, there remain barriers to

realising the potential use of this data as a routine input to value added service and product generation.

Technical complexities associated with the handling and manipulation of the data remain, alongside a

lack of standardisation of the form in which the data should be made available. To allow immediate

analysis of the data without additional significant user effort, these barriers need to be addressed.

Through a collaboration between the Satellite Applications Catapult (Catapult), Aberystwyth University,

and the Defra Earth Observation Centre of Excellence (EO CoE) with support from the UK Space Agency’s

(UKSA) Space for Smarter Government Programme (SSGP), the Sentinel-2 Analysis Ready Data (ARD)

project has created a series of recommendations designed to ensure that UK users across government

and industry can maximise routine exploitation of Sentinel-2 data (See Appendix A for detailed Sentinel-

2 characteristics and processing considerations). These recommendations focus on the requirements for

the processing of Sentinel-2 data to a new standard termed ARD, the creation of an open toolset for

implementing the generation of Sentinel-2 ARD products, and the generation of a Sentinel-2 ARD data

set for the UK covering the full Sentinel-2 acquisition campaign up to June 2017. This project ensures

continued UK contribution to wider international community efforts to develop standards and methods

of Earth Observation (EO) data access.

Background

The UK firmly recognises that the space industry must adapt to meet the ever-growing interest and user

requirements for EO data products and services. When considering the exponential growth of EO data

availability and wide range of potential users, it becomes quickly apparent that it is simply not technically

feasible or financially affordable to consider traditional methods of storing, handling and manipulating

EO data. Local processing and data distribution methods currently exploited by industry and government

are not suitable to address the challenge of scalability, increases in the size of data volumes, and the

growing complexities in preparation, handling, storage, and analysis required to meet user requirements.

To access the valuable information contained within EO data, users are required to undertake a series of

complex pre-processing steps to turn the data from a ‘raw’ unprocessed format into a state that can be

1 http://www.copernicus.eu/


Status: Final

Version: v0.1

Date: 18/10/2017

Page 10 of 56

analysed. Unless the user has the expertise, software and infrastructure to handle and process this

information, efficient exploitation of the data is not realised.

A solution to this, in part, is through the systematic and regular provision of ARD. This is defined as

“satellite data that have been processed to a minimum set of requirements and organised into a form

that allows immediate analysis without additional user effort and interoperability with other datasets

both through time and space”2.

The main steps in the production of ARD, include:

• Geometric correction (it is assumed that the Level-1 data provided by ESA is to a desirable

standard).

• Atmospheric correction

• Radiometric correction

The potential for regular provision of ARD is achievable given developments and advancements in

contemporary computing infrastructures, technologies, and data architectures that overcome data

management and analyses challenges that have arisen from increasing data volumes. One such example

is Data Cube (DC) technology, which considers a scalable architecture of spatially aligned time series stack

of pixels for ease of data interrogation and analysis. Such a solution has great potential to streamline

data distribution and management for providers while simultaneously lowering the technical barriers for

users to access and exploit EO data.

It is a critical time for the UK to contribute to development around ARD technologies given the work that

is actively being undertaken by several key organisations on the international stage (see further details

in Appendix B). The main protagonist is the Committee on Earth Observation Satellites (CEOS), which is

working to coordinate this activity, bringing together entities such as Geoscience Australia, ESA, Centre

National D’Etudes Spatiales (CNES), the National Aeronautics and Space Administration (NASA), the

United States Geological Survey (USGS) and the Japan Aerospace Exploration Agency (JAXA). CEOS is

developing a series of standards for ARD, branded ‘CEOS Analysis Ready Data for Land (CARD4L)’, through

an initiative primed by the CEOS Land Surface Imaging Virtual Constellation (LSI-VC). Directly supporting

this work is the CEOS Working Group on Calibration and Validation (WGCV) that is investigating issues of

data interoperability between ARD produced from different EO sensors. More broadly feeding into

generation of the standards and assessment of best practice is an international collaborative initiative to

inter-compare a set of atmospheric correction (AC) processors, led jointly by ESA and NASA. Both

organisations have a vested interest in progressing consolidation of an ARD standard due to a desire to

publish Sentinel and Landsat ARD retrospectively as part of their core data dissemination activities.

Despite limited UK involvement thus far on the international stage within the aforementioned groups,

there is significant activity being undertaken by UK government, academia, industry and the Catapult to

2https://www.google.co.uk/search?q=ceos+card4l&oq=ceos+card4l&aqs=chrome..69i57.3367j0j7&sourceid=chrome&ie=UTF-8#


Status: Final

Version: v0.1

Date: 18/10/2017

Page 11 of 56

consolidate requirements for ARD standards and investigate future EO data architecture approaches.

Most notable is the work being undertaken by the Defra EOCoE within their EO Data Integration Pilot

(EODIP) Programme. This includes ARD production for government users built upon initial work carried

out by the Welsh government and their data access hub, which makes available pre-processed Landsat

data and the work of the Catapult, contributing to the definition of ARD standards for Sentinel data. To

maintain the UK’s strong leadership position in space technology, UKSA, The National Centre for Earth

Observation (NCEO), Defra and the Catapult are working together to feed this activity into international

discussions and provide a stronger steer on future development activity.

Sentinel-2 Context

While the concept of ARD is directly applicable to all EO data, initial efforts have been focused on

increasing the exploitation of openly available free to access data sets given the increased user interest

driven by the launch of the Sentinel satellites. The Sentinel-2 dataset is prioritised due to its high

temporality, rich multi-spectral characteristics, effective continuation of the extensive Landsat record,

and familiarity to end users given the optical nature of data (see further details in Appendix A).

In terms of routine access, ESA has been providing a Top of Atmosphere (ToA) reflectance Sentinel-2

product since October 2015. To realise a consistent data set for exploitation, corrections for variation

resulting from atmosphere, including cloud cover masking, and terrain illumination must be implemented

on the end user side. To account for this requirement, ESA made available the Sen2Cor3 processor to

allow users to generate Sentinel-2 Bottom of Atmosphere (BoA) surface reflectance products.

Despite its open source nature, Sen2Cor still requires significant EO and IT expertise to use, and remains

a barrier to non-expert users. Furthermore, it is temperamental in terms of its installation and corrections

implemented are known to be limited and poorly geographically transferable. A variety of proprietary

alternatives exist; however, their Sentinel-2 correction capabilities remain publicly unproven through

wide-scale comparison exercises, provide only partial solutions, or become impractical for routine

implementation through automation.

Given the investments that ESA has made in relation to the Sentinel satellites, it has realised that to

enable mass exploitation and to remain relevant in an increasingly commercial industry, it must

disseminate datasets of greater accessibility. Since May 2017, ESA has been piloting the provision of

Level-2A (BoA) products over Europe. However, many unanswered questions remain regarding its

suitability for routine use, such as:

• The finality of the algorithm and its transferability

• Service levels associated with data provision, most notably relating to coverage and latency

3 http://step.esa.int/main/third-party-plugins-2/sen2cor/


Status: Final

Version: v0.1

Date: 18/10/2017

Page 12 of 56

• The responsibility for data dissemination, whether it is through the core Copernicus hub or part

of future services such as the ESA Data Information and Access Service (DIAS).

There is also uncertainty as to how ESA will position dissemination of this data against parallel services

being developed by organisations such as Amazon Web Services (AWS) and Google.

Sentinel-2 ARD Project

If the anticipated industry growth in the UK is to be realised, the community must quickly establish access

to reliable, operational data services for ARD. Demand from users familiar with the use and exploitation

of EO data is clear, driven by requirements for efficient, operative, scalable and commercial solutions.

However, more importantly there is increasing interest from organisations and markets outside of the

space sector who are keen to realise the benefit of EO data in new digital services and applications. As a

sector, we will be failing significantly if we do not embrace this demand. This project sought to support

the UK moving towards this ideal, although with much future discussion required as to where the

responsibility for hosting and financing a solution might lie.

Currently, a suitable tool for producing Sentinel-2 ARD is not openly available, nor a standard associated

with this product type. Catapult and Defra’s EO CoE identified an opportunity to fund a piece of work to

help facilitate UK exploitation of Sentinel-2, while simultaneously raising the profile of EO and UK

expertise in this field. The project leverages UK capability in processing and exploiting EO datasets to

develop a method for generating Sentinel-2 ARD products that meets the standards required by the UK

geospatial and EO community, both public and private, and is also well aligned with the proposed CEOS

CARD4L standards.

The project has made available an open source and readily accessible tool that delivers state-of-the-art

functionality in correcting optical satellite imagery; including atmospheric correction, terrain illumination

mitigation, novel band-sharpening correction, and neighbouring multi-image processing capabilities to

deliver a Sentinel-2 BoA surface reflectance product. This tool is an expansion of the Atmospheric and

Radiometric Correction of Satellite Imagery (ARCSI)4 software developed via leading research at

Aberystwyth University and provides an accessible means of atmospherically correcting optical satellite

imagery that aids automation (see further details in Part 2).

4 http://rsgislib.org/arcsi/


Status: Final

Version: v0.1

Date: 18/10/2017

Page 13 of 56

Identified advantages of using the ARCSI tool over ESA’s Sen2Cor processor include:

• The image products produced by ARCSI provide a comparable result to that currently being

piloted by ESA but without artefacts creating within the topographic correction.

• The innovative image band sharpening approach implemented with ARCSI provides a significant

improvement over traditional approaches of image resampling when using the 20-metre image

bands within a 10-metre resolution image stack.

• The ability to process multiple image granules together, thus avoiding boundary effects between

the image granules.

The support for Sentinel-2 within ARCSI provides a step forward towards realising Sentinel-2 ARD

provision, although further work is recommended, particularly on the approaches to cloud masking and

correcting for bidirectional reflectance.

Further to the processor development, an extensive UK Sentinel-2 ARD dataset has been made openly

available for users to test the suitability of and to explore the added value in exploiting Sentinel-2 data

sets in a format that is analysis ready. Dissemination of this ARD will be made through the Catapult’s

Sentinel Data Access Service (SEDAS)5 data portal, the user-facing UK Copernicus Collaborative Ground

Segment. All acquisitions over the UK have been processed that comprise more than 25% useable

imagery (not obscured by cloud) since commencement of operational acquisition in October 2015. If

further funding is identified, a more complete dataset can be made available.

Summary

Funding of the current project has developed a highly enthusiastic UK community around Sentinel-2 and

ARD, with 60 representatives from across Government, industry and academia attending two workshops.

Over 90 participants have also joined the online discussion via the project's LinkedIn group and this

includes international participation from Australia, France, India, Italy, Spain, the Netherlands, and the

United States.

It is generally accepted by the community that the requirement to have access to an ARD product sooner,

outweighs the need to achieve perfection in the definition of the product, although the provision of

suitable metrics for quality are considered of high importance.

The project has not only helped address standards questions related to Sentinel-2 ARD, but has allowed

the UK to constructively contribute to wider conversations in the international ARD sphere, positioning

the UK as a leading authority in EO exploitation. An open source tool for generating Sentinel-2 ARD is

now readily accessible, capturing UK satellite expertise, as well as being very well aligned with the latest

international ARD standards definition (CARD4L). An extensive dataset of Sentinel-2 ARD has been

generated for the UK and made readily available to the UK community. This dataset shall further

5 https://geobrowser.satapps.org/


Status: Final

Version: v0.1

Date: 18/10/2017

Page 14 of 56

demonstrate the value offering of EO, as well as amplifying UK conversations around the necessary

routine provision of ARD suitable for UK end users; no such provision currently exists.

The rationale for an ARD data service is clear and the user demand strong. The contribution of space and

space-enabled services to UK GDP continues to grow at a significantly higher rate than the rest of the

economy6. ARD would only further this growth by accelerating the development of new applications and

services through improved routine access to EO data. It would enable the delivery of solutions with

greater efficiency, at reduced cost and allow effective embedding of EO within new markets.

Furthermore, government increasingly recognises the value of EO, evidenced by Defra EO CoE’s

commitment to integrate “EO into at least five Defra policy areas to ensure cheaper, more effective, more

customer-focused delivery by 2020”7. Despite these positive steps, the UK has some decisions to make

about how to champion these concepts in the long-term and whether it should take more of a leadership

role on the international stage.

Development of a UK ARD and Data Cube Strategy

Successful implementation of ARD and Data Cube (DC) technologies within the UK geospatial ecosystem

will only be achieved with clear direction, ownership and alignment with wider UK and international

programmes. In the very short-term, a UK lead needs to be established to drive work in this area and to

define a coherent, implementable strategy that reflects the interests of stakeholders across government,

academia and industry. This strategy should tell a strong user story, consolidating evidence collected

through programmes such SSGP8 and the Space Growth Partnership (SGP)9. It is recommended that the

leading entity should ideally come from a government agency such as the UKSA or from the Catapult,

either of which can exploit their neutral trusted status to appropriately represent the interests of all

parties. With clear governance and strategy, the UK EO community will be well positioned to deliver

services and products to both new and existing users at increased efficiency and value.

The UK needs to firmly establish its engagement with the wider international community, notably

organisations such as CEOS and initiatives such as the Open Data Cube (ODC). With an active presence,

the UK can not only draw on the experience of others, but also influence the future direction of travel for

major EO data developments. Through nominated UK leads from the UKSA, the Catapult and the UK

Science community, the UK can contribute to critical working groups such as LSI-VC and WGCV.

Engagement has started with these groups. However, the UK needs to guarantee resource availability to

ensure continued active involvement within key conversations and meetings that are rapidly shaping

developments around ARD and DC technologies. Through this engagement, the UK has an exciting

6 https://sa.catapult.org.uk/wp-content/uploads/2016/04/C222628-VISION-2030-Design-Stg8.pdf

7 https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/488133/defra-earth-obs-roadmap-2015.pdf

8 http://www.spaceforsmartergovernment.uk/

9 http://spaceigs.org/


Status: Final

Version: v0.1

Date: 18/10/2017

Page 15 of 56

opportunity to develop partnerships, increase its credibility and develop an exportable model for ARD

generation and exploitation.

To consolidate its position, the UK community needs to address the following fundamental questions:

• What are the major technical requirements for UK ARD?

• What needs to exist to maximize the exploitation of ARD in the UK?

• How does the UK need to structure itself to best take advantage of ARD?

It is the consensus of stakeholders engaged through this project that the Sentinel-2 ARD data

demonstrated meets requirements for a minimum viable product. There is also overwhelming support

for tailored UK ARD data making use of, for example, high resolution Digital Elevation Models (DEMs)

that are available. The highest prioritised technical requirement is for transparency and ease of

repeatability of processing method. Regarding exploitation, stakeholders agreed that the key

requirement was for ease of access. Stakeholders strongly advocated for free access to ARD products as

an entry threshold but that higher level products such as mosaics, cloud free composites and spectral

indices (e.g. NDVI – Normalised Different Vegetation Index) products are also of high priority. It is

generally perceived that UK Government and public funding should anchor the costs for long term

provision but that private sector co-investment is also encouraged. A UK ARD standard supported by UK

Government gives confidence to industry and ensures sustainability of the technical solution.

Sentinel-2 ARD - next steps for the project and beyond

With regards to this project, several follow-on activities are required:

• Collaborative work with the UK and wider community on the technical challenges associated

with realising a truly interoperable BoA Sentinel-2 dataset that is comparable in space and time

with itself, notably harmonisation across neighbouring image granules and robust correction for

bidirectional reflectance.

• Work with the UK and wider community on the technical challenges associated with realising a

truly interoperable BoA Sentinel-2 dataset that is comparable in space and time with other

sensors. This ideal for ARD presents enormous technical challenges that are actively being

addressed by the scientific community. Harmonisation across sensors would allow users to work

with any EO data from similar sensors interoperably e.g. Sentinel-2 and Landsat-8.

• Move towards the routine delivery of a Sentinel-2 ARD dataset to the UK and to explore export

models. This will require significant work to explore possible routes of access, governance

structure and business models. For example, the relative merits and demerits of consumption

of ARD from a third-party service provider, versus construction of a national UK capability,

championed by government or achieved through public-private partnership, versus provision

through wholly commercial enterprise(s).


Status: Final

Version: v0.1

Date: 18/10/2017

Page 16 of 56

• Further develop the definition of Sentinel-1 ARD, building on previous work undertaken10.

Sentinel-1 is an important dataset for enabling services and applications, particularly in a mid-

latitude environment with cloud, such as the UK.

• Investigate geometric elements associated with generated Sentinel-2 ARD products.

• Investigate ARD possibilities for future Sentinel and commercial missions.

• Investigate the value and use cases for exploiting DC technologies.

In conclusion, the UK is in a strong position, in many ways leading Europe and certainly in the top-tier

globally. Our exploitation focus is a great advantage and we must not lose our position as others progress.

The community is positive and overall there is a strong desire to make ARD a success.

Requirements for UK routine production of Analysis Ready Data

The batch processing of the UK Sentinel-2 data was completed on the Catapult’s infrastructure during

August and September 2017. The process injected the Sentinel-2 archive for the UK, covering a time

window that spanned from 2015 to early September 2017. To accomplish the task, Catapult engineers

developed a data pipeline flow based on open source software. The main technology used is an open

source data flow system called Apache NiFi11. Apache NiFi is a software project from the Apache Software

Foundation which enables the automation of data flow between systems. It can be described as data

logistics. The project was created by the United States National Security Agency (NSA) and originally

named Niagarafiles. In 2014 the NSA released it as open source software, via its technology transfer

programme.

The developed pipeline included custom processes, developed ad-hoc for the batch processing. The

entire pipeline was composed of 99 processors. Figure 1 shows the high level architecture of the solution.

For the UK batch processing the following hardware was used:

• 3 x 380 GB RAM, 24 CPU, 12 TB storage

The observed performance and statistics of the re-processing campaign were as follows:

• Total input: 543 GB

• Total output: 1.02 TB

• 16 parallel processes per machine, with a total parallelization of 48 processes for the cluster

• User time ~ 120hours (48 parallel processes)

• Average time of the process: 2.5 hours

10 http://starhub.sa-catapult.co.uk/wordpress/wp-content/uploads/2016/05/C223766-IPSP-Cat-Australia-4WS.pdf

11 https://nifi.apache.org/


Status: Final

Version: v0.1

Date: 18/10/2017

Page 17 of 56

Figure 1 - High level diagram of the pipeline data flow

The development of this pipeline has opened up multiple scenarios, where ARD production would

become a straightforward process. The Catapult is currently looking to produce a more scalable and ready

to use pipeline that can be shared and used in the CEMS (Climate, Environment and Monitoring from

Space (CEMS) facility) environment. The main components of the pipeline under development are:

• Easy to use SEDAS API query for Sentinel-2 data: a simplified query interface is under

development, to locate, copy locally and process the data. End users will need to state their

area of interest and the time window that are interested.

• Scalable pipeline: an automate pipeline process to be deployed over multiple servers to

consume high volume of data.

• Landsat plugin to create ARD products.

• Enabling the end users to generate ARD themselves for an area of interest through providing

them the necessary toolset. For this purpose the entire data pipeline will be built into a CEMS

template. When a new data re-process would be required, a VM(s) can spin up with the correct

settings and the all required software.

In parallel, the Catapult has been investigating the use of DC technology through completing a DC

prototype for Wales. This contains all the Sentinel-2 ARD for Wales to date and will be made open to the

community to test. The enhancement of the DC prototype with further data and its sustainability will

depend on budget availability.


Status: Final

Version: v0.1

Date: 18/10/2017

Page 18 of 56

Table 1 provides a high level overview of different options and associated costs of ARD exploitation within

a DC. The table is to be read in accordance with the below assumptions:

• ARD volume in this particular project is 2 x input ESA data for Sentinel-2. This may vary depending

on the requirement.

• ARD volume is 5 x input ESA data for Sentinel-1. This is untested for other Sentinels and will need

validation.

• No new buildings / server rooms are needed.

• Public cloud pricing models will persist into the future.

• Bespoke service wrappers would be in addition to the options presented; these could be low-

cost or expensive depending on service levels / complexity.

• No assumptions for service-legal agreements have been made

• Formal quotes would be needed as figures presented here will change depending on who

implements any solution

• Largest cost-driver is storage

• Savings realised - today’s prices assumed and worst case scenario

• Preference for Open / Closed / Hybrid solution not clear


Status: Draft

Version: v0.1

Date: 18/10/2017

Page 19 of 56

Table 1 – High level overview of options for routine ARD exploitation within a Data Cube

Pros Cons Scale Cost

Do Nothing No additional

funding needed

Lost

opportunity to

improve

efficiency

across

Government

and the UK

industry

Lost

opportunity to

place the UK in

a leading role

in these

technologies

and data

exploitation

N/A – Users

manage with

what they need /

can do locally

N/A

Do the minimum Very low cost

Improves current

working situation

Multiple

versions of

data across

different

departments

Specialist

uptake likely

No economies

of scale

Ad Hoc – a

selection of areas

available on

“request”

Covered by

existing budgets

UK coverage Central store of ARD

products for

multiple

departmental use

Significant reduction

in replicated work,

and hence cost

More powerful data

analytics capability

Commercial

clouds could

be expensive

for information

retrieval or

data download

May not cover

all government

user needs

(e.g. FCO, DfID,

MOD, BEIS etc)

UK-wide (~32 S1

scenes per

coverage) –

overseas

territories would

be extra

£250-450k / year

depending on

number or

missions


Status: Final

Version: v0.1

Date: 18/10/2017

Page 20 of 56

Could take

advantage of public

cloud offers

EU+ coverage Opportunity to lead

the way in European

service provision

Opportunity to offer

this capability into

Copernicus

programme and

attract Copernicus

operational funding

Opportunity to

meaningfully engage

with Global Network

of Data Cubes

Data and capabilities

available across

government with

corresponding

efficiencies

BREXIT and

hence the UK’s

position in

Copernicus

unclear –

access to

budgets

May not cover

all government

user needs

(e.g. FCO, DfID,

MOD, BEIS etc)

European

countries plus N.

Africa. Russia?

(estimated at

~7PB data per

year – ESA + ARD

for all Sentinels

+ Landsat)

~£5m for initial

setup and 1st

year of S1

operations,

£2.5m-3m per

year for each

mission

Global coverage Opportunity to lead

the way in Global

service provision

Opportunity to offer

this capability into

Copernicus

programme and

attract Copernicus

operational funding

Opportunity to

meaningfully engage

with Global Network

of Data Cubes

Data and capabilities

available across

government with

corresponding

efficiencies

BREXIT and

hence the UK’s

position in

Copernicus

unclear –

access to

budgets

Cost to HMG –

initial and

ongoing

Global - ~25PB

data per year

(ESA data + ARD

products – for all

Sentinels +

Landsat)

~£12m for initial

setup and 1st

year of S1

operations, £7m-

10m per year for

each mission


Status: Final

Version: v0.1

Date: 18/10/2017

Page 21 of 56

PART 2: SUPPORT FOR ESA SENTINEL-2 DATA WITHIN

ARCSI

Key Points:

1) The ARCSI software can now be used to produce analysis ready data (ARD) as a topographically

corrected surface reflectance product for Sentinel-2.

2) The quality of the product is comparable and in some ways an improvement (i.e., image band

sharpening and reliable topographic correction) of the ARD surface reflectance currently (July

2017) provided by ESA through the Copernicus Open Access Hub.

3) Further improvements are required to classify cloud and cloud shadows for Sentinel-2.

4) ARCSI is a flexible, open source and free to use software system which supports many satellite

sensors and can be further developed and improved by the wider community.

5) Methods and datasets for validating the surface reflectance product are required these are

currently unavailable and would require resources to generate and then valid the product

generated through ARCSI processing chain.

6) It is recommended that ARCSI should be tested and approved to provide ARD data which meets

the CEOS CARD4L standard.

Introduction

The aim of this project was to provide functionality with the ARCSI software

(http://www.rsgislib.org/arcsi) to enable the production of analysis ready data (ARD) for the Sentinel-2

sensor. ARCSI already provides this functionality for several sensors including:

• Landsat (1-5, 7 & 8)

• WorldView 2 & 3

• Pleiades

• RapidEye

• SPOT 5, 6 & 7

Functionality varies for different sensors depending on the wavelengths and metadata provided for a

given sensor. For Sentinel-2, the aim is to provide functional equivalence to Landsat TM, ETM+ and OLS.

However, Sentinel-2 does not have any thermal wavelengths, which makes consistent and reliable cloud

masking more difficult.

To install ARCSI, binaries are provided through the conda (https://conda.io) package management

system. Once conda is set up, to install the stable release the following command is used:

conda install -c conda-forge arcsi

http://www.rsgislib.org/arcsi)

https://conda.io)/


Status: Final

Version: v0.1

Date: 18/10/2017

Page 22 of 56

The development channel, through which development versions of ARCSI are made available, is au-eoed.

The development version can be installed using the following command:

conda install -c au-eoed -c conda-forge arcsi

The source code for ARCSI is available online through the bitbucket platform:

https://bitbucket.org/petebunting/arcsi

Data Analysis

ARCSI provides a number of product options, those relevant to Sentinel-2 are listed in Table 2.

Table 2 - ARCSI Processing Products

ARCSI Product Definition

RAD At sensor radiance image.

SHARP Sharpen the 20 m image bands to 10 m using a local correlation filter.

SATURATE Mask of saturated pixels

TOA At sensor reflectance (top of atmosphere; TOA)

DOS Perform a dark object subtraction to retrieve an estimate of surface reflectance.

DEM Extract and reproject the inputted DEM for the image.

TOPOSHADOW Produce a mask for topographic shadow using the DEM.

CLOUDS Perform a classification to mask cloud and cloud shadow.

CLEARSKY Identify large continuous areas without cloud, providing an image mask

DOSAOTSGL Estimate a single value of Aerosol Optical Thickness (AOT) for the scene.

SREF Using the 6S atmospheric model calculate the surface reflectance pixel value for the scene.

STDSREF Calculate standardised reflectance, normalising for the sun angle and topography (i.e.,

topographic correction)

FOOTPRINT Extract the image data footprint as a shapefile.

METADATA Create a metadata file with key information from the input image and information calculated

during the analysis.

The user can also choose whether the outputs are at 10 m or 20 m image resolution and the resampling

method. If the sharpening option is select then the output image resolution needs to be at 10 m and a

nearest neighbour resampling needs to be selected. Additionally, the user can also select an output

coordinate system and projection that is different to input image projection and coordinate system.

Parsing Sentinel-2 Headers

ARCSI can parse all the known ESA SAFE header formats extracting the relevant data for the Sentinel-2

data. However, for the whole scenes (i.e., a number of granules within a single SAFE file, as disseminated

before September 26th, 2016) these need to be split before processing within ARCSI, which operates on

a per granules basis. A command arcsisplitsen2granules.py has been provided for splitting these files into

individual granules.

https://bitbucket.org/petebunting/arcsi


Status: Final

Version: v0.1

Date: 18/10/2017

Page 23 of 56

At-sensor (top of atmosphere) radiance

Sentinel-2 Level-1c data is provided as top of atmosphere reflectance. To retrieve at-sensor radiance

from these data the equation for deriving top of atmosphere reflectance needs to be rearranged for at-

sensor radiance:

𝐿𝜆 =ρλ ∙ ESUNλ ∙ cos θs

π ∙ d2

where λ is the wavelength, ρλ is the spectral (planetary or TOA) reflectance for wavelength λ, Lλ is the

spectral radiance (W m-2 sr-1 μm-1) d is the Earth-Sun distance in astronomical units, ESUNλis the mean

solar exoatmospheric irradiance in units of W m-2 μm-1, and cosθs is the solar zenith angle.

At-sensor (top of atmosphere) reflectance

At sensor or TOA reflectance is the ratio of the at-sensor radiance and the incoming radiance energy from

the sun (ESUN values). However, Sentinel-2 is provided as with pixel values as TOA reflectance, multiplied

by 10,000. ARCSI has a user selectable multiplier (default 1000) and therefore if different from 10,000

the TOA reflectance data as provided by ESA is rescaled.

Surface Reflectance (SREF)

Surface reflectance also called bottom of atmosphere reflectance is the ratio of incoming radiance (i.e.,

from the sun) with the radiance that is reflected from the ground surface. To derive this measurement

the effect of the atmosphere needs to be removed from the top of atmosphere radiance measured at

the sensor. The Second Simulation of the Satellite Signal in the Solar Spectrum (6S) atmospheric model is

used within the ARCSI software to generate a modelled atmosphere for the scene being corrected. It is

one of the most widely used, rigorously validated, and heavily documented RT codes in the scientific

remote-sensing community (Vermote et al., 2006). By using 6S, the main atmospheric effects (gaseous

absorption by water vapour, carbon dioxide, oxygen, and ozone, and scattering by molecules and

aerosols) are considered. The primary inputs relate to geometrical conditions, the atmospheric model for

gaseous components, the aerosol model (type and concentration), spectral condition, and ground

reflectance (type and spectral variation) (Tripathi et al., 2005).

Outputs from the model include:

• Direct and diffuse irradiance onto a horizontal surface, referred to as Ehdir and Eh

diff respectively.

• Radiance at the satellite level.

• Apparent surface bi-directional reflectance (ρh) for a horizontal surface.

The key parameters for achieving a good atmospheric correction, requiring a good atmosphere

representation within 6S, are the Aerosol Optical Thickness (AOT) also referred to as Aerosol Optical

Depth (AOD) and the atmospheric water vapour (Table 3). AOT most strongly affects the visible


Status: Final

Version: v0.1

Date: 18/10/2017

Page 24 of 56

components of the spectrum, particularly the blue. Water vapour most strongly affects the shortwave

infrared (SWIR) components of the spectrum.

Table 3 - The relative percentage of contribution for each input parameter to 6S (Powell et al., 2017).

AOT Altitude Water Vapor Ozone View Angle

Landsat 5 78.74 % 4.51 % 16.74 % 0.0 % 1.79 %

Landsat 7 80.03 % 4.80 % 15.45 % 0.0 % 1.68 %

Landsat-8 87.98 % 2.30 % 10.52 % 0.0 % 0.05 %

Sentinel-2 86.67 % 2.35 % 10.96 % 0.0 % 1.85 %

AOT has the most effect on the overall atmospheric correction accuracy, however it also varies on short

spatial and temporal baselines. Therefore, the estimation of AOT needs to be undertaken on a per scene

basis. The method applied is based on the application of a dark object subtraction (DOS) applied to the

at-sensor reflectance for the blue wavelength. For the darkest feature within the scene, the resulting

surface reflectance estimation from the DOS is then used to estimate an AOT (at 550nm) for the scene.

This is performed by iteratively testing AOT values (at steps of 0.05) until a value is identified that most

closely corresponds to the surface reflectance estimated via the DOS. This has been found to be robust

in identifying a single global value for the scene. However, it should be noted that variation of AOT is

often identified within the scene and approaches have been proposed which identified a AOT surface for

the scene. Future work within ARCSI is to develop and include such methods.

Standardised Surface Reflectance

The method of Shepherd & Dymond (2003) was implemented to provide a standardised reflectance

product. Standardised reflectance refers to a product which is normalised for the solar and sensor view

angles and in this case topography. In terms of providing a topographic correction, this method only

works for images where the solar elevation is between 50 and 70 degrees (i.e., from late spring to early

autumn) but does not produce artefacts outside of this range. Standardised reflectance is defined using

the following equation

𝜌ℎ𝑑ir =

𝜋𝐿

𝐸dir𝛾⁄ + 𝛽𝐸dif

where 𝜌ℎ𝑑ir is the direct reflectance for a horizontal surface, L is the radiance at the bottom of the

atmosphere, 𝐸dir is the direct component of the irradiance and 𝐸dif is the diffuse component, the ratio 𝛽

is evaluated using a bidirectional reflectance distribution model but in the study a constant of 1 was used

(Shepherd & Dymond, 2003):

𝛾 =cos 𝑖 +cos 𝑒

cos 𝑖ℎ +cos 𝑒ℎ

where 𝑖 and 𝑒 are the incidence, and exitance angles on an inclined surface, and 𝑖ℎ and 𝑒ℎ are the

incidence and exitance angles on a horizontal surface (Figure 2).


Status: Final

Version: v0.1

Date: 18/10/2017

Page 25 of 56

Figure 2 - Standardised reflectance normalised for topographic and view angle (adapted and redrawn from Shepherd &

Dymond 2003)

Cloud Masking

The FMASK algorithm of Zhu and Woodcock (2012), implemented within the python-fmask

(http://pythonfmask.org/) library has been used to classify cloud and cloud shadow features within the

scene. Unfortunately, Sentinel-2 does not have a thermal image band (unlike Landsat) and therefore the

quality of the result cloud mask is of poorer quality than that produced for Landsat imagery. This is an

area where further work is required to derive an ARD Sentinel-2 product.

Clear-Sky Masks

Following cloud and cloud shadow masking it is common to get many small selections/pieces of imagery

between the clouds left over (e.g. Figure 3). In many cases, those small regions of imagery are either not

useful for a given application and/or atmospheric correction of the regions is of poor quality due to

adjacency effects from the clouds and atmosphere associated with a cloudy environment.


Status: Final

Version: v0.1

Date: 18/10/2017

Page 26 of 56

Figure 3 - There are many small pieces of imagery between the clouds following cloud masking.

A solution to this problem is to select only the areas of the scene above a particular size threshold as

being ‘useful', discarding the rest, within ARCSI this product is called ‘Clear Sky'. A two-step process was

deployed where first a buffer of 30 km was taken from all clouds. Within those regions, only those with

a size greater than 3000 pixels were selected and then they were grown, so they were not within 10 km

of a cloud object.

It should be noted, that for Sentinel-2 the number of false positives within the cloud masking is quite high

and therefore, until the cloud masking is improved, the clear sky product can remove a large amount of

useful imagery.

Topographic Shadow Masks

A ray tracing algorithm (Figure 4), like the process implemented for the cloud shadow masking, has been

implemented. The algorithm produces a binary mask for each pixel within the scene identifying pixels

that are within regions of direct shadow and therefore are not going to provide useful and reliable data

for automatic analysis. A binary mask is provided as output.


Status: Final

Version: v0.1

Date: 18/10/2017

Page 27 of 56

Figure 4 - Ray tracing for identifying regions of topographic shadow.

Image Band Sharpening

The sharpening product to enhance the 20 m image bands using the 10 m bands to produce a sharper

stacked 10 m product works through the application of local linear regression models. The method was

first proposed by Dymond & Shepherd (2004) for pan-sharpening Landsat-7 imagery and has

subsequently been extended for Sentinel-2. Within ARCSI it is applied to the radiance image where the

20 m bands have been oversampled to 10 m resolution image using a nearest neighbour interpolation. A

7x7 pixel filter is applied to the 10-m image stack where within the 7x7 window a linear regression is

performed independently for each lower resolution band to each higher resolution band. The linear

model for each lower resolution image band with the best fit, above 0.5, is then used to predict the image

pixel value for the band. If not fit above 0.5 is identified, then the image pixel value is not altered.

Processing Multiple Granules

One of the issues with many approaches of atmospheric correction is that individual images (e.g.,

Sentinel-2 granules) are independently processed creating boundary effects and artificial changes in

reflectance between scenes due to the atmospheric correction parameterisation. To mitigate those

issues within ARCSI a new option has been implemented through this project which allows a list of

adjacent image header files to be passed to the ARCSI command and for them to be processed together

such that the parameters are harmonised to avoid tile boundary effects. Currently, within ARCSI the

inverted AOT values are only considered for harmonisation and in this first implementation a mean AOT

value is derived from the values inverted from each of the input scenes. The mean value is then used to

correct all of those scenes. While this approach has a number of limitations and can be developed further,

this functionality provides a starting point for further harmonisation as ARCSI develops in the future and

is applicable to all the sensors (e.g., Landsat) which ARCSI supports. For multi-granule processing ARCSI

provides support for multi-core processing allowing all available computational cores to be used,

speeding up processing. Two multi-core solutions have been implemented, firstly via the python multi-

processing module which supports multi-threading on a shared memory machine (e.g., single desktop

computer or virtual machine such as AWS). Secondly, via the Message Passing Interface (MPI) which


Status: Final

Version: v0.1

Date: 18/10/2017

Page 28 of 56

supports processing on computational clusters, where memory is not shared. Examples of such machines

include NERC’s JASMIN cluster and the Supercomputing Wales cluster used for testing in this project.

Access to Sentinel-2 Imagery

For downloading the Sentinel-2 imagery commands have been provided within ARCSI to a) download

database of all the Sentinel-2 scenes globally (arcsisetupsen2db.py) and b) query the database and

retrieve a list URLs to download the scenes using the Google cloud tools (gsutils; arcsigensen2downlst.py)

from the Google Sentinel-2 archive: https://cloud.google.com/storage/docs/public-datasets/sentinel-2

The Google repository of Sentinel-2 imagery (note, Landsat imagery is also available; ARCSI has equivalent

commands for searching and downloading Landsat imagery) has advantages in that the whole global

achieve of imagery is available and the bandwidth of the Google service is significant allowing imagery to

be quickly and efficiently downloaded. For example, 11,000 Landsat scenes were downloaded from the

Google service in only 48 hours on to the Supercomputing Wales HPC system on the UK Janet academic

network.

Results

Quality of the Cloud Masking

To assess the accuracy of the FMASK cloud masking approach, 6000 random points (2000 in each class)

were manually interpreted against the input imagery. The accuracy of the classification is shown in Table

4, where the overall accuracy is 73 %, with a kappa of 0.6. However, the primary source of the error is an

over estimation of the clear sky area, which is reflected in the low producers accuracy (58 %) and users

accuracy for the cloud (67 %) and cloud shadow (69 %). The most common errors are associated with

urban areas and ploughed or recently cut fields where they are classified as clouds as they are spectrally

similar to some cloud regions within the scene. As a consequence of over estimation in cloud cover there

is also an over estimation in the extent of cloud shadow as for each cloud feature identified a search is

performed for a corresponding cloud shadow. This effectively doubles the error for false clouds.

Table 4 - Accuracy Assessment of the cloud masking. Overall accuracy is 73% with a kappa of 0.6.

Clear Cloud Shadow User (%)

Clear 1693 223 84 84.65

Cloud 649 1333 18 66.65

Shadow 578 48 1374 68.7

Producer (%) 57.98 83.1 93.09 73.33

https://cloud.google.com/storage/docs/public-datasets/sentinel-2


Status: Final

Version: v0.1

Date: 18/10/2017

Page 29 of 56

Quality of the Surface Reflectance Spectra

Unfortunately, there are no ground spectra available which are coincident with the Sentinel-2

acquisitions. However, a visual inspection of the spectra for common land covers with expert knowledge

on the expected shape and magnitude of the values can be undertaken (Figure 5). Spectra can also be

compared to other sensors (e.g., Landsat 5; Figure 6).

The spectra illustrated in Figure 5 were selected from a set of regions of interest for a Sentinel-2 scene

acquired on April 8th 2017 while the spectra illustrated in Figure 6 were from a Landsat 5 scene over a

similar area from April 2011. There is a strong correspondence in the shape and magnitude of the

reflectance for the land covers illustrated and these also correspond with the expected shapes and

magnitudes of those land covers from our expert knowledge.

Image Pixel Comparison to ESA ARD Product

To compare the surface reflectance product produced from ARCSI to the ESA ARD product released

through the Copernicus Open Access Hub a number of scenes have been downloaded, visually compared

(Figure 7) and a histogram based comparison of the images undertaken (Figure 8). Through the visual

analysis, a number of artefacts (Figure 7b) were identified in the ESA ARD SREF (July 2017). The artefacts

appear to be associated with the topographic correction routine, where slopes have been over corrected.

However, the ARCSI topographic correction routine has successful flattened the topography (Figure 7a).

To directly compare the ESA ARD and ARCSI ARD surface reflectance products on a per pixel level images

were plotted against one another (e.g., Figure 8). The blue image band had an r2 higher than 0.8 while all

other image bands had an r2 higher than 0.9. The spread from the one-to-one line is in part due to the

ESA ARD image artefacts, particularly within the visible channels. The single AOT values used within ARCSI

verses the AOT surface used within the ESA product will also contribute to the variance between the two

products. However, the overall correspondence between the two products is very close and comparable.


Status: Final

Version: v0.1

Date: 18/10/2017

Page 30 of 56

Figure 5 - Example spectra examples from Sentinel-2. A) Conifer Forest, B) Arable Grass, C) Water, D) Urban, E) Non-

Photosynthetic Vegetation and F) Sand.

a) b)

c) d)

e) f)


Status: Final

Version: v0.1

Date: 18/10/2017

Page 31 of 56

Figure 6 - Example spectra examples from Landsat 5. A) Conifer Forest, B) Arable Grass, C) Water, D) Urban, E) Non-

Photosynthetic Vegetation and F) San

a)

f)

b)

e)

c) d)


Status: Draft

Version: v0.1

Date: 18/10/2017

Page 32 of 56

Figure 7. a) ARCSI ARD Surface reflectance b) ESA ARD Surface reflectance product (July 2017) which has a number of artefacts where the topography has been over corrected.


Status: Draft

Version: v0.1

Date: 18/10/2017

Page 33 of 56

Figure 8 - Per-pixel comparison of the ESA ARD and ARCSI ARC SREF product for 17th June 2017 for tile 30UVD. All bands

other than the blue have an r2 of above 0.9 demonstrating a strong correspondence.


Status: Final

Version: v0.1

Date: 18/10/2017

Page 34 of 56

Standardised Reflectance

Producing standardised reflectance normalises the surface reflectance with respect to the solar angle

and topography. Therefore, a topographic correction is applied; Figure 9 illustrates this correction. The

application of a topographic correction has been shown in several studies to increase the accuracy of the

resulting derived products (e.g., land cover classification).

Figure 9 - An example of a Sentinel-2 scene with a) Surface Reflectance and b) Standardised Surface Reflectance which

performs a topographic correction.

Imagery Sharpening

Figure 10 shows a region of a Sentinel-2 image where the 20 m image bands have been resampled to 10

m using a cubic convolution interpolation (Figure 10a) versus the same area which has been processed

using the image sharpening option (Figure 10b). The image sharpening makes the spatial detail much

sharper maintaining the 10-m resolution while using the 20 m image bands.


Status: Draft

Version: v0.1

Date: 18/10/2017

Page 35 of 56

Figure 10 - An example of a) an image without sharpening and b) an image which has been sharpened


Status: Draft

Version: v0.1

Date: 18/10/2017

Page 36 of 56

ARCSI Recommendations & Conclusions

Recommendations

The report has presented the updated version of ARCSI to support the production of ARD Sentinel-2

imagery. ARCSI provides a command line user interface which is easy to batch process and automate with

scripting. In terms of further development of ARCSI for the production of Sentinel-2 ARD data the

following should be considered:

• The over estimation of cloud and cloud shadow with the cloud classification approach reduces the

amount of available imagery. Further work is required to improve the overall accuracy of the cloud

masking approach. For a monitoring system over a specific area (e.g., the UK) a time series

approach would be recommended as the whole image archive can be maintained and utilised.

However, for a system where single or a small number of images are required for a given area a

time series approach is limited by the need to download and process a time series for the area of

interest. Therefore, it is recommended that both time series and single scene approaches are

explored and developed as options within ARCSI.

• Improvements in the recovery of AOT such that a surface across the image can be reliably retrieved

even with large areas of missing data due to cloud cover. Issues were found with reliably recovering

an AOT surface for scenes with significant gaps due to cloud cover, therefore for consistency a

single value was recovered from each scene and used for the whole scene. Further research on the

recovery of AOT is needed to improve this aspect of the atmospheric correction.

• Only a simple implementation of an approach to harmonise between neighbour’s image granules

has currently been provided within ARCSI (i.e., taking a mean AOT). This is providing a starting point

from which further developments can be made and considered, particularly when considering the

generation of AOT surfaces within and between image granules.

• Further work on correcting for bidirectional reflectance would be beneficial. Specifically, reviewing

more closely the degree to which the standardised reflectance product has corrected for changes

in sun angle between scenes. The method of Shepherd & Dymond (2003) has been applied

throughout providing a significant increase in the quality of the scenes from late spring to early

autumn. However, methods such as Flood et al., (2013) has been presented and may improve the

quality of the output product, but would require parameterisation which not trivial.

• A formal test and validation of the ARD surface reflectance product further datasets are required.

During this project, a basic visual check to ensure the spectra produced are of the right shape, and

a comparison to the ESA ARD product were carried out. However, these are insufficient and

provide no information on the uncertainty associated with the derived spectra. To perform a

formal validation, third party datasets of ground spectra that are temporally coincident with the

satellite acquisition are required for sites which are spectrally homogeneous at the appropriate

resolution for the sensor. Alternatively or alongside, spectrally invariant features can be used but

these are difficult to source for a range of spectral targets. Methods and appropriates have been

proposed for lower resolution instruments, such as MODIS, and similar work is therefore required


Status: Final

Version: v0.1

Date: 18/10/2017

Page 37 of 56

to validate these Sentinel-2 ARD products. In undertaking this work significant progress in

demonstrating ARCSI meets the CARD4L standard (Point 6) would have also been carried out.

• CEOS have proposed a standard for ARD data under CARD4L standard. The details are this are still

be finalised but it is expected that a peer-review process will be undertaken to approve a given

system produces data which meets the CARD4L standard. It would be desirable to have ARCSI

tested ensuring that ARCSI produces data which meets the CARD4L standard.

Conclusions

The project has provided an open source tool for providing an ARD surface reflectance product for the

ESA Sentinel-2 instruments. As demonstrated, the product produced by ARCSI provides a comparable

result to the product currently being released by ESA through the Copernicus Open Access Hub but

without artefacts within the topographic correction. The image band sharpening approach implemented

with ARCSI provides a significant improvement over traditional approaches of image resampling when

using the 20 m image bands within a 10 m resolution image stack. The ability to process multiple image

granules together, avoiding boundary effects between the image granules is also a significant step and

advantage of ARCSI over other similar software systems. While further work is recommended,

particularly on the approaches to cloud masking and product validation, the support for Sentinel-2 within

ARCSI provides a good base by providing a similar ARD product as to that which has been demonstrated

in the Landsat archive for Wales. The outputs are also comparable to the ARD product provided by ESA

and ready to be used with a production environment to deliver an ARD service for Sentinel-2.


Status: Final

Version: v0.1

Date: 18/10/2017

Page 38 of 56

APPENDIX A - SENTINEL-2 CHARACTERISTICS AND PROCESSING CONSIDERATIONS

Mission Rationale

The objective of Copernicus is to use multi-source data to get timely and quality information, services

and knowledge, and to provide autonomous and independent access to information in relation to the

environment and security on a global level. The twin satellites of Sentinel-2 provide continuity of Landsat-

type image data, through providing multispectral observations at high temporal and spatial resolutions

that are unprecedented within a fully open access data policy. This data benefits services and applications

such as land management, agriculture and forestry, disaster control, humanitarian relief operations, risk

mapping and security [1]. Most importantly, unprecedented access to this type of data shall enable

services to be developed and insights attained that were not previously possible.

Satellite Characteristics

The Sentinel-2 constellation comprises twin polar-orbiting satellites in the same orbit, phased at 180°

apart. A sun-synchronous orbit at an average altitude of 786 km ensures a consistent angle of sunlight

upon the Earth’s surface is maintained. A resulting mean local solar time of 10.30 am is comparable to

that of Landsat and almost identical to SPOT-6, enhancing sensor time-series and spatial compatibility

and continuity within monitoring applications.

Sentinel-2A launched June 2015 and Sentinel-2B launched March 2017. While lifespan of the

constellation is designated as 7.25 years, core components can accommodate up to 12 years of operation

(up to 2027). Continuity of the Copernicus Sentinel missions is ensured with launches of Sentinels-2C and

D to complement current Sentinel-2A and B.

The optical multi-spectral imaging (MSI) telescope payload design enables a 290-km wide swath and,

consequentially high frequency revisit of 5 days at the equator and revisit of 2-3 days at mid-latitudes.

Systematic acquisition is completed over land and coastal areas between latitudes 56° South and 83°

North as shown in Figure 11.

A summary of the 13 spectral bands with respects to their radiometric configuration and spatial

resolution up to 10 m is provided in Table 5. These are contrasted to the Landsat-8 OLI sensor and SPOT

6/7 sensor characteristics in Figure 12.


Status: Final

Version: v0.1

Date: 18/10/2017

Page 39 of 56

Figure 11 - Sentinel-2 geographical coverage (green) [3].

Table 5 - Summary of Sentinel-2 MSI band radiometric and spatial characteristics [4].

WaveBand Central Wavelength (μm) Spatial Resolution (m)

Band 1 - Coastal Aerosol 0.443 60

Band 2 - Blue 0.490 10

Band 3 - Green 0.560 10

Band 4 - Red 0.665 10

Band 5 - Vegetation Red Edge 0.705 20



Band 8 - NIR 0.842 10

Band 8A - Vegetation Red Edge 0.865 20

Band 9 - Water Vapour 0.945 60

Band 10 - SWIR - Cirrus 1.375 60

Band 11 - SWIR 1.610 20

Band 12 - SWIR 2.190 20


Status: Final

Version: v0.1

Date: 18/10/2017

Page 40 of 56

Figure 12 - Comparison of Sentinel-2 MSI radiometric and spatial characteristics against Landsat-8 OLI and SPOT 6/7 [2].

Product Levels Available

Sentinel-2 products are systematically made available to users at processing Level-1C: top-of-atmosphere

reflectances in fixed cartographic geometry (combined UTM projection and WGS84 geodetic system)

along with cloud, land and water masks. Level-1C images are a set of tiles of 100 km2 and approximately

500 MB. This product contains applied radiometric and geometric corrections; including

orthorectification and spatial registration to sub-pixel accuracy [5]. Scene products comprising

Land/Water, Cloud Masks and ECMWF data are also provided alongside the multi-spectral bands.

For robust exploitation of Sentinel-2 products alongside other sensors, data is required in surface

reflectance that is spatially and temporally consistent and comparable. Responsibility for generation of

this product currently falls on the user side, with identical processing from Level-1C to Level-2A

duplicated for the same product by different users. To implement this processing step ESA made the

Sen2Cor atmospheric correction tool available. This Level-2A tool, and others, currently have very limited

accessibility, documentation or proven implementation.


Status: Final

Version: v0.1

Date: 18/10/2017

Page 41 of 56

Existing Data Dissemination

Open dissemination of all Sentinel mission data is provided through the Copernicus Programme via a

combination of Core and Collaborative Ground Segments.

Key challenges associated with the dissemination of Sentinel-2 and other Sentinel datasets include:

• Storage required

• Timely data provision

• Accessibility of data

• Engaged community

Figure 13 - Overview of Sentinel Ground Segment infrastructure [6].

Sentinel Core Ground Segment

The Sentinel Core Ground Segment comprises a European distributed infrastructure that coordinates to

monitor and control the Sentinel satellites and routinely download, process and disseminate quality

controlled data products to users.

Fundamentally, the Core Ground Segment is responsible for ensuring systematic and timely processing

of all Sentinel data to their designated product levels and to serve these products through a single virtual

access point (Sci Data Hub) [7].


Status: Final

Version: v0.1

Date: 18/10/2017

Page 42 of 56

Sentinel-2 Level-1C data products are routinely available within 3-24 hours of being sensed by the

satellite, with this latency largely related to the delay following data acquisition to overpassing a ground

station.

Sentinel Collaborative Ground Segment

The Sentinel Collaborative Ground Segment complements the exploitation of the Sentinel missions in

many areas, such as:

• Data acquisition and (near-) real-time production. Local ground stations are configured to receive

Sentinel data as the satellite passes overhead.

• Complementary products and algorithm definitions – these ‘collaborative data products’ may be

tailored for regional coverage or specific applications. These types of products may extend the

Sentinel core product chains.

• Data dissemination and access, supporting redistribution of Sentinel core products by establishing

additional pick-up points.

• Development of innovative tools and applications.

• Complementary support to calibration/validation activities.

The Sentinel Collaborative Ground Segment is funded by third parties. It provides the basis for

international cooperation, and is supported where necessary by generic interfaces and/or Copernicus

Ground Segment interfaces.

The UK collaborative ground segment for Copernicus, SEDAS, comprises Airbus DS Geo-intelligence who

operate the Processing and Archiving facilities for Sentinel-1 and 2; Satellite Applications Catapult who

are responsible for managing and disseminating the data to UK commercial users and STFC-CEDA who

are responsible for managing and disseminating the data to UK academic users.

Informal Data Dissemination

An increasing number of organisations collect all or a subset of Sentinel data products from the Scientific

Data Hub (or collaborative ground segment equivalent) and make these openly available alongside other

geospatial datasets (e.g. AWS, Google, Planet). Such data dissemination results from the recognised value

of exploiting satellite datasets directly alongside additional geospatial (and ancillary) datasets as well as

close to dynamically available compute resources. While such informal data dissemination is largely

open, providing contemporary satellite datasets in this type of environment is an attractive prospect for

users.


Status: Final

Version: v0.1

Date: 18/10/2017

Page 43 of 56

Image processing and correction considerations

Optical Earth Observation satellite imaging sensors, such as Sentinel-2, passively measure radiation

reflected from the Earth’s surface. Radiance measured directly by a sensor therefore includes

contributions from Rayleigh and aerosol scattering, gas absorptions of the atmosphere, surface

bidirectional reflectance distribution function (BRDF) effects of anisotropic surfaces and topographic

illumination effects resulting in shadowing. Optical imagers are also particularly vulnerable to cloud

cover, a key pre-Sentinel-2 limitation to satellite image exploitation over the UK.

Robust processing of optical satellite datasets is required to mitigate or remove these effects to obtain

spatially and temporally comparable measurements of surface reflectance that are truly representative

of Earth surface characteristics. Corrections to a processing level that is ‘analysis ready’ are increasingly

required in an operational environment to enable full exploitation of the unprecedented volume of

satellite Earth observation data from the full variety of sensors available.

This section briefly summarises methods for implementing corrections required to process Sentinel-2

Level-1C products to Level-2A: (i) atmospheric correction, (ii) BRDF correction, (iii) topographic

correction, and (iv) cloud masking.

Atmospheric Correction

Atmospheric correction is the process of correcting from a TOA radiance product to a BOA surface

reflectance product. This retrieval of surface reflectance requires mitigating effects predominately

associated with Mie, Rayleigh and particle (aerosol) scattering and absorption of atmospheric gases (inc.

water vapour, CO2, etc.) [11], [12]. With techniques for performing these corrections very well established,

implementation of physics-based as opposed to empirical models is feasible, and methods for VIR (visible

and infrared) bands particularly mature [13].

All techniques entail the use of a radiative transfer model, the parameterisation of which (aerosol optical

depth, ozone, water vapour etc.) is directly related to the estimation quality of atmospheric parameters [14], hence quality of corrected surface reflectance product. Atmospheric parameters being estimated by

a radiative transfer model include ratio of diffuse to total irradiance for both sun and sensor directions,

atmospheric albedo, path radiance and transmittance for sun and sensor directions.

MODerate resolution TRANsmission (MODTRAN) [15] and the Second Simulation of a Satellite Signal in

the Solar Spectrum (6S) [16] are two of the most commonly used radiative transfer models [14]. MODTRAN

is complex yet highly flexible, whereas 6S is relatively of lesser complexity. Accuracy of atmospheric

parameter estimates are, however, associated much more with the quality of input data than choice of

model. A variety of interfaces exist to enable parameterisation of these models, within both proprietary

and open software. For example, Fast Line-of sight Atmospheric Analysis of Spectral Hypercubes

(FLAASH) [17] associated with MODTRAN, and Py6S and (ARCSI [18] associated with 6S.


Status: Final

Version: v0.1

Date: 18/10/2017

Page 44 of 56

BRDF Correction

Mitigation of surface BRDF is an important step to correct view and illumination angle effects and to

normalize surface reflectance both in a single image and across multiple images. Due to different view

and solar angles and anisotropic surfaces, observed surface reflectance is different even if the surface

cover is the same [19]. This occurs for a single scene with different view and solar angles and different

scenes sensed at different seasons and geographical regions due to the solar angle variation [13], [19].

Like correcting for atmospheric effects, the quality of BRDF mitigation is directly related to the quality of

parameterisation. If these parameters may be derived then BRDF and atmospheric correction may be

coupled together into a single correction [13], [20].

For lower spatial resolution, finer temporal resolution satellite imagers (i.e. MODIS), the frequent revisit

(up to twice a day with MODIS) enables BRDF parameters to be derived from the data itself [21]. For finer

spatial resolution datasets, and typically limited temporality, BRDF parameters must be obtained from

other sources (i.e. MODIS data) [12].

Topographic Illumination Correction

Terrain illumination (topographic) correction is a processing step particularly required for areas of

elevated terrain. On inclined surfaces irradiation measured by a satellite sensor is regularly modified such

that slopes facing toward the sun receive more solar irradiance and appear brighter in satellite images

than those facing away from the sun [22]. This effect combined with those associated with BRDF result in

hill shade across a satellite image.

Use of a good quality DEM that is very well aligned with the satellite data is necessary for derivation of

accurate terrain parameter computation; including slope, aspect, incident and exiting angles, relative

azimuth angles, etc. Traditionally empirical models have been utilised for applying the terrain correction

and within a separate process to atmospheric and BRDF correction stages.

Cloud Masking

A fundamental difficulty in exploiting the full acquisition capacity of optical satellite imagery is the

prevalence of cloud and associated shadowed regions, with global annual mean cloud cover estimated

at approximately 66% [23]. Improved temporality of Sentinel-2 over Landsat missions greatly improves the

probability for frequent cloud free coverage of an area of interest. It remains important to reliably

identify these clear skies areas within an ARD product to fully exploit available datasets. Significant

research has therefore been associated with extracting cloud free areas from legacy Landsat data in an

automated way, with mature techniques that are being transferred to Sentinel-2 datasets.

The most effective cloud masking techniques exploit multiple Thermal Infrared (TIR) bands for masking

clouds, and others Cirrus bands specifically designed for effectively mitigating the effects of semi-

transparent clouds, whose signal comprises both clouds and surface underneath.


Status: Final

Version: v0.1

Date: 18/10/2017

Page 45 of 56

Absent of a TIR band, cloud masking of Sentinel-2 datasets can leverage information within Aerosol,

Water Vapour and Cirrus bands. Sentinel-2 Level-1C products contain an associated vector cloud mask [24], however this is often of insufficient quality, with most users requiring over-identification of cloud to

ensure effective clear-skies regions.

Masking cloud shadow has traditionally been achieved using spectral masking techniques, however these

unsurprisingly also result in masking of spectrally similar non-shadowed land covers. State of the art

methods therefore exploit cloud masks alongside sensor geometry to infer regions of probable shadow

prior to implementing spectral masking techniques [25].


Status: Final

Version: v0.1

Date: 18/10/2017

Page 46 of 56

APPENDIX B - SENTINEL-2 ARD INTERNATIONAL CONTEXT

Background

Satellite image product generation as ARD and the growing need to lower the barrier to entry for

exploitation of these datasets is not a new concept. Many organisations globally have been working

towards this aim for several years. From defining standards for accessibility of geospatial data to space

agencies understanding the need to justify from a business perspective the cost of Earth observing space

craft, through making associated data readily exploitable. This has led to an environment in which work

is increasingly being undertaken on how these missions and data can be accessed and exploited by non-

expert users.

This section acts as a consolidated summary of international activities that embed the context of the

heritage of ARD and organisations that are acting in the space of or directly relevant to the uptake of a

Sentinel-2 ARD product standard.

The process of Calibration and Validation could be considered the precursor to wide-spread production

of ARD. The Calibration and Validation of satellite data ensures that when satellite data is processed to a

higher-level product it is directly comparable to data from the same satellite but from a different

acquisition or in some cases to enable the comparison of data from different satellites. Calibration may

be defined as the process of quantitatively defining a system’s responses to known, controlled signal

inputs. Contrastingly, Validation is the process of assessing, by independent means, the quality of the

data products derived from those system outputs [26].

The volume of data acquired by Earth observing satellites is increasing at an exponential rate and

providing unprecedented insights into the dynamics of our planet. Due to the nature of satellites often

monitoring within different regions of the electromagnetic spectrum, at different spatial resolutions,

from varying geometries, and processed using different methodologies, it can be difficult to compare one

satellite dataset with another, making it essential to have globally recognised guidelines for the

Calibration and Validation process. By having a calibration process the data can be quantitatively defined

to controlled system inputs, the validation process in turn independently assessed by independent means

the quality of the data derived from the system outputs.

Organisation such as the Group on Earth Observations (GEO) and CEOS support the development of these

processes by facilitating participation in workshops to define such things as Calibration/Validation site

characterisation and classification, satellite and in site data access, the methodologies and guidelines for

the Calibration/Validation and Harmonisation and quality information. Five Calibration and Validation

domains are generally considered: radiometric, spectral, spatial, temporal, polarization and inter-

calibration of instruments against a common reference instrument allows achieving consistency across

satellite measurements at a given point in time and space.

Calibration and Validation activities, while varying across sensors, do not produce fully prepared and

consistent data through space and time that may be incorporated directly into analyses. In addition to


Status: Final

Version: v0.1

Date: 18/10/2017

Page 47 of 56

costs associated with generating, storing and disseminating additional higher level products, this is due

to variation in user application requirements; hence final corrections, have traditionally been

implemented independently by users. Such bespoke processing stages are essential in the exploitation

of satellite EO datasets for rigorous and specific scientific analyses. There will therefore always be a

requirement for the widespread distribution of lower level products to enable these activities and hence

an increasingly evident trend that definition of standards for these lower level products should sit within

the remit of calibration and validation.

International Community

A growing international community has developed over the last 20 years that specifically deals with the

sharing of geospatial information and the interoperability between different systems, both traditional

geospatial systems and satellite based. These organisations largely deal with the development of

standards. The standards these organisations deal with enable data to be curated in a way that not only

deals with the inherent hostility associated with what are traditionally scientific data formats but lower

the barrier to entry to the use of these data and their parent systems. Some of the key stakeholders are

identified below.

Heterogeneous Missions Accessibility (HMA)

The HMA initiative addresses issues of data publishing, discovery and access by engaging with national

space agencies, satellite or mission owners and operators, and industry, providing a forum for promoting

standaridisation.

Open Geospatial Consortium (OGC)

The OGC is an international industry consortium of companies, government agencies and universities

participating in a consensus process to develop publicly available interface standards.

Group on Earth Observations (GEO) and Global Earth Observation System of Systems (GEOSS)

GEO promotes the exploitation of Earth observation data to address global challenges. A fundamental

part of GEO’s Mission is to build GEOSS, a set of coordinated, independent Earth observation, information

and processing systems that interact and provide access to diverse information for a broad range of users

in both public and private sectors. GEOSS links these systems to strengthen the monitoring of the state

of the Earth and facilitates the sharing of and access to data.

Committee on Earth Observing Satellites (CEOS)

CEOS has long recognised a need for data processing infrastructure to support Earth science objectives.

Through its network of global connections, CEOS has determined that global users share many common

needs, such as data access, preparation, and efficient analyses to support applications. CEOS recognise


Status: Final

Version: v0.1

Date: 18/10/2017

Page 48 of 56

that many of these can be addressed through the Open Data Cube (ODC) initiative, a data architecture

solution that enables proper connection between data, applications and users. ODC is particularly

focused on supporting nations that lack the in-country expertise and computational infrastructure to

utilise EO data. There are currently deployments in Colombia, Vietnam and Kenya. CEOS too have been

working to develop a recognised ARD ‘brand’, CARD4L, which they define as - satellite data that have

been processed to a minimum set of requirements and organized into a form that allows immediate

analysis with a minimum of additional user effort and interoperability both through time and with other

datasets. As part of this work they have been instrumental in facilitation of the Atmospheric Correction

Australian Geoscience Data Cube (AGDC)

The Australian Geoscience Data Cube (AGDC) is a collaboration between Geoscience Australia, CSIRO and

NCI Australia and has been a key contributor to the ODC. The AGDC is a series of structures and tools that

calibrate and standardise datasets, enabling the application of time series and the rapid development of

quantitative information products. It uses open source standards and is designed to help both

government and private industry make informed decisions.

United States Geological Survey (USGS)

The United States Geological Survey (USGS) is a scientific agency of the United States government. A core

responsibility of the USGS is to produce Higher-Level Science data products, data which are processed

consistently to create a continuous record of the effects of climate change on Earth’s surface. This

includes the production of a Landsat ARD product. The USGS is a key contributor to the ODC.

European Space Agency (ESA)

ESA has a long history of publicly funded Earth observing missions and data publishing. With the new

availability of data through the Copernicus programmes and the associated volumes of data, the

European Space Agency is core to the global movement towards standardised data access and

processing. Various activities and work streams by ESA have influenced and driven the production and

dissemination of ARD as well as the emerging ARD standards. Below is a non-exhaustive list of the most

prominent initiates. ESA is a key contributor to the ODC.

CNES / THEIA

The Theia Land Data Centre is a national inter-agency organization designed to foster the use of images

coming from the space observation of land surfaces. Theia offers a broad range of images at different

scales, methods and services related to the land surface observation from space. For Sentinel-2

THEIAproduces and distributes level 2A data, corrected for atmospheric effects and provided with a good

cloud and cloud shadow mask. The method used to produce these data was developed in CESBIO and

implemented in the operational processor MACCS developed for CNES by CS-SI company.

Copernicus


Status: Final

Version: v0.1

Date: 18/10/2017

Page 49 of 56

Copernicus is the world's largest single Earth observation programme, directed by the European

Commission in partnership with the European Space Agency (ESA). It delivers core services that provide

standardised multi-purpose information common to a broad range of application areas relevant to EU

policies. Feeding these services are a series of Sentinel satellite missions (Figure 14). Each Sentinel

mission is based on a constellation of two satellites to fulfil revisit and coverage requirements, providing

robust datasets for Copernicus Services. A summary is seen in the figure below.

Figure 14 - Overview of Copernicus Sentinel satellites

• Copernicus Masters

The Copernicus Masters is an international competition, launched by AZO in 2011 on behalf of ESA, and

it awards prizes to innovative solutions for business and society based on Earth observation data.

• Sentinel Hub

In 2016 the Sentinel Hub, a cloud-based web service that allows non-expert user to browse, process and

download EO data, was the overall Copernicus Masters’ winner. The Sentinel Hub significantly simplifies

the way users can access and process satellite data. With applications like these, the competition has

again clearly demonstrated its potential to drive the innovative use of Earth observation data and make

the Copernicus programme accessible to new user groups. The Sentinel Hub is one of many portals

disseminating EO data.

Thematic Exploitation Platforms

In context of this growing volume of environmental data from space, enabled by the Sentinel missions,

along with the Copernicus Contributing Missions as well as Earth Explorers and other, Third Party

missions, in 2014 ESA started the EO Exploitation Platforms (EPs) initiative, a set of R&D activities that in

the first phase (up to 2017) aims to create an ecosystem of interconnected Thematic Exploitation

Platforms (TEPs) on European footing, addressing:

https://sentinels.copernicus.eu/

https://copernicusdata.esa.int/

https://earth.esa.int/web/guest/missions/esa-operational-eo-missions

https://earth.esa.int/web/guest/missions/3rd-party-missions/overview

https://earth.esa.int/web/guest/missions/3rd-party-missions/overview


Status: Final

Version: v0.1

Date: 18/10/2017

Page 50 of 56

• Coastal,

• Forestry,

• Hydrology,

• Geohazards,

• Polar,

• Urban themes; and

• Food Security (under definition).

In short, an EO exploitation platform is a collaborative, virtual work environment providing access to EO

data and the tools, processors, and Information and Communication Technology resources required to

work with them, through one coherent interface. As such the EP, may be a new ground segments

operations approach, complementary to the traditional operations concept.

Data and Information Services (DIAS)

With the Copernicus Data Policy of open and free access to Sentinel products for any user in the current

operations context, it is projected that the volume of Sentinel data available to users will increase at a

rate of approximately 10PB per year once the Sentinels missions will have reached their full operational

capacity. While open distribution of the Copernicus data and information is actively promoted, the

European Commission (COM) and ESA aim at offering to users the capability to exploit Copernicus data

and information without the complexity and the cost associated to its transfer and storage on user side.

In this respect, the European Commission envisages that through the creation of DIAS, Copernicus data

and information will be routinely available to users through a cloud-based architecture.

CEOS-WGCV Atmospheric Correction Inter-Comparison Exercise (ACIX)

ACIX is an international collaborative initiative to inter-compare a set of atmospheric correction (AC)

processors for high-spatial resolution optical sensors, coordinated by ESA and NASA. The exercise will

focus on Landsat-8 and Sentinel-2 imagery over a set of test areas. The inter-comparison of the derived

BOA products is expected to contribute to the understanding of the different uncertainty contributors

and help in improving eventual ARD production. Participating algorithms/organisations are summarised

in Table 6.


Status: Final

Version: v0.1

Date: 18/10/2017

Page 51 of 56

Table 6 - ACIX summary

Pilot production of Sentinel-2 Level-2A

In May 2017 ESA started the pilot production of Sentinel-2 Level-2A products over Europe. Level-2A

products are nominally published 48-60 hours after the publication of corresponding L1C products, on

the Copernicus Open Access Hub. Any potential gap in the production will be recovered automatically

within 4-5 days [28]. The pre-operational production of Sentinel-2 Level-2A atmospherically corrected data

over Europe (EEA-39) is done using Sen2Cor processor (Version 2.3.1). Sen2Cor is one of the algorithms

explored under ACIX and ESA’s next steps in relation to ACIX and the move from feasibility to global

production include [29]:

• Q2/2017: 2nd workshop ACIX for comparison metrics across 14 algorithms

• Q3/2017: Results from feasibility study L2A best European algorithm

• Q1/2018: Validated Global Reference Image (GRI): best geometric quality globally (<0.3 pixels @3

sigma multi-temporal)

• 2018: start of GRI-based production for S2A and S2B

• 2018: move from SEN2COR towards operational L2A processor for global production

• 2018: start major reprocessing exercise of all S2A/S2B archived data with GRI and operational L2A

processor (archive size ~5PB)

Organisation/

companyAC Processor

Input

DataOutput Format

Eric Vermote

Jean-Claude Roger

2 Jermoe Lours Telespazio Sen2Cor S2 JPEG2000 (*.jpg2)

3 Pflug Bringdred DLR Sen2Cor, ATCOR L8 & S2

4 Andre Hollstein GFZ SCAPE-M S2 JPEG2000 (*.jpg2)

5 Olg Dubovik LOA GRASP N/A

6 Steve Adler-Golden Spectral Sciences, Inc. FLAASH L8 & S2 N/A

7 Larry Leigh SDSU SMACAA L8 & S2 N/A

Erwin Wolters

Sterckx Sindy

Quinten Vanhelllmont

Kevin Ruddick

10 Antoine Mangin ACRI-HE LAC L8 & S2 N/A

11 Olivier Hagolle CNES MACCS L8 & S2 N/A

Fuqin Li

Lan-Wei Wang

13 David Frantz Trier University Ind-prepro L8 & S2 ENVI (*.hdr)

14 Grit Kirches Brockmann Consulting TBD L8 & S2 GeoTiff (*.tif)

9 RBINS ACOLITE L8 & S2 NetCDF, GeoTiff (*.nc, *.tif)

12 Geoscience Australia GA-PABT ENVI (*.hdr)

VITO OPERA L8 & S2 GeoTiff (*.tif)8

Data Format

Name

1 NASA/ UMd L8SR, S2SR L8 & S2 HDF (*.hdf)


Status: Final

Version: v0.1

Date: 18/10/2017

Page 52 of 56

APPENDIX C - A SUMMARY OF STANDARDS

As discussed throughout the present document, assigning an appropriate standard to a data product is a

reasonably straightforward process so long as it complies with relevant internationally-defined criteria.

International standards associated with EO products are very well established across the organisations,

however have not yet been defined within the international community for Sentinel data products. Many

activities are ongoing within this space, with an informal contrast between users that require a standard

to be unequivocally scientifically rigorous and would therefore be very difficult to comply with, compared

to the clear majority of users that require a standard that predominantly ensures consistency in

generated products that may be incorporated into value added services. It is the latter group of users

that will play the lead role in exploiting contemporary satellite datasets to offer unprecedented services

across applications and markets.

A summary of relevant standards that a UK Sentinel-2 ARD product should therefore adhere to is

provided below. It is by no means an exclusive list and includes those that are both formal and informal,

nationally and internationally.

It is clear, however, that a Sentinel-2 ARD standard product must comprise a minimum of two key

elements: (i) a standard associated with the accuracy and precision of corrected data comprising the

product, and (ii) a standard associated with the output product format, including associated metadata.

Therefore, individual standards that would be of relevance to these two elements are identified.

Through the current project, both accessible open source tool and open source datasets made available

to the community adhere to the key geospatial standards and are therefore appropriate for incorporation

into typical remote sensing and GIS systems. Part 2 of this report covers in detail the technical standard

to which the developed tool adheres, the published UK Sentinel-2 ARD dataset.


Status: Final

Version: v0.1

Date: 18/10/2017

Page 53 of 56

Table 7 – Summary of Standards

Organisation Standard Reference

ISO ISO/TS 19159-1:2014 Geographic

information — Calibration and validation

of remote sensing imagery sensors and

data — Part 1: Optical sensors

https://www.iso.org/obp/ui/#iso:std:iso:ts:19159:-1:ed-

1:v1:en

ISO ISO/IEC 18026:2009 Information

technology — Spatial Reference Model

(SRM)

https://www.iso.org/obp/ui/#iso:std:iso-iec:18026:ed-

2:v1:en

ISO ISO 19115-2:2009 Geographic

information — Metadata — Part 2:

Extensions for imagery and gridded data

https://www.iso.org/obp/ui/#iso:std:iso-iec:18026:ed-

2:v1:en

ISO ISO/DIS 19115-2 Geographic information

— Metadata — Part 2: Extensions for

acquisition and processing

https://www.iso.org/obp/ui/#iso:std:iso:19115:-2:dis:ed-

2:v1:en

ISO ISO/TR 19120:2001 Geographic

information — Functional standards

https://www.iso.org/obp/ui/#iso:std:iso:tr:19120:ed-

1:v1:en

ISO ISO 19115-1:2014 Geographic

information — Metadata — Part 1:

Fundamentals

https://www.iso.org/obp/ui/#iso:std:iso:19115:-1:ed-

1:v1:en

ISO ISO/TS 19115-3:2016 Geographic

information — Metadata — Part 3: XML

schema implementation for fundamental

concepts

https://www.iso.org/obp/ui/#iso:std:iso:ts:19115:-3:ed-

1:v1:en

INSPIRE Orthoimagery Thematic Area http://inspire.ec.europa.eu/Themes/124/2892

INSPIRE Coordinate Reference Systems Thematic

Area

http://inspire.ec.europa.eu/Themes/130/2892

INSPIRE Geographical Grid http://inspire.ec.europa.eu/Themes/131/2892

OGC Earth Observation Metadata profile of

Observations & Measurements

http://docs.opengeospatial.org/is/10-157r4/10-

157r4.html#1

OGC OpenSearch Extension for Earth

Observation

http://docs.opengeospatial.org/is/13-026r8/13-

026r8.html

OGC Ordering Services Framework for Earth

Observation Products Interface Standard

https://portal.opengeospatial.org/files/?artifact_id=4392

8

UK GEMINI Metadata Standards http://www.agi.org.uk/agi-group/standards-

committee/uk-gemini

CEOS CARD4L – OSR TBD

https://www.iso.org/obp/ui/#iso:std:iso:ts:19159:-1:ed-1:v1:en


https://www.iso.org/obp/ui/#iso:std:iso-iec:18026:ed-2:v1:en




https://www.iso.org/obp/ui/#iso:std:iso:19115:-2:dis:ed-2:v1:en

https://www.iso.org/obp/ui/#iso:std:iso:19115:-2:dis:ed-2:v1:en

https://www.iso.org/obp/ui/#iso:std:iso:tr:19120:ed-1:v1:en

https://www.iso.org/obp/ui/#iso:std:iso:tr:19120:ed-1:v1:en

https://www.iso.org/obp/ui/#iso:std:iso:19115:-1:ed-1:v1:en

https://www.iso.org/obp/ui/#iso:std:iso:19115:-1:ed-1:v1:en






http://docs.opengeospatial.org/is/10-157r4/10-157r4.html#1

http://docs.opengeospatial.org/is/10-157r4/10-157r4.html#1

http://docs.opengeospatial.org/is/13-026r8/13-026r8.html

http://docs.opengeospatial.org/is/13-026r8/13-026r8.html



http://www.agi.org.uk/agi-group/standards-committee/uk-gemini

http://www.agi.org.uk/agi-group/standards-committee/uk-gemini


Status: Final

Version: v0.1

Date: 18/10/2017

Page 54 of 56

REFERENCES

[1] “Sentinel-2 Overview,” [Online]. Available:

https://sentinel.esa.int/web/sentinel/missions/sentinel-2/overview.

[2] “Sentinel-2 Revist & Coverage,” [Online]. Available: https://sentinel.esa.int/web/sentinel/user-

guides/sentinel-2-msi/revisit-coverage.

[3] “Sentinel-2 Resolutions,” [Online]. Available: https://earth.esa.int/web/sentinel/user-

guides/sentinel-2-msi/resolutions.

[4] “Sentinel-2 Heritage,” [Online].

[5] “Sentinel-2 Product Types,” [Online]. Available: https://earth.esa.int/web/sentinel/user-

guides/sentinel-2-msi/product-types.

[6] “Copernicus Ground Segment Infrastructure,” [Online]. Available:

http://www.esa.int/Our_Activities/Observing_the_Earth/Copernicus/Ground_Segment_overview

[7] “Copernicus Open Access Hub,” [Online]. Available: https://scihub.copernicus.eu/dhus/#/home.

[8] C. Song, C. E. Woodcock, K. C. Seto and M. P. Lenney, “Classification and Change Detection Using

Landsat TM Data. When and How to Correct Atmospheric Effects?,” Remote Sensing of

Environment, 2001.

[9] N. Flood, T. Danaher, T. Gill and S. Gillingham, “An operational scheme for deriving standardised

surface reflectance from Landsat TM/ETM+ and SPOT HRG imagery for eastern Australia,” Remote

Sensing, pp. 83-109, 2013.

[10] T. Malthus and F. Li, “A technical handbook supporting calibration and validation activities of

remotely sensed data products,” 2015.

[11] A. S. Mahiny and B. J. Turner, “A Comparison of Four Common Atmospheric Correction Methods,”

Photogrammetric Engineering & Remote Sensing, 2007.

[12] A. Berk, G. P. Angerson, L. S. Bernstein, P. K. Acharya, H. Dothe, M. W. Matthew, S. M. Adler-

Golden, J. H. Chetwd, Jr., S. C. Richtmeier, B. Pukall, C. L. Allred, L. S. Jeong and M. L. Hoke,

“MODTRAN4 Radiative Transfer Modeling for Atmospheric Correction,” SPIE's International

Symposium on Optical Science, Engineering, and Instrumentation. International Society for Optics

and Photonics, 1999.

[13] D. Tanre, T. L. Deuze, M. Herman and J. Morcrette, “Second Simulation of the Satellite Signal in

the Solar Spectrum, 6S: An Overview,” IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE

SENSING, 1997.

[14] S. Adler-Golden and et al., “FLAASH, a MODTRAN4 atmospheric correction package for

hyperspectral data retrievals and simulations,” Proc. 7th Ann. JPL Airborne Earth Science

Workshop, 1998.

[15] P. Bunting and D. Clewley, “Atmospheric and Radiometric Correction of Satellite Imagery (ARCSI),”

[Online]. Available: http://rsgislib.org/arcsi/index.html.


Status: Final

Version: v0.1

Date: 18/10/2017

Page 55 of 56

[16] J. Roujean, M. Leroy and P. Deschamps, “A bidirectional reflectance model of the Earth's surface

for the correction of remote sensing data,” Journal of Geophysical Research: Atmospheres, 1992.

[17] F. e. a. Li, “An evaluation of the use of atmospheric and BRDF correction to standardize Landsat

data,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2010.

[18] A. H. e. a. Strahler, “MODIS BRDF/albedo product: algorithm theoretical basis document version,”

MODIS documentation.

[19] R. Richter, “Correction of satellite imagery over mountainous terrain,” Applied Optics, 1998.

[20] Zhang, “Calculation of radiative fluxes from the surface to top of atmosphere based on ISCCP and

other global data sets: Refinements of the radiative transfer model and the input data,” J.

Geophys. Res, 2004.

[21] “Sentine-2 L1C Cloud Masks,” [Online]. Available: https://sentinel.esa.int/web/sentinel/technical-

guides/sentinel-2-msi/level-1c/cloud-masks.

[22] Z. Zhu and C. E. Woodcock, “Object-based cloud and cloud shadow detection in Landsat imagery,”

Remote Sensing of Environment, 2012.

[23] “CEOS Working Group on Calibration & Validation,” [Online]. Available:

http://ceos.org/ourwork/workinggroups/wgcv/.

[24] “ESA's Sentinels Online,” April 2017. [Online]. Available:

https://sentinels.copernicus.eu/web/sentinel/missions/sentinel-2/news/-/article/pilot-

production-of-sentinel-2-l2a-products-over-europe.

[25] “Sentinel-2 status as relevant to CARD4L discussion,” ESA, March 2017. [Online]. Available:

http://ceos.org/meetings/lsi-vc-3/.

[26] “Amazon Web Services & Sentinel-2,” [Online]. Available: http://sentinel-pds.s3-website.eu-

central-1.amazonaws.com/.

[27] “Google Earth & Sentinel-2 & Landsat Availability,” [Online].

[28] “Sentinel-2 & Landsat-8 Data Available on Planet Platform,” [Online].

[29] “ESA's Heterogeneous Missions Accessibility (HMA) work yields EO Satellite Ground Segment

Interoperability Manual,” [Online]. Available: http://www.opengeospatial.org/blog/1662.

[30] Bunting, P., Clewley, D., Lucas, R. M., & Gillingham, S. (2014). The Remote Sensing and GIS

Software Library (RSGISLib). Computers and Geosciences, 62, 216–226.

[31] Chander, G., Markham, B. L., & Helder, D. L. (2009). Summary of current radiometric calibration

coefficients for Landsat MSS, TM, ETM+, and EO-1 ALI sensors. Remote Sensing of Environment,

113(5), 893–903.

[32] Dymond, J. R., & Shepherd, J. D. (2004). The spatial distribution of indigenous forest and its

composition in the Wellington region, New Zealand, from ETM+ satellite imagery. Remote Sensing

of Environment, 90(1), 116–125.

[33] Flood, N., Danaher, T., Gill, T., & Gillingham, S. (2013). An Operational Scheme for Deriving

Standardised Surface Reflectance from Landsat TM/ETM+ and SPOT HRG Imagery for Eastern

Australia. Remote Sensing, 5(1), 83–109. http://doi.org/10.3390/rs5010083


Status: Final

Version: v0.1

Date: 18/10/2017

Page 56 of 56

[34] Shepherd, J. D., & Dymond, J. R. (2003). Correcting satellite imagery for the variance of

reflectance and illumination with topography. International Journal of Remote Sensing, 24(17),

3503–3514.

[35] Tripathi, S. N., Dey, S., Chandel, A., & Srivastava, S. (2005). Comparison of MODIS and AERONET

derived aerosol optical depth over the Ganga Basin, India. Annales Geophysicae, 1093–1101.

[36] Vermote, E. F., Saleous, N. Z., & Privette, J. L. (2006). Surface reflectance Earth system data

record/climate data record White paper. Nasa.

[37] Zhu, Z., & Woodcock, C. E. (2012). Object-based cloud and cloud shadow detection in Landsat

imagery. Remote Sensing of Environment, 118, 83–94.

Date post:	20-Aug-2020
Category:	Documents
Upload:	others
View:	1 times
Download:	0 times

Developing standards and automated production for Sentinel...

Documents