© Crown copyright Met Office

User requirements gathering – experience from the ESA Climate Change Initiative project on SST Simon Good

© Crown copyright Met Office

The Climate Change Initiative

© Crown copyright Met Office

The ESA Climate Change Initiative (CCI)

“To realise the full potential of the long-term global Earth Observation archives that ESA, together with its member states, has established over the last thirty years, as a significant and timely contribution to the ECV databases required by United Nations Framework Convention on Climate Change (UNFCCC).”

www.esa-cci.org

© Crown copyright Met Office

The ESA Climate Change Initiative (CCI)

Each CCI project was required to produce a user requirements document.

© Crown copyright Met Office

User requirements gathering

© Crown copyright Met Office

Types of user requirements studies1. Focus on the requirements of a particular

group of users.• Can make sure that the products really meet their

specifications.• It is likely that products will still also be useful to

others.

2. Try to capture everyone’s requirements.• Comprehensive list of requirements that will be a

useful resource for the future.• Useful to be able to show that our products meet the

requirements of lots of people. • There may be conflicting requirements and it is not

possible to satisfy everyone.

© Crown copyright Met Office

User requirements gathering

• For the SST CCI project we did a broad study.• We were asked to gather requirements for a

wide variety of aspects of SST data.• Including getting feedback on current SST datasets

and finding out user’s future needs.

• And at different levels of requirement:• Threshold: the limit at which the observation

becomes ineffectual and is no use for the application. • Breakthrough: the level at which a significant

improvement in this application would be achieved. • Objective: the maximum performance limit for the

observation, beyond which no significant improvement in the application would be achieved.

© Crown copyright Met Office

Methods of user requirements gathering• Review of documents produced by other

projects/organisations.• E.g. GCOS user requirements.• Useful resource but don’t get to communicate directly

with the users.

• Discussion sessions.• Get a really good understanding of particular user’s

needs.• It is only possible to do this with a limited number of

people.

• Asking other projects for lessons learned.• Useful way to avoid pitfalls! • Replies may be confidential.

© Crown copyright Met Office

Methods of user requirements gathering• Questionnaire.

• Good way to reach many people.• This was the primary source of information for the

SST CCI project user requirements.

• Getting ‘trailblazers’ to try the data and send feedback.• Really good way to find out about good and bad

things about the data.• Only possible once data products are available.

© Crown copyright Met Office

Experience from the questionnaire

© Crown copyright Met Office

Getting people to fill out the online questionnaire• Invites were sent to over 800 email addresses.

• Mined from papers that cited SST datasets, and• A list of climate research areas was compiled and

contacts for each were found.

• The questionnaire was long! (30 minutes+ to complete.)

• There were 108 completed responses.• At least as many started and then gave up!

Keeping a questionnaire short and focussed should maximise the number of complete responses.

© Crown copyright Met Office

Demographic of responses

0

10

20

30

40

50

60

Africa Asia Australia Eurasia Europe NorthAmerica

SouthAmerica

© Crown copyright Met Office

Demographic of responses

0

5

10

15

20

25

Argen

tina

Austra

lia

Belgium

Botsw

anaBra

zil

Camer

oon

Canad

a

China

Croat

ia

Franc

e

Germ

any

GhanaIn

diaItaly

Japa

n

Kenya

Luxe

mbo

urg

Mex

ico

Nethe

rland

s

Nairob

i

Niger

Norway

Portu

gal

Rep. o

f Kor

ea

Roman

ia

Russia

Spain

South

Afri

ca

Sweden

Turke

yUK

USA

© Crown copyright Met Office

Demographic of responses

0

10

20

30

40

50

60

On the samegrid/swath that thesatellite instrumentrecorded (level 2

data)

Averaged onto a grid(level 3 data)

Analysed so thatdata gaps are filled

in (level 4 data)

Other

Clouds

Dataset production

High latitude modelling

Coastal oceanography

Ocean biology or chemistry

Atmospheric reanalysis

Ocean reanalysis

Climate variability

Monitoring of climate

Detection and attribution of climate change

Decadal forecasting

Seasonal forecasting

Regional modelling

Climate model evaluation

Climate model initialisation0 10 20 30 40 50 60 70

Climate model initialisationClimate model evaluation

Regional modellingSeasonal forecastingDecadal forecasting

Detection and attribution of climate changeMonitoring of climate

Climate variabilityOcean reanalysis

Atmospheric reanalysisOcean biology or chemistry

Coastal oceanographyHigh latitude modelling

Dataset productionCloudsSea iceAerosol

Other

Number of responses

RelatedPrimary

© Crown copyright Met Office

Example result: length of record required

• A mixed set of results per application area.

• Is this because people’s use ofthe data is very different even within application areas?

• Or is it because people are not used to being asked to quantify things in this way?

Threshold requirements

0

5

10

15

20

25

30

<1 year 1 year 10 years 20 years 30 years >30 years

© Crown copyright Met Office

Example result: length of record required

• Clear increase in requirements going from threshold to breakthrough.

Breakthrough requirements

0

5

10

15

20

25

30

35

40

45

50

<1 year 1 year 10 years 20 years 30 years >30 years

© Crown copyright Met Office

Example result: length of record required

• Highest quality satellite SST datadoes not extend>30 years!

• Sometimes it is not possible to meet a requirement – can’t make apples out of oranges.

• But it is useful to capture these requirements as it could help identify opportunities for future work.

Objective requirements

0

10

20

30

40

50

60

70

80

<1 year 1 year 10 years 20 years 30 years >30 years

© Crown copyright Met Office

Example result: grid spacing

• Some questions that are asked can directly influence the product specification.

Breakthrough requirements

0

2

4

6

8

10

12

14

16

18

20

<1 km 1 km 4 km 0.05° 0.1° 0.2° 0.25° 0.5° 1° >1°

© Crown copyright Met Office

Example result: artificial drift

• Some requirements canbe used to determine if theproducts meet user’s needs.

• However, in somecases it may bedifficult or impossible to know because of a lack of reference data!

Breakthrough requirements

0

5

10

15

20

25

30

<0.0

1°C/d

ecad

e

0.01

°C/d

ecad

e

0.02

°C/d

ecad

e

0.05

°C/d

ecad

e

0.1°

C/dec

ade

0.2°

C/dec

ade

>0.2

°C/d

ecad

e

Not a

pplic

able

© Crown copyright Met Office

Conclusions

© Crown copyright Met Office

Conclusions

• For the SST CCI project we used a variety of methods to gather user requirements.

• An online questionnaire became the main source of requirements.• Important to keep the questionnaire short and focussed.

• Answers to questions may:• Be confusing and require further investigation.• Directly influence product design.• Help to benchmark the performance of products.• Not be possible to achieve or not be possible to demonstrate that

they are achieved because of limitations of the data/reference data.

• Getting trailblazing users involved can be very helpful.

© Crown copyright Met Office

Questions and answers