Post on 18-Jan-2016
transcript
© Crown copyright Met Office
User requirements gathering – experience from the ESA Climate Change Initiative project on SST Simon Good
© Crown copyright Met Office
The Climate Change Initiative
© Crown copyright Met Office
The ESA Climate Change Initiative (CCI)
“To realise the full potential of the long-term global Earth Observation archives that ESA, together with its member states, has established over the last thirty years, as a significant and timely contribution to the ECV databases required by United Nations Framework Convention on Climate Change (UNFCCC).”
www.esa-cci.org
© Crown copyright Met Office
The ESA Climate Change Initiative (CCI)
Each CCI project was required to produce a user requirements document.
© Crown copyright Met Office
User requirements gathering
© Crown copyright Met Office
Types of user requirements studies1. Focus on the requirements of a particular
group of users.• Can make sure that the products really meet their
specifications.• It is likely that products will still also be useful to
others.
2. Try to capture everyone’s requirements.• Comprehensive list of requirements that will be a
useful resource for the future.• Useful to be able to show that our products meet the
requirements of lots of people. • There may be conflicting requirements and it is not
possible to satisfy everyone.
© Crown copyright Met Office
User requirements gathering
• For the SST CCI project we did a broad study.• We were asked to gather requirements for a
wide variety of aspects of SST data.• Including getting feedback on current SST datasets
and finding out user’s future needs.
• And at different levels of requirement:• Threshold: the limit at which the observation
becomes ineffectual and is no use for the application. • Breakthrough: the level at which a significant
improvement in this application would be achieved. • Objective: the maximum performance limit for the
observation, beyond which no significant improvement in the application would be achieved.
© Crown copyright Met Office
Methods of user requirements gathering• Review of documents produced by other
projects/organisations.• E.g. GCOS user requirements.• Useful resource but don’t get to communicate directly
with the users.
• Discussion sessions.• Get a really good understanding of particular user’s
needs.• It is only possible to do this with a limited number of
people.
• Asking other projects for lessons learned.• Useful way to avoid pitfalls! • Replies may be confidential.
© Crown copyright Met Office
Methods of user requirements gathering• Questionnaire.
• Good way to reach many people.• This was the primary source of information for the
SST CCI project user requirements.
• Getting ‘trailblazers’ to try the data and send feedback.• Really good way to find out about good and bad
things about the data.• Only possible once data products are available.
© Crown copyright Met Office
Experience from the questionnaire
© Crown copyright Met Office
Getting people to fill out the online questionnaire• Invites were sent to over 800 email addresses.
• Mined from papers that cited SST datasets, and• A list of climate research areas was compiled and
contacts for each were found.
• The questionnaire was long! (30 minutes+ to complete.)
• There were 108 completed responses.• At least as many started and then gave up!
Keeping a questionnaire short and focussed should maximise the number of complete responses.
© Crown copyright Met Office
Demographic of responses
0
10
20
30
40
50
60
Africa Asia Australia Eurasia Europe NorthAmerica
SouthAmerica
© Crown copyright Met Office
Demographic of responses
0
5
10
15
20
25
Argen
tina
Austra
lia
Belgium
Botsw
anaBra
zil
Camer
oon
Canad
a
China
Croat
ia
Franc
e
Germ
any
GhanaIn
diaItaly
Japa
n
Kenya
Luxe
mbo
urg
Mex
ico
Nethe
rland
s
Nairob
i
Niger
Norway
Portu
gal
Rep. o
f Kor
ea
Roman
ia
Russia
Spain
South
Afri
ca
Sweden
Turke
yUK
USA
© Crown copyright Met Office
Demographic of responses
0
10
20
30
40
50
60
On the samegrid/swath that thesatellite instrumentrecorded (level 2
data)
Averaged onto a grid(level 3 data)
Analysed so thatdata gaps are filled
in (level 4 data)
Other
Clouds
Dataset production
High latitude modelling
Coastal oceanography
Ocean biology or chemistry
Atmospheric reanalysis
Ocean reanalysis
Climate variability
Monitoring of climate
Detection and attribution of climate change
Decadal forecasting
Seasonal forecasting
Regional modelling
Climate model evaluation
Climate model initialisation0 10 20 30 40 50 60 70
Climate model initialisationClimate model evaluation
Regional modellingSeasonal forecastingDecadal forecasting
Detection and attribution of climate changeMonitoring of climate
Climate variabilityOcean reanalysis
Atmospheric reanalysisOcean biology or chemistry
Coastal oceanographyHigh latitude modelling
Dataset productionCloudsSea iceAerosol
Other
Number of responses
RelatedPrimary
© Crown copyright Met Office
Example result: length of record required
• A mixed set of results per application area.
• Is this because people’s use ofthe data is very different even within application areas?
• Or is it because people are not used to being asked to quantify things in this way?
Threshold requirements
0
5
10
15
20
25
30
<1 year 1 year 10 years 20 years 30 years >30 years
© Crown copyright Met Office
Example result: length of record required
• Clear increase in requirements going from threshold to breakthrough.
Breakthrough requirements
0
5
10
15
20
25
30
35
40
45
50
<1 year 1 year 10 years 20 years 30 years >30 years
© Crown copyright Met Office
Example result: length of record required
• Highest quality satellite SST datadoes not extend>30 years!
• Sometimes it is not possible to meet a requirement – can’t make apples out of oranges.
• But it is useful to capture these requirements as it could help identify opportunities for future work.
Objective requirements
0
10
20
30
40
50
60
70
80
<1 year 1 year 10 years 20 years 30 years >30 years
© Crown copyright Met Office
Example result: grid spacing
• Some questions that are asked can directly influence the product specification.
Breakthrough requirements
0
2
4
6
8
10
12
14
16
18
20
<1 km 1 km 4 km 0.05° 0.1° 0.2° 0.25° 0.5° 1° >1°
© Crown copyright Met Office
Example result: artificial drift
• Some requirements canbe used to determine if theproducts meet user’s needs.
• However, in somecases it may bedifficult or impossible to know because of a lack of reference data!
Breakthrough requirements
0
5
10
15
20
25
30
<0.0
1°C/d
ecad
e
0.01
°C/d
ecad
e
0.02
°C/d
ecad
e
0.05
°C/d
ecad
e
0.1°
C/dec
ade
0.2°
C/dec
ade
>0.2
°C/d
ecad
e
Not a
pplic
able
© Crown copyright Met Office
Conclusions
© Crown copyright Met Office
Conclusions
• For the SST CCI project we used a variety of methods to gather user requirements.
• An online questionnaire became the main source of requirements.• Important to keep the questionnaire short and focussed.
• Answers to questions may:• Be confusing and require further investigation.• Directly influence product design.• Help to benchmark the performance of products.• Not be possible to achieve or not be possible to demonstrate that
they are achieved because of limitations of the data/reference data.
• Getting trailblazing users involved can be very helpful.
© Crown copyright Met Office
Questions and answers