'Making Sense of Uncertainty' Guide

8/13/2019 'Making Sense of Uncertainty' Guide

1/28

SENSE ABOUT SCIENCE

MAKING SENSE OF UNCERTAINTYWhy uncertainty is part of science


2/28

CONTRIBUTORS

The contributors met over 2012 to review what is being said

about scientific uncertainty in the media, policy and publiclife; to identify the misconceptions in these discussions andshare insights that help to dispel these; and to draft andedit the guide.

With thanks to everyone who reviewed all or part of theguide, or gave contributions on specific questions,including: Dr Suzanne Aigrain, Dr Nina Alphey, Dr RichardBetts, Professor Sheila Bird, Professor Andy Challinor, DrNathan Green, Professor Joanna Haigh, Professor MikeHulme, Christian Hunt, Richard Johnston, Dr ShellyLachish, Professor John Mitchell, Professor Peter

Sammonds, Dr Emily Shuckburgh, Dr Juliet Stevens, DrMatthew Sullivan, Dr Tom Wells, Dr Ben Wheeler; and forvaluable input from the Royal Society meeting Handlinguncertainty in weather and climate prediction convened byProfessor Tim Palmer in October 2012.

We are grateful to the BBSRC, Clark Bradbury CharitableTrust, John Innes Centre, NERC, University of Reading andthe Walker Institute for supporting the production anddistribution of this guide.


3/28

PROF. DAVIDSPIEGELHALTER FRS

Winton Professor of thePublic Understandingof Risk, Universityof Cambridge

TRACEY BROWNManaging Director,Sense About Science

PROF. ANGELA

MCLEAN FRSProfessor ofMathematical Biology,University of Oxford

DR MARK NAYLORScottish Government

and Royal Societyof EdinburghPersonal Fellow

PROF. LEONARDSMITHDirector, Centre forthe Analysis of TimeSeries, LSE

DR TABITHAINNOCENTScientific Liaison,Sense About Science

PETER GIBBS

BBC weather presenter,Met Office

MICHAEL HANLON

Journalist

DR KEN MYLNEHead of WeatherScience NumericalModelling, Met Office

HEATHER MAYFIELDDeputy Director,Science Museum

PROF. AVERIL

MACDONALDJJ Thomson PhysicalLaboratory, Universityof Reading

PROF. SIR MICHAELRAWLINSChair, NationalInstitute ofClinical Excellence

PROF. PAUL

HARDAKERChief Executive, Instituteof Physics; Chairman,Sense About Science

DR DAVIDSTAINFORTH

Senior Research Fellow,Grantham Research Instituteon Climate Change and theEnvironment, LSE

DR CHRIS TYLERDirector, Parliamentary

Office of Science andTechnology (POST)

KATHY MASKELLWalker Institutefor ClimateSystem Research

DR KATHERINE ROYSEScience Director

Environmental Modelling,British Geological Survey;NERC KnowledgeExchange Fellow

DR ED HAWKINS

Climate scientist inNCAS-Climate,Department ofMeteorology, Universityof Reading

PROF. TIM PALMER FRSRoyal Society Research

Professor, University ofOxford; Co-DirectorOxford Martin Programmeon Modelling andPredicting Climate

DR ELIZABETH

MORRIS OBEScott Polar ResearchInstitute, Cambridge

PROF. IANSTEWART FRS

Emeritus Professor ofMathematics, Universityof Warwick

DR EMILY SOLecturer, Department of

Architecture, Universityof Cambridge

DR FRANK DE VOCHTLecturer in Occupationaland EnvironmentalHealth, Universityof Manchester


4/28

INTRODUCTIONwhy make sense of uncertainty?

Scientific uncertainty is prominent in researchthat has big implications for our society: couldthe Arctic be ice-free in summer by 2080?Will a new cancer drug be worth its sideeffects? Is this strain of flu going to be adangerous epidemic?

Uncertainty is normal currency in scientificresearch. Research goes on because we dontknow everything. Researchers then have toestimate how much of the picture is knownand how confident we can all be that theirfindings tell us whats happening or whatsgoing to happen. This is uncertainty.

But in public discussion scientific uncertainty ispresented as a deficiency of research. Wewant (even expect) certainty safety, effective

public policies, useful public expenditure.Uncertainty is seen as worrying, and even areason to be cynical about scientific research particularly on subjects such as climatescience, the threat of disease or the predictionof natural disasters. In some discussions,uncertainty is taken by commentators to meanthat anything could be true, including thingsthat are highly unlikely or discredited, or thatnothing is known.

This conflict frustrates us at Sense AboutScience, and we know that it frustratesresearchers we work with and the public wehear from. Some clearer ideas about whatresearchers mean by scientific uncertainty and where uncertainty can be measured and

where it cant would help everyone with howto respond to the uncertainty in evidence.

This guide has brought together specialists inmany areas climate science, clinical research,natural hazard prediction, public health,biostatistics and epidemiology. We asked themfor the reasons why they are not automaticallyso troubled by the presence of uncertainty inthe most heated debates.

We have looked at what uncertainty means

and doesnt mean in science, how it ismeasured, when it cant be measured and howthat might change through research into thebig questions. Above all we asked how otherpeople can grapple constructively withadvances in knowledge and changes inthinking, instead of despairing at thoseuncertain scientists.

TRACEY BROWNTABITHA INNOCENT

SENSE ABOUT SCIENCE


5/28

THOSE UNCERTAINSCIENTISTS01

Uncertainty is normal inscientific research but to policymakers, journalists and widersociety it sounds likeunreliable. Despite life tellingus otherwise, the assumption inmany debates is that we shouldexpect certainty.

SO WHAT IS UNCERTAINTYFOR SCIENTISTS?02

There are different types of

uncertainty that are an ordinarypart of scientific research. Thesecan be addressed or taken intoaccount in several ways.

PREDICTIONSAND MODELS

03As numerical models can beused flexibly and updated asknowledge changes, they areroutinely used in research thatdeals with high levels ofuncertainty. These sometimes geta bad press, which misreads howand why they are used.

PLAYINGON UNCERTAINTY05

Uncertainty does not mean we

know nothing, that evidencecannot be trusted, that anythingcould turn out to be correct orthat decisions cant be made.

DO WE EVEN NEEDMORE CERTAINTY?04

We need to know when adecision is, and is not,affected by whether we knowsomething completely. Thisidea is beginning to shapethe way that scientists andpolicy makers use andcommunicate uncertainty.

DELVINGDEEPER

06Further resources on scientificuncertainty, including blogs,books and guides.

CONTENTS


6/28

THOSEUNCERTAIN

SCIENTISTS

01

In the areas of research that are most often inthe public eye, uncertainty has become a bigpoint of misunderstanding (even conflict)between scientists and commentators. Aresearcher presents his or her findings, theradio interviewer (or the politician, journalistor official) asks: can you be certain?. Theresearcher has to answer truthfully 'no' andthen defend their findings, for fear they will beinterpreted as meaningless. In fact, they haveprovided important limits to the uncertainty.

Researchers use uncertainty to express howconfident they are about results, to indicatewhat scientists dont yet know, or tocharacterise information that is by naturenever black and white. But saying thatsomething is uncertain in everyday languagehas a negative connotation. When aresearcher says the predictions we made onthe basis of our research have a margin ofuncertainty, they mean they are very

confident that the outcome will fall within thepredicted range. But a commentator is likely tounderstand from this the piece of researchis unreliable.

This is the type of disconnection we see inmedia reports of global warming, publichealth risks and earthquake prediction.

like with many neural

disorders and conditions

..there is always uncertainty

in their answers

when distinguished

scientists say it

isn't actually proven

you can't absolutely prove,

can you, that co2is responsible

for global warming?

it becomes ever more

obvious that none of

them really has a clue

05


7/28

BUT WHY IS UNCERTAINTY SEEN TOUNDERMINE RESEARCH?

Put crudely, scientists tend to think science isabout things we dont know fully. Journalistsand politicians (and to a large extent, manypeople) think it is about things we do know,and theyre impatient with maybes.

The problem here is that not all scientificknowledge is the same, so treating it asthough it were is misleading.

In the first place, there is rarely such thing as100% certainty and everything less than thisis uncertain. Scientific inquiry makes some(potentially imperfect) observations, thenmakes predictions to test how widely the

observed pattern holds true from looking athow a molecule will behave in combating adisease to understanding the metabolisms ofdinosaurs. Most people know this to someextent. They know that there are varying risksin the things they do every day, and thatperfect knowledge is rarely possible (mostknowledge is not like a mathematical proof).But reports and commentaries like thoseheading this section show that completecertainty is still seen in society as the test ofnew knowledge.

Secondly, new research needs to be looked atdifferently from settled science. Settled scienceis concerned with fundamental scientificprinciples, well-established and supported bylarge bodies of evidence (generally, the sciencewe all learned in school). Everyone can beconfident about a great deal of knowledge:that the earth goes round the sun, the germtheory of infectious disease, Pythagorasstheorem. Some science is widely accepted asexplaining the world around us: it underpinscommonplace inventions such as TV or

pasteurisation, and can get us to the moon,predict an eclipse and explain why our feetstay on the ground. And while exceptions arefound that show settled science can be revised(game-changers, see section 5), researchdiscoveries rarely change our overallunderstanding of the underlyingscientific principles.

For the most part, settled science is not whatscientists spend time thinking about,questioning or researching. Researchers

aspire to add something new to what weknow. Uncertainty is what they are interestedin. Scientists design experiments ordata-gather to get new evidence which testsmodels and theories in as minimally biased away as possible but there are limits on whatcan be done, and simplifying assumptions thathave to be made. Research science is movedforwards by knowledge that is modified withnew evidence and most areas of researchinclude some knowledge that is settled, andsome that is more uncertain.

Across society, we dont talk much about thesettled things we are confident of the fact

that an antibiotic works, say. We talk about theinteresting, newer issues such as assessingwhen antibiotic resistance might occur.

The essential point is that new scientificknowledge usually includes greateruncertainty and researchers often dont knowhow much of the picture it shows. It doesntmean scientists know nothing, and we shouldnot be exasperated that it is less settled thanthe explanation for why the sky is blue.

THOSE UNCERTAIN SCIENTISTS

MICHAELRAWLINS

The term uncertainty is unquestionablyfraught with misinterpretation especially for non-scientists. I'd prefer thephrase how confident am I?, thereciprocal of uncertainty.

ELIZABETHMORRIS

The word certain is like perfect in thesense that it describes an end state, so itis understood that there cannot bedegrees of certainty or perfection, onlydegrees of uncertainty or imperfection.We can say how far we are from the endstate, almost certain for instance.

06


8/28

WHAT DOESUNCERTAINTY

MEAN FORSCIENTISTS?

02

Scientific researchers have to work outwhether uncertainty can be calculated andhow to do so, and then whether it matters andwhat can be done about it.

THERE ARE SEVERAL DIFFERENT

CONCEPTS OF UNCERTAINTY

Uncertainty is a hard thing to discuss becauseit is inherently intangible. The difficultyexpressing it is not exclusive to science: itbecame familiar after a rather baffling speechby US politician Donald Rumsfeld in 2002:

Reports that say that something hasn'thappened are always interesting to me,because as we know, there are known knowns;there are things we know we know. We also

know there are known unknowns; that is to saywe know there are some things we do notknow. But there are also unknown unknowns the ones we don't know we don't know.

Rumsfeld had a point. Some scientificuncertainty can be known: in a sense,signposted. There is uncertainty about howmuch global temperatures will change ifcarbon dioxide emissions continue at currentlevels for the next 50 years. But researchers

...astronomers are certain

the asteroid will miss

the emerging consensus was

that the benefits of hrt

outweigh the risks...

satellite data

solves antarctic seaice mystery

07


9/28

WHAT DOES UNCERTAINTY MEAN FOR SCIENTISTS?

know what information they would need to work this out it is a known unknown. Knowing justwhat it is that we dont yet know is very useful because it guides further research to find an answer.

But we also face unknown unknowns questions not yet thought of, and information we do notrealise is missing at all.

These concepts of uncertainty are like the difference between a Victorian map with central Africamarked unexplored, which indicates known unknown data, and medieval maps made before

Europeans knew that the New World existed, which show sea where the land should be.

DAVIDSTAINFORTH

If I drop a tennis ball, what will happen?There is some uncertainty: I dont know

details about this particular ball. I haventdone experiments to see whether the ballfalls every time. But I still know, almostcertainly, that the ball will fall. I know thatthis would be true if I was standing inAustralia, and that the way it falls would bedifferent if I was standing on the moon.There isnt 100% certainty because thesituation might be different from what weexpect if I were working with a magician Imight be suspicious about the possibility oftricks; maybe magnets in the ball and the

ceiling. This could be the case. But withoutevidence for it I would be foolish to betagainst the ball falling. Indeed Id bet a lotthat the ball would fall. Now think of a

different question. When I drop the tennisball, where will it stop? This is difficult to

tell. It might bounce off a chair, a table orsomeones foot. I can calculate some limitsto give me a rough idea of where the ballmight end up but there will remainsubstantial uncertainty.

When were thinking about how the climatemight change, the system is morecomplicated but the same thinking applies.There are many details about which we areextremely uncertain but there is very littleuncertainty over the big picture of serious

warming and significant disruption forhuman societies. The ball will fall we justdont know where it will stop.

UNCERTAINTIES WE CAN WORK OUT AND UNCERTAINTIES WE CANT

08


10/28


The best approach to uncertainty ispragmatism. While it is important to be alert tothe possibility of unknown unknowns ofdiscovering that ideas about how the worldworks are stronger or weaker than we realised that possibility alone doesnt point us towardsbetter explanations.

On the other hand, if researchersare describing the uncertainty inhow they understand aparticular problem, it actually

means quite a lot is known inthis case about the strengthsand weaknesses of thescience. They may knowenough about the uncertaintyto be able to quantify it, e.g.there is a 20% chance of rain.So if we hear scientists talkingabout the uncertainty in theirfindings in some detail we should bemore reassured rather thanmore worried!

There are some well-established principles instatistics for calculating and expressinguncertainty (these have been explained indetail elsewhere, see Resources). They askwhether an observation or result is in the rightball-park, or how likely it is that an observationmade in a piece of research is not simply arandom event.

Confidence intervals tell us how reliableresearchers think their observations are.

Researchers gather data; if, for example, theywant to find the average height of people inLondon, they measure the heights of a sampleof Londoners and calculate the mean average.There is some chance that the mean average(the value) the researchers have calculated is

not correct. The real average height ofLondoners could be a bit higher or a

bit lower.

The researchers need toexpress that uncertainty

about the value as preciselyas possible, so that theirresult is useful to otherresearchers and everyoneunderstands how likely it isto be right. Using the range

of observed individual heightsthey can calculate the range

(the confidence limits) that theywould expect the mean average to

fall within for at least 95 out of 100repeats of the measurement exercise. This is a

95% confidence interval. If they wanted toknow the limits of that range for a 99%confidence interval, i.e. where the meanaverage is likely to fall in 99 out of 100 repeatsof the experiment, they would expect to give abigger range.

This trade off between precision andcertainty is a fairly intuitive point. If youthrow a ball at a target on the floor andare asked to draw a circle within whichyou are confident it will land, you woulddraw a bigger circle around the target ifit had to land in the circle 99 times outof 100 than if it had to land in the circle90 times out of 100.)

HOW DO SCIENTISTS WORK OUT UNCERTAINTY?

100%

09


11/28

When researchers are looking at cause andeffect relationships, they usually calculateuncertainty by asking how likely it is that theirobservations represent a real effect, ratherthan simply happening by chance. Thisquestion is incorporated into the design of thestudy. Researchers start with a description ofwhat they would expect to see if there was noeffect (a null hypothesis); they then look athow close the match is between this patternand their observations. Researchers can put anumber on the extent to which the results andthe hypothesis match (a p-value). This isessentially asking, Is our result differentenough from a pattern of no effect that therereally looks to be something going on?.

Where uncertainty is known, it can beincorporated into the way that effects are

predicted. For example, there is someuncertainty in the initial conditions that formthe basis of a hurricane forecast. In the past,forecasters showed one predicted path of ahurricane, with caveats about the uncertaintyof the initial conditions. Now they tend to usethe uncertainty to vary the initial conditionsthey put into the forecast and show theresulting range of what the hurricane will do an ensemble forecast so we can see howmany of the forecasts come up with asimilar path.

PRECISE NUMBERS SHOULDONLY BE USED WHERE THEY

ARE JUSTIFIED

DAVIDSPIEGELHALTER

In clinical medicine, doctors cannotpredict exactly what will happen toanyone, and so may use a phrase suchas of 100 people like you, 96 willsurvive the operation. Sometimes thereis such limited evidence, say because apatients condition is completely novel,that no number can be attached withany confidence.

MICHAELHANLON

When the uncertainty makes the rangeof possibilities very broad, we shouldavoid trying to come up with a single,precise number because it creates afalse impression of certainty spurious precision.


10


12/28

USE SEPARATE MEASURES FORHOW GOOD THE EVIDENCE IS,

AND HOW CONFIDENT THE

CONCLUSION

Organisations have come up with newways of expressing their confidence inthe data. In medicine, the GRADE scale(a grading system for evidence andrecommendations) is widely used, whichtakes into account the quality of theunderlying evidence. And, in climatescience, the most recentIntergovernmental Panel on ClimateChange (IPCC) assessment used one

verbal scale to express their confidencein the scientific understanding, andanother to give the likelihood ofsomething occurring (where virtuallycertain corresponds to a level ofmore than a 99% likelihood ofhappening, for example).


WHAT CAN BE DONEABOUT UNCERTAINTY?

Sometimes it is a question of making moreobservations or designing experiments orcomputational models to improve theavailable information. This is not always asstraightforward as it might seem, and it cantake a while to identify the information thatis missing:

The early climate model simulations only

included the effect of greenhouse gases. Whenrun over the last 100 years or so for which wehave instrumental measurements of surfacetemperature, these simulations were warmcompared to observations. In time, the effect ofother factors (industrial aerosols produced byburning fossil fuels, the effect of volcaniceruptions, possible variations in solarirradiance) were added. The agreement ofscientists models with our observationsimproved, particularly due to the inclusion ofthe effect of industrial aerosols.JOHN MITCHELL

ELIZABETH

MORRIS

SEPARATE LONG TERM EFFECTSFROM SHORT TERM EFFECTS

"Glaciologists use satellite data to assesschanges in ice-sheet elevation andestimate their impact on sea level. Weneed to know whether these arelong-term changes (for exampleproduced by climate change) or justshort-term fluctuations (for exampleproduced by a particularly heavysnowfall, or if a particularly warmsummer makes the surface snowdenser). We can understand theuncertainty in the satellite observationsof long-term change better by definingthe size of the effect that short-termfluctuations could have on the satellite

observations. If we observe changeswhich are significantly bigger than this,and we can characterise the uncertaintywith the satellite observationsthemselves, then we can more clearlyattribute these changes to long-termprocesses such as climate change."

11


13/28


For example, Newtons law of gravity is stillsufficient to get us to the moon and back inspite of not being a complete model of theway gravity works. Or if, say, all values withina confidence interval point to the same clinicaldecision, further narrowing the uncertaintymay be a waste of public funds. Researchers

can incorporate uncertainty into calculationsor work round it.

Sometimes it is only through technologicaldevelopments that uncertainty can be reduced. Thishas happened in the search for exoplanets(planets that go round stars outside our solarsystem) a step towards detecting whetherthere is life on other planets. These planetswere only detected very recently wheninstruments and techniques becamesufficiently accurate, but progress hasbeen rapid. The first exoplanets werediscovered in the 1990s, and

thousands have now been identified.Their discovery is progressivelyreducing the degree of uncertainty wehave about things like the existence oflife on other planets, by improving ourestimates of the number of planetsoutside the solar system, and of thefraction of these that may be able tosupport life.

But reducing uncertainty is often not possibleor necessary and this is not a barrier to using

scientific knowledge.

The question is notdo we know everything?

it is do we know enough?

or how can we best make

a decision using what wedo know?.

TECHNOLOGICAL LIMITATIONS

12


14/28

Quantify routine uncertainty andincorporate it into calculationsUncertainty is considered routine in a lot ofcases it cant be eliminated but its notnecessarily problematic (and a decision can bemade beyond reasonable doubt). Forexample, there is a degree of imprecision inassessing the fetal age from ultrasound scansduring pregnancy (which are used to estimatethe due date). This arises from the naturalvariation of size-for-age and from differencesin the skill, experience and equipment of the

ultrasonographer. However, we can quantifythis uncertainty and use it to explain to womenthat the calculated due date should be used asa guide, rather than an exact prediction of theactual due date.

Calculate risk, a more usablemeasure of uncertaintyWhen uncertainty relates to a real-lifesituation with important and perhapsdetrimental consequences we can work outhow the uncertainty affects the risk, which is

more tangible.

There is some confusion between scientificand everyday uses of the words uncertaintyand risk. In everyday language, we might saythat something that is uncertain is risky. But inscientific terms, risk broadly means uncertaintythat can be quantified in relation to aparticular hazard and so for a given hazard,the risk is the chance of its happening. Forexample, research shows that oestrogentherapy (a form of HRT, to relieve the symptoms

of menopause) appears to increase the risk ofwomen suffering from heart disease (anassociated hazard). Current evidence suggeststhe risk is small, but there is a lot of uncertaintyaround the estimate and so the real effectcould be larger or smaller than estimated.

Risk is also a way of thinking about whichuncertainties to worry about. For example,there is a greater risk of agricultural chemicalsrunning off into waterways and damagingaquatic life when land is bare. We shouldtherefore be more interested in reducinguncertainty about the hazard to aquatic life ofthose chemicals used on bare land. Risk isespecially important when we wantinformation that is useful for decision-making.(But risk itself is a whole other story, told insome of the resources at the end.)

Mitigate the effects ofuncertain eventsAnother response to uncertainty is mitigation.There is currently irresolvable uncertaintyabout earthquakes: researchers know wherethey are going to occur, but cannot predictwhen they will happen.

In a scientific sense, earthquakes areunpredictable. But that does not mean that youcant predict things about them.

PETER SAMMONDS

Using historical data and geology, researcherscan calculate how frequently a particular scaleof earthquake is likely to occur in a specificlocation. A one-in-ten year calculation doesntsay anything about which ten years in 100 willsee an earthquake. But it means researcherscan calculate what kind of reinforcement tobuildings would be needed to make sure theycould withstand this frequency of quakes. If aserious earthquake is a one-in-200 yearevent, a community might only put resourcesinto reinforcing a building that is designed tolast ten years if its function was sufficientlyimportant a school or hospital, for example.


13


15/28

PREDICTIONSAND MODELS

03Modelling is an area that regularly causesmisunderstanding about uncertainty;recriminations quickly follow events that dontmatch predictions.

Many of us look at models as a black boxwhere data go in and answers come out.Some scientists find that image annoyingbecause of the scientific insight that goes intohow models are constructed. Scientific modelsdiffer greatly across fields of research, andspecialists in one field might have no idea

about what is involved in modelling inanother. Suggesting all models are the same islike saying that all experiments are the sameor all tools are the same.

SCIENTIFIC MODELS

The scientific models we are concerned withhere are numerical representations of aprocess or system to simulate some aspect ofthe world: how will a disease epidemicprogress? Will a new bridge hold sufficient

weight? How have genetic traits spreadamong human populations? These modelsusually involve joining up different ideas orinformation about how different parts of aprocess work.

new computer model predicts

when avian influenza strain

becomes infectious

economy will shrink this year,

treasury forecasts show

which weather forecast

should you believe?

forecasts for global

warming too high

14


16/28

These models are a necessary simplification ofthe real world and use experimental data orobservations, or physical understanding of

nature, or some combination of these.Sometimes the components are empirical,sometimes statistical, and sometimes directapplications of physical laws. Differentproblems require different modellingapproaches and, because not all models arethe same, the uncertainty around models canbe very different.

WHAT MODELS CAN DO

The vast majority of models that researchers

make have a fairly routine use in research andin everyday clinical applications, and are oftena simple descriptor of a process.

Models are usually used for two reasons, to:

1 ...understand patterns or theinterplay of different factorsIf researchers already know an outcome theycan use a model to work out the possibleways this could have come about or whichfactors have an important influence. Modelscan fill in the gaps between observations forexample, in the process of reconstructingevolutionary trees.

2 ...make predictionsResearchers start with observations about howthings are now, and use models to predict thefuture evolution or to create different futurescenarios that could result from this startingpoint. This gives an idea of the range of thingsthat could happen.

Modelling can be one of the most useful toolsfor reducing uncertainty in a crisis.Researchers turn to models when there isalready a lot of uncertainty about a question,and they need more information to supportdecisions on what to do next. Modelling cangenerate a shopping list of thingsresearchers would like to know more about.

In epidemiology, one difficulty is that peoplesbehaviour directly affects the course of anepidemic (unlike, say, the weather, which willbe the same regardless of how people respondto it). How people mix how many people, andfrom which age groups tells us a lot inpredicting how a virus will spread. Pastestimates of how much mixing occurs were veryuncertain and epidemiologists knew this wasan important unknown in their models. Then a

recent study recorded and quantified how oftendifferent age groups interact with each otherover an individuals lifetime, for the first time.We can now use this to inform models ofdisease epidemics. By identifying the importantuncertainties, we can try to reduce them.ANGELA MCLEAN

Do models reinforce assumptions?Because models simplify the world they haveto make assumptions, some of which may notbe correct. Are models verified independently

or do they reinforce the same assumptions?That depends. Some models can be verified for example the accuracy of weather forecastscan be tested against what actually happenswith the weather. But some models cannot beverified in this way. If two models are basedon the same assumptions they are likely toproduce more similar results so an answerfrom one is not entirely independent of theother. Where models based on differentassumptions give a similar answer, these aremore independent and can therefore provide

an indirect way of verifying the original model.(This question about independence is currentlythe subject of a lot of debateamong researchers.)

PREDICTIONS AND MODELS

15


17/28

PREDICTIONS AND MODELS

MODELS ARE NOT

intrinsically uncertainThere are many models in routine, everydayuse and often these are already settled science

and have low uncertainty.

Early signs of a measles outbreak weredetected in New Zealand in 2011, alongside avery low level of immunity among youngpeople. Because the government was slow torespond, an outbreak started. Fortunately, aswe know a lot about this virus, they alreadyhad models of how a measles outbreakescalates and had plans in place to control itsspread. They were well able to stop theoutbreak escalating. Because the uncertainty in

the models was relatively low, they generatedconfident predictions that informed a usefulintervention.ANGELA MCLEAN

staticModels change as the information researchershave changes.

We used to think that everyone who caught flubecame ill, but the 2009 H1N1 (swine flu)pandemic taught us that this clearly isnt true.The biggest uncertainty throughout the

pandemic was the proportion of people whowould become ill after getting the virus. Modelsof the worst-case scenarios predicted this couldbe high, which was picked up in alarmingheadlines saying that lots of people would diefrom swine flu. In the end, roughly one third ofthe UK population were infected, which was asexpected. But it was something of a surprisethat only a tiny fraction became ill; it turns outthat for H1N1 pandemic flu it is about 1 in 20.This makes the distinction between infectionand disease very clear: lots of people were

infected, far fewer were ill.

There had to be a real-life pandemic event forus to find this out because it is impossible todiscriminate clearly between strains of seasonalflu virus. Now that we know this, models canchange to make more accurate predictions.This will improve our ability to managepandemic threats. It will also inform the type ofstatements scientists and others make about thepublic health risk.ANGELA MCLEAN

an answer waiting tobe discoveredThere is often more than one way to modelany problem. Most processes and scenariosare not straightforward to model, and expertsdo not all agree on the best approach. Thisdisagreement is sometimes taken to mean thateverything is contested and unreliable. In fact,using diverse approaches is a good thing inany area of scientific research, and isencouraged. Researchers can then seewhether different models produce a similaranswer independently of each other; if theydo, this helps to quantify the uncertainty.

always very interesting (tomost people)Small changes to models never make thenews. They rarely capture much attentionamong scientists either. These tweaks areoften not even well documented but shouldbe, by the modellers at least because mostof them are not viewed to be scientificallyimportant enough. But this means that theeffects on data over time can be hard to track similar to the problem of negative trialresults going unreported and introduceunknown bias. Some modellers havesuggested there should be a Journalof Tweaking!

able to answer everythingThe capacity of models to answer increasinglysophisticated questions is sometimes limitedby technology. For example, current limits withcomputing power put some constraints on theprogress climate modellers and others canmake there are not yet powerful enough

computers to consider all possible futureclimate scenarios to the same level of detail asit is possible to forecast the weather. Thismeans that researchers have to make choicesthat restrict models to answering specific partsof a problem. In the case of climate modellingthis might mean researchers can use a modelto look at climate for a large geographic area,or for a much smaller area in greater detail,but not both at the same time.

16


18/28

DO WE EVENNEED MORE

CERTAINTY?

04

The UK government used its stockpile ofantiviral medication in 2009 when a swine flupandemic was predicted, at great expense; didthey over-react to the uncertainty inherent inthe swine flu model?Striving for certainty in scientific research,even research that affects public policy, can bea waste of effort and resources. What we need

instead is to talk about just how muchinformation is enough to make a sounddecision, because if we ask whether we reallyneed more certainty, sometimes the answer isa clear no.

DO WE EVEN NEED TO KNOW?

In policy-oriented research, there is atendency to focus on trying to establish morecomplete predictions, for example of futuretemperature variations, to feed into decisions.This leads to too much emphasis on improvingknowledge of all the uncertainty ranges.Instead, we should identify which details aremost relevant to a decision and extract therelevant information from models aboutprocesses and trade-offs, working on theuncertainties that really matter for theparticular problem we are tryingto address.ANDY CHALLINOR

more bse cases

but minister insists

beef is safe

scientists blamed

for quake deaths

declaring a swine flu pandemic

was a 'monumental error'

we were told no risk,

claim flood families

17


19/28

DO WE EVEN NEED MORE CERTAINTY?

OPERATIONAL KNOWLEDGE

Decision-makers generally look for a higher

level of certainty for an operational decision(such as introducing body scanners in airports)than for a decision based on broader ideologyor politics (such as reducing crime rates).

When are we certain enough? is never goingto be easy to answer. It depends on the desireto act, robust public discussion, leadership,willingness to address criticism and the risksinvolved in getting things wrong. Decisionsare usually made by policy-makers andofficials, not researchers science doesnt tell

us what to do and what they need isoperational knowledge.

Engineers work with operational knowledgeall the time this comes down to knowingenough to be confident about carrying out aparticular task. Take aeroplanes: everycomponent and every aeroplane is regularlytested. As a result they are very safe machinesthat operate within well-known limits. So eventhough it is impossible for an engineer topredict every variable that an aeroplane will

be subjected to, they can still be flown withconfidence. And engineers can manage anyevents that introduce greater uncertainty. Forinstance, when the volcanic ash cloudaccumulated over Northern Europe in 2010,planes were initially grounded as thisincreased the uncertainty about when it wassafe to fly. Researchers subsequentlydeveloped remote ash sensors for aeroplanesand ran experiments to understand the extentto which engines could tolerate ash, to sayconfidently when it was safe to fly.

All the time, multi-million pound decisions aremade using weather forecasts that containuncertainty: in energy trading to predict futureenergy demands, or by local councils decidingwhen to grit roads, for example. If you knowthe potential cost or damage, and theparameters that matter, you can work out whenit pays to take action. LEONARD SMITH

SCIENTIFIC UNCERTAINTY &PUBLIC POLICY

Decisions are not made on the basis ofscientific evidence alone, though it may playan important role.

Scientific uncertainty is only one of a wholeraft of factors that influence decision making:the impact on constituents, party votes,ideological perspective, economic implicationsand so on. The impression is sometimes giventhat policy makers and politicians areuncomfortable with handling uncertainty, butevery day they have to make decisions on thebasis of largely uncertain information.CHRIS TYLER

Sometimes researchers dont have all theevidence and need to be clear when that isthe case. In areas of scientific uncertainty thisis particularly important given the scope for

the uncertainty itself to become politicised. Wehave seen this when uncertainties in thedetails of predictions about climate changeand disease epidemics have been amplified tocast doubt on any and all research.

The reality is that scientists are not, forinstance, debating the evidence for man-madeglobal warming; they are debating uncertaintyabout the extent and timing of changes, andthe most accurate methods for predictingthese. We should recognise this misuse of

uncertainty for what it is, essentially smokeand mirrors that distract from thedecision-making process.

18


20/28


WE CANT ALWAYS IGNOREUNCERTAINTY SO WHAT ARETHE ALTERNATIVES?

Few people would argue that clearcommunication about uncertainty isunimportant, but there are different ideas onthe most effective approach. Although there isstill a long way to go, there are insightsemerging from some areas of research public health, climate science, economics where understanding and communicating

uncertainty is especially difficult.

As discussed in section 2, it is misleading toquantify uncertainty that cannot be quantified in these cases there is an even greater needto talk equally clearly about what researchersdo not know as what they do. Unknownunknowns cannot be identified, much lessquantified, and the best approach is torecognise this.

The Bank of England produces forecasts with

quantified uncertainties and contingentpredictions these completely leave aside someof the uncertainties deemed to beunquantifiable. They might say, for instance,this is what will happen unless the Eurocollapses, acknowledging the deeperuncertainty of whether or not the Euro willcollapse, which they cannot know withoutputting a spurious number on this. This has twodirect effects. It makes clear the scale they areprepared to work on, and what they simply donot know and cannot say. And this encouragesothers using these forecasts financialinstitutions, say to adopt an approach forresponding that focuses on resilience. There aredangers in thinking you can quantify somethingwhich cannot be quantified.DAVID SPIEGELHALTER

CURRENT PLAN

FORECAST

19


21/28


RESEARCHERS CAN EXPLAIN THEEVENTS THAT WOULD CHANGETHEIR MIND

Sometimes changes in uncertainty can bepredicted. In other words there may beuncertainty around future events butresearchers can work out clear signs of achange in the level of uncertainty to look outfor. This information helps people faced withsimilar, real-world scenarios to respond andmake decisions.

One example, from the UK BSE outbreak inthe early 1990s, would be when the first catbecame ill with mad cow disease. This is whenlots of infectious disease experts stopped eatingbeef. However, the fact that a cat getting sick

with a prion disease (TSE) for the first time wasa big danger sign, and represented an increasein the certainty of health risk to humans, wasnot spread as widely as it might have been. Ithink being clear about what future eventsmight change an expert opinion about the levelof uncertainty would be helpful. This boilsdown to researchers saying we may be veryuncertain now but are there warning signs thatpeople should look out for?ANGELA MCLEAN

LETS FOCUS ON WHAT WE NEED TOKNOW, NOT ON THE MOSTCOMPLICATED ANSWER

A more complicated solution to a problem isnot necessarily a better one. Keeping thingssimple is important when communicating whatpublic policy decisions are made, and why.

Take the debate about whether or not the UKsmandatory breast cancer screening programmeof women above a certain age can be justifiedby a reduced incidence of breast cancer. Thereis considerable scientific dispute among expertswho strongly disagree. And, any decisions haveto be communicated to both policy makers andindividuals. A recent review of evidencedeliberately used a very simple model of the

benefits and harms and was selective aboutwhat was included in the model. When theypublished their findings, the groupacknowledged the considerable uncertainty intheir estimates and explicitly said that, thoughthey could not quantify how uncertain theywere about the numbers, they were confidentenough to make policy recommendations.DAVID SPIEGELHALTER

20


22/28

PLAYING ONUNCERTAINTY

05On emotive, economically important andpolitical subjects, uncertainty in researchfindings has been played up and played downaccording to whether people favour or objectto the implications. And many implausibleproducts and theories are promoted byemphasising the uncertainty of mainstreamknowledge, such as alternative medicaltherapies that remind customers, scientistsdont really know what causes cancer....

IT JUST SHOWS HOW WRONG

SCIENTISTS CAN BE...In 2011, physicists were in the media talkingabout data that suggested a particle hadbroken the speed of light. Somecommentators immediately took this tosuggest that a fundamental law of physics hadbeen undermined (it was, in fact, a fault in therecording equipment, not nearly so widelyreported). A climate change contrarian arguedthat if we cannot have confidence in suchfundamental scientific principles, we certainlycannot have any confidence in the evidence

for global warming. In fact, researcherswillingness to find out whether this particlemight exist is a good example of why we canbe confident in scientific findings: if they'rewrong, they'll be corrected - but only whenevidence that stands up to scrutinycomes around.

are cell phones safe?

researchers still uncertain

why everything youve been told

about evolution is wrong

uncertainty over mobile phone

and brain cancer links

never trust anyone

who is certain

about anything...

21


23/28

PLAYING ON UNCERTAINTY

The idea of epigenetics in evolutionary biology suggests an individuals environment, as well astheir genes, can affect the way traits are inherited. When researchers first suggested it, creationistsasked whether this showed that the theory of evolution was wrong. In fact, epigenetics is adevelopment of evolutionary thinking rather than an invalidation of the theory of evolution.

These presentations of uncertainty take discussions down a blind alley because they distort theworking knowledge we do have. Findings are usually revisions and developments, not gamechangers. Uncertainty doesnt mean scientists know nothing.

DAVIDSTAINFORTH

The history of science is scattered withgame changers, rare discoveries whichsubstantially change our understanding ofhow the world works. These discoveries canchange what we think is happening, beyondanything previously considered a possibility.Examples include the realisation that theearth isnt the centre of the universe,working out the structure of DNA (and thatit encodes the information required to makeliving things), identifying that germs causedisease, or finding out that particles existthat are smaller than atoms.Although such discoveries are important forscience and can open up new opportunities

for society, they neednt necessarilyundermine conclusions based on our earlierunderstanding. Where earlier results arebacked up by robust and relevantobservations we can have confidence inthem even if they are later shown to be

based on an incomplete understanding.Scientists can often tell whether theirconclusions have a significant risk ofchanging as our understanding improves, orare likely to stay the same regardless of anyfuture discoveries. Sometimes observationscan point to the limits of current theory longbefore a new theory is developed. Forinstance, measurements of the orbit of theplanet Mercury suggested limitations inNewtons laws which were only explainedmany years later by Einstein.As a result some uncertainty estimates areextremely robust, thanks to their foundationon widely demonstrated phenomena and

methods, but others are much less so. Thedifficulty for non-specialists is that there israrely an easy way of identifying thedifference between the two. This pointneeds to be included in the questions put tothe experts.

GAME CHANGERS ARE EXCEPTIONAL

22


24/28


25/28

PLAYING ON UNCERTAINTY

UNCERTAINTY IS NOT A WEAKNESSOF SCIENCE!

1. Scientific research works on thebasis that there are things wedont know.Research is not done to prove things byshowing that there is no uncertainty, but tofind the best explanation for our observationsby testing each theory against alternatives.The most likely explanation, or theory, is theone that accounts for as much as possible ofwhat we see. Scientists can also say inadvance what evidence, if found, would refutetheir theory. And so to work out which of twocompeting alternatives is the better theory, we

shouldnt focus on the amount of uncertaintybut should ask how much of what we observecan each theory explain.

For example, sunspots (solar activity) havebeen suggested as an explanation for globalwarming. There is some evidence that solaractivity affects the atmosphere in ways thatcause the temperature to change. However,this theory cannot account for some importantfeatures of the observed warming patterns most significantly, while temperature hasincreased steadily over the last 50 years,solar activity has not. As a theory, it doesntaccount very well for our observations ofchanging climate.

You should ask anyone who promotes analternative idea of what is going on toindicate the uncertainty levels in theirown theory.

2. Scientists dont draw conclusionsbased on a single piece of evidence.

Scientists design experiments to gather datathat answer specific questions; the aim is notto agree but to explore, test results and retestthem. All measurements have an associateduncertainty this uncertainty in data isdifferent to uncertainty about a conclusion.Different interpretations of the same data areoften possible, and different datasets on thesame problem might reach differentconclusions. This is precisely why uncertaintyin any given piece of evidence does notnecessarily undermine an overall conclusion.

3. Scientific research seeks evidence

not consensus.Consensus suggests that scientists aim toagree. This incorrectly implies that scientists tryto minimise uncertainty for the sake of findingconsensus. When researchers in a field assessthe weight of evidence, they dont simplymean the number of studies on a particularquestion, or patients per study, but howcompelling the evidence is and howthoroughly alternative explanations have beenlooked at a result of the scientific processand peer review of new research.

4. Scientific research is not political,but the implications of researchcan be.When research has a bearing on public policyit may seem that a consensus, if that is whatemerges, tells people how to act or whatdecision to make. Similarly, if the policyimplications of research findings are clear,questioning the uncertainty of the sciencebecomes an easy way to dodge thoseimplications. The conclusions of research can

be, and often are, used as a proxy for politicalarguments. Researchers might wellrecommend or discourage a course of actionbut this is in the realm of policy making notresearch. Scientific evidence will seldom bethe only factor in making a decision orcreating policy.

WHY DOES ANY OF THIS MATTER?

Until we understand scientific uncertainty, werisk being seduced or confused by

misrepresentation and misunderstanding.Without an understanding of uncertaintyamongst the public and policymakers alike,scientists will struggle to talk about uncertaintyin their research and we will all find it hard toseparate evidence from opinion.

24


26/28

DELVING DEEPER

HANDLING UNCERTAINTY IN SCIENCEwas an interdisciplinary Royal Society meeting in 2010 to look

at how uncertainty is used in different areas of research.

http://rsta.royalsocietypublishing.org/content/369/1956.toc

MERCHANTS OF DOUBT (2011)is a book by Naomi Oreskes and Erik Conway on the tacticaluse of uncertainty by interest groups.

UNCERTAINTY AND STATISTICS:

UNDERSTANDING UNCERTAINTYis David Spiegelhalters website using everyday examples to help

people make sense of chance, risk, luck, uncertainty and probability.www.understandinguncertainty.org

HOW TO READ A PAPER (2010)is a book by Trisha Greenhalgh that explains the different statistical

approaches in research papers; some key insights are summarised

in this collection of BMJ articles.

www.bmj.com/about-bmj/resources-readers/publications/how-read-paper

MAKING SENSE OF STATISTICSis a guide, produced by Sense About Science with Straight

Statistics, that gives some questions to ask and identifies some pitfallsto avoid to help people get behind news stories that use statistics.

www.senseaboutscience.org/resources.php/1/making-sense-of-statistics

06

25


27/28

ALL MODELS ARE WRONGis a blog discussing uncertainty in relation to modelling and

climate science.

www.allmodelsarewrong.com

UNCERTAINTY IN MEDIA AND POLICY-MAKING:

COMMUNICATING UNCERTAINTYin a Soundbite is a guide from the Science Media Centre for

researchers doing brief news interviews about research.

www.sciencemediacentre.org/wp-content/uploads/2012/09/

Communicating-Uncertainty-in-a-Soundbite.pdf

CLIMATE SCIENCE, THE PUBLIC AND THE NEWS MEDIAis a report looking at the communication of evidence, including

uncertainty about evidence, in the context of discussions about

climate science.

www.lwec.org.uk/publications/climate-science-public-and-news-media

REDUCING RISK OF FUTURE DISASTERSis a report from the Government Office for Sciences Foresight

team that discusses policy making in relation to the risk of

natural disasters.

www.bis.gov.uk/foresight/our-work/policy-futures/disasters

POST NOTESare short briefings for Parliamentarians on scientific subjects and

include many topical areas that deal with scientific uncertainty

and the complexities of integrating uncertain science with actual

policy decisions.www.parliament.uk/mps-lords-and-offices/offices/bicameral/

post/publications/postnotes/

26


28/28

design

www.peel-me.com

IF YOU ARE

a commentator, medical charity, journalist, employer,

educator, national body, information service, local

authority, parliamentarian, health care provider,

professional association, community

group, NGO, lifestyle writer

or any other civic group in need of

help or comment on a difficult or

controversial area of science call

Sense About Science on

020 7490 9590.

For more copies or further information,contact Sense About Science on

[email protected]

or 020 7490 9590.

PUBLISHED IN 2013 BY

Sense About Science

Registered Charity No. 1146170

Company No. 677102714a Clerkenwell Green

London EC1R 0DP

Registered in England and Wales.

www.senseaboutscience.org

This document is prepared by Sense About Science, which

has final responsibility for the content.

It is licensed under Creative commons

Attribution-Noncommercial-No Derivative Works 2.0 UK:

England & Wales Licence.

Date post:	04-Jun-2018
Category:	Documents
Upload:	ahmetis
View:	221 times
Download:	0 times

'Making Sense of Uncertainty' Guide

Documents