Evaluation Framework 2.0June 2018
Acknowledgements:
Thanks are provided to the Evaluation Council, Directors of Communication, Engage Programme Board, the Cross-Government Insight and Evaluation forum, Professor Kevin Money (and associates), Dr Carla Groom, Dr Sarah Gates, Dr Cecile Morales and the Cabinet Office Insight and Evaluation team for their contributions to this guide.
Introduction from Alex Aiken, Executive Director of Government Communication
Evaluation Framework 2.0
This updated Evaluation Framework is provided to communicators across the wider public sector to assist in measuring the success of our work and appraising our activities. Evaluation remains a critical function for delivering effective communication activity, and this guide will help colleagues plan campaigns in a way that can be meaningfully evaluated. This will drive improvements across our profession, including our capability to provide impactful behaviour change and policy delivery. Ultimately this is about listening to stakeholders and the public so that we know which messages are landing, and how we can learn from that to make our communication more effective.
This Framework builds on the foundations created by the International Association for the Measurement and Evaluation of Communication (AMEC) and the Evaluation Framework that was a product of the Evaluation Council in 2016. These have been tailored to reflect our public service role and the latest campaign optimisation principles developed by the Engage1 programme, which brings data and science to the heart of our communication activity.
The new Framework edition is primarily aimed at paid campaign activity. It adds further guidance on calculating return on investment (ROI), recommends specific metrics for measurement depending on your campaign type, and enhances the guidance on measurement methods. It also introduces guidance on measuring reputation and the ethical use of data.
As all government communicators will know, successful evaluation depends entirely on setting meaningful C-SMART objectives. These are SMART objectives, with an additional C for ‘challenging’. New guidance is provided on the best practice for setting objectives in an OASIS plan to effectively evaluate communication activity and calculate the benefits.
Running a successful campaign requires clear objectives, underpinned by a theory of behaviour change that understands how communication activity will be effective. The GCS team have a number of publications and guides to assist campaign planning, the theory of behaviour change, and all elements of campaign planning. This guide should not be used in isolation, but will assist in effective evaluation from the outset of planning a campaign. All communication activity should consider evaluation and understand that measurement enables evaluation, which in turn becomes insight for future activity.
1 https://gcs.civilservice.gov.uk/about-us/improving-gcs/engage-programme
2
Evaluation Framework 2.0
Evaluation Framework 2.0This Framework provides guidance for major paid-for campaigns and other communication activity. The Engage programme has identified three distinct types of funded campaign activity.
Behaviour change
The vast majority of government communication seeks to change behaviours in order to implement government policy or improve society. In the benchmarking categories that sit within the Engage programme, we also distinguish the main types of behaviour change: start, stop and maintain. This way we can start to learn about which methods, messages and channels are effective for certain types of behaviour change. Raising awareness will nearly always be part of an activity to change behaviour and should also be measured.
Recruitment
Recruitment is a specific form of behaviour change where people are encouraged to start an activity. This is a major concern for government and is vital to maintaining public services and protecting the country. Government invests a lot of money in recruiting people for important jobs (teachers, armed forces, nurses etc) and so this campaign type has been isolated because of its size. This is targeted at major employment campaigns rather than recruiting people to ‘register’ or ‘take part’.
Awareness
Some campaigns solely seek to raise awareness of an issue or to change people’s attitude. Raising awareness will almost certainly be an intermediary step for all communication activity, so behaviour change campaigns are also encouraged to measure awareness. The awareness metrics here are mainly suggested for campaigns that seek to change attitudes but not immediately change behaviours.
3
Evaluation Framework 2.0
Each of these campaign types has a set of recommended evaluation metrics. Consistent use of these metrics will assist campaign planners in choosing appropriate objectives and enable our profession to establish benchmarks for success.
Metrics are divided according to the four categories identified by AMEC:
• inputs (what we put in, our planning and content creation)
• outputs (what is produced, such as audience reach)
• outtakes (subject-oriented stakeholder experiences and communicator-oriented learning about communication practice)
• outcomes (stakeholder behaviour, what the impact of communication and engagement activity is, and whether we achieved the desired organisational impact or policy aim)
The most important of these is outcomes: how effective communication activity is in achieving policy aims and delivering organisational impact. Outtakes are also important for measuring how well communication activity has worked, for example, by assessing the penetration of a campaign message.
This Framework also provides a set of metrics that can be used to assess low cost and no cost activity, which can equally be applied to internal communications and stakeholder engagement activity.
4
Evaluation Framework 2.0
EVALUATION
INPUTS OUTPUTS OUTTAKES OUTCOMES
INPUTS (Evidence-based insights about issues and stakeholders)
OUTPUTS (Communication and stakeholder engagement activities)
OUTTAKES (Stakeholder experience of communication activities- lessons learned)
OUTCOMES (Stakeholder behaviour and behaviour change)
What? Research, planning, design, cost and effort that were put in. What has been learned from previous communications activities? What is your theory of change? Content creation
E.g. partnerships secured, press coverage, target audience reach
Reception, perception and reaction of stakeholders. Campaign efficiency metrics for communicators
Changes in behaviour, changes in attitude and contribution to policy objectives
Why? This reflects what we have done to enable the activity
Tracking assets and collateral allows us to evaluate message selection as well as asset type
See if we have chosen effective messages and channels for engagement. Consider what we have learned about our communication practices
Consider if we have achieved what we set out to do in terms of fulfilling a policy or organisational impact, and see if our communications been effective
Example 1 (encourage people to apply for a home insulation subsidy)
£200,000 Previous campaigns had provided narratives and numbers. Theory of change suggested that infographics are a powerful way to present the complex information about the benefits of home insulation
85% of (2,000,000) target audience group reached with impression
Awareness of issue moved from 52% to 65% in target audience
Applications for a subsidy increased from 25,000 to 50,000.
Example 2 (e.g. decrease instances of drink driving)
5 FTE daysTheory of change suggested that local messages and messengers are effective for persistent offenders
1,000,000 impressions recorded on social media using geographically targeted messages and messengers
Active engagement rate was 4.6% compared to forecast 2.5%
Reported instances of ‘Y’ decreased from 500 to 450 per X number of tests in targeted areas
The stages of a communication activity should be viewed as a linked process. Each element is created by the previous element, and that in turn will be causally linked to the next stage. The evaluation of outcomes, outtakes and other elements should be used to inform campaign optimisation both in realtime and for future iterations.
5
Evaluation Framework 2.0
OASIS for evaluationThe OASIS2 campaign planning guide provides government communicators with a framework for preparing and executing effective communication activity. Within OASIS, Objectives and Scoring are especially important for the purpose of evaluation.
ObjectivesObjectives should be C-SMART: Challenging, Specific, Measurable, Attainable, Relevant and Time-bound. For the purpose of evaluation it also important that objectives contain three elements.
1. Baseline: A numerical prediction of what will be observed if no communication activity takes place. Some people would take out a pension even if the government ran no communication activity. A baseline should be set using the most recent data available, but some subject areas can use data from last year’s campaign, or exceptionally even earlier. In most cases we can assume that the no-campaign activity observation would be same as the last relevant measurement. It is important to consider predictable movements of the baseline in addition (e.g. we can predict that there will be around a 30% increase in wearable technology ownership in 2018). If it is cost prohibitive to establish a baseline specifically for campaign purposes, planners can use pre-existing publicly available data, research commissioned by policy colleagues, or a proxy measure as a substitute.
2. Change: A numerical forecast of the difference that the campaign activity will make. For example, improving a level of registrations from a baseline of 80,000 to 100,000 (an increase of 20,000 or 25%). Changes should be for a defined period of time, typically three to six months after a campaign for many large behaviour change campaigns. Where major changes are targeted over a longer period (e.g. five years) then milestones or intermediary targets should be provided for a period of no longer than one year.
3. Explanation: Campaign planners should use an evidence base as a justification for the level of change that is being targeted. Typically this might be in line with previous observations from the last time the campaign was run or by comparison to other similar campaigns that can offer some guidance on what level of change could reasonably be expected. Making assumptions is acceptable as long as they are clearly identified and justified. It is important to signal if the behaviour targeted is a start, stop or maintain behaviour. It is important to distinguish the effect of the campaign from other influential factors such as seasonality, fashion, public concern and evolving social norms.
There is more specific guidance available on Audience Insight (especially motivations and barriers), Strategy and Implementation in the full OASIS3 guide.
2 https://gcs.civilservice.gov.uk/wp-content/uploads/2015/09/OASIS-Guide-1-amended.pdf3 https://gcs.civilservice.gov.uk/wp-content/uploads/2015/09/OASIS-Guide-1-amended.pdf
6
Evaluation Framework 2.0
Scoring (Evaluation)Evaluation, focusing primarily on outcomes and outtakes, should take place throughout a campaign and will inform dynamic optimisation of active campaigns.
It is recommended that approximately 5 to 10% of total campaign expenditure is allocated to evaluation. In addition to operational data, evaluation costs will often include commissioning research to measure awareness and message penetration levels.
A complete evaluation will include the following aspects.
• A comparison of actual outcome data with targets set in objectives. Were the objectives met? If not, what reasons can be offered to explain the variation? If the objectives were surpassed, what has driven that?
• A comparison of outtakes with the targets set in objectives. This will typically include various data sources such as qualitative and/or quantitative research findings.
• Considering the causal link between the subject-oriented outtakes and the outcomes. Some campaigns will be more effective in converting awareness and attitudinal changes into tangible behavioural outcomes. To what extent could this campaign convert awareness to behaviour change?
• Findings for current or future campaign optimisation. Ideally this will include attribution modelling and econometric analysis (scientifically assigning a proportion of ‘cause’ to different elements, messages and channels of a campaign). Even without advanced studies, campaigners can often draw conclusions about which channels have been more or less effective than anticipated. Is there anything that others can learn from your theory of change?
• For active campaigns it is advisable to make small incremental adjustments (a slight up or down weighting of elements, messages or channels) to test theories for improvement. Ideally activity will be tested in a pilot beforehand, but this can even be in a live activity by varying message content and delivery channels.
• A conclusion including whether or not the campaign was successful in achieving its policy aims. This should also include what we would do differently next time or for future similar campaigns.
7
Evaluation Framework 2.0
Consistent metricsThe following tables present lists of recommended and potential metrics for campaign measurement and evaluation.
Inputs are presented in blue, outputs presented in orange, outtakes are presented in red, and outcomes are presented in green.
For low cost and no cost campaigns that follow, some metrics are identified as either outtakes or outcomes dependent on the aim of the campaign. These are represented in grey.
Please note:
• This is not an exhaustive list, and campaign planners are encouraged to measure any additional more specialised metrics.
• Typically, all communication activity will want to measure the level of awareness and/or agreement with a campaign message. This might be an ‘end’ or outcome in itself, but it will normally be an intermediary step for campaigns looking to achieve behavioural change.
• Not all metrics will necessarily be applicable for all campaigns. For example, campaigns without websites will not be able to measure dwell time.
• Large budget paid-for campaigns are advised to measure as many of these as possible. Low-cost or no-cost campaigns are advised to choose the one or two properties most suitable to their campaign in each activity phase.
• Thinking through outtakes and outcomes can be used to assist in choosing suitable objectives for campaigns.
8
Evaluation Framework 2.0
Behaviour changeMetric Online/
offlineDefinition Measurement
method
Inputs
Total spend to date Both Aggregate total spend so far £
Spend to date Online How much money has been spent on digital media
£
Spend to date Offline Sum of one-off set up costs (manual from PASS) and periodic offline media spend updates
£
What is your theory of change (including evidence base)?
N/A Implementation of behavioural science in planning effective communication
Binary – yes/no is in place? Yes/no – current evidence base
Content creation Both Infographics, videos etc Volume by type
Outputs
Estimated total reach Both Aggregate audience reach Absolute number and proportion of target audience
Reported online reach Online The estimated reach as reported by digital platforms
Absolute number and proportion of target audience
Estimated offline reach Offline Reported audience reach for offline media
Absolute number and proportion of target audience
9
Evaluation Framework 2.0
Outtakes
Cost per outcome N/A The unit cost per behaviour change
£
Engagements/interactions
Online The % of impressions generating an interaction (share/like/comment)
Actions which involve active engagement (e.g. typing, not just 'one-click' endorsements)
Completion/registration rate
Both The proportion of contacts/impressions that go on to complete sign-up/registration
%
Cost per completion/registration
Both Unit cost of registration/completion
£
Unprompted campaign issue awarenesse.g. spontaneous recall metric
Offline The number and proportion of target audience that has unprompted campaign issue awareness
Absolute number and proportion of target audience
Experience of different messages that relate to aspects of theory of change
Both The extent to which different groups agree/disagree with messages related to theory of change
5 point scale (agreement/ disagreement with aspects of message)
Outcomes
Behaviour change(#, %) e.g. number of licensed anglers vs baseline
Offline The number and proportion of target audience that has changed behaviour
Absolute number and proportion of target audience
Stated/intended behaviour change
Offline The proportion of target audience that claim they will act in accordance with campaign aim
Absolute number and proportion of target audience
Advocacy e.g. agreement with value for money statement
Offline The number and proportion of target audience that agree with the campaign message (have positive sentiment)
5 point scale recommended (strongly agree/slightly agree/don't know etc)
Current ROI N/A Unit benefit multiplied by number of behaviour changes
£
10
Evaluation Framework 2.0
RecruitmentMetric Online/
offlineDefinition Measurement
method
Inputs
Total spend to date Both Aggregate total spend so far £
Spend to date Online How much money has been spent on digital media
£
Spend to date Offline Sum of one-off set up costs (manual from PASS) and periodic offline media spend updates
£
Content creation Both Infographics, videos etc Volume by type
Outputs
Estimated total reach Both Aggregate audience reach Absolute number and proportion of target audience
Reported online reach Online The estimated reach as reported by digital platforms
Absolute number and proportion of target audience
Estimated offline reach Offline Reported audience reach for offline media
Absolute number and proportion of target audience
11
Evaluation Framework 2.0
Outtakes
Expressions of interest (EOI)
Both The number of people actively expressing interest in applying
Absolute number and % of target
Cost/recruit N/A The unit marketing cost per successful recruit
£
EOI/applicant conversion ratio
Both The proportion of EOIs that go on to be applicants
%
Applicant/recruit conversion
Both The proportion of applicants that go on to be employed
%
Cost/EOI Both Total spend / EOIs £
Cost/applicant Both The unit marketing cost per applicant
£
Engagements/interactions
Online The % of impressions generating an interaction (share/like/comment)
Actions which involve active engagement (e.g. typing, not just 'one-click' endorsements)
Sentiment toward profession
Offline The regard in which the profession is held by the target audience or general public
5 point scale recommended (strongly agree / slightly agree / don't know etc)
Outcomes
Recruits Offline The number of people successfully recruited
Absolute number and % of target
Intra-profession advocacy
Offline The degree to which current professionals would recommend the job to friends/family
5 point scale recommended (strongly agree / slightly agree / don't know etc)
Influencer advocacy Offline The degree to which important influencers (e.g. parents) would support entry to profession
5 point scale recommended (strongly agree / slightly agree / don't know etc)
Applications Both Unit cost of registration/completion
Absolute number and % of target
12
Evaluation Framework 2.0
AwarenessMetric Online/
offlineDefinition Measurement
method
Inputs
Total spend to date Both Aggregate total spend so far £
Spend to date Online How much money has been spent on digital media
£
Content Creation Both Infographics, videos etc Volume by type
Spend to date Offline Sum of one-off set up costs (manual from PASS) and periodic offline media spend updates
£
Outputs
Estimated total reach Both Aggregate audience reach Absolute number and proportion of target audience
Reported reach Online The estimated reach as reported by digital platforms
Absolute number and proportion of target audience
Estimated offline reach Offline Reported audience reach for offline media
Absolute number and proportion of target audience
13
Evaluation Framework 2.0
Outtakes
Cost per outcome N/A The unit cost of raising awareness
£
Active engagements/interactions
Online The % of impressions generating an interaction (share/like/comment)
Actions which involve active engagement (e.g. typing, not just 'one-click' endorsements)
CTR Online Click through rate: the proportion of impressions generating a click-through
%
VTR Online View through rate: the proportion of impressions meeting a minimum view-through percentage
%
Dwell time Online The average length of time spend on a campaign site
Minutes and seconds
Bounce rate Online % of site visitors that navigate no further than the landing page
%
Prompted campaign recognisers
Offline The proportion of target audience that recalls seeing the campaign when prompted
Absolute number and proportion of target audience
Outcomes
Advocacye.g. agreement with value for money statement
Offline The number and proportion of target audience that agree with the campaign message (have positive sentiment)
5 point scale recommended (strongly agree / slightly agree / don't know etc)
Unprompted campaign issue awareness
Offline The number and proportion of target audience that has unprompted campaign issue awareness
Absolute number and proportion of target audience
Agreement (# and %) Offline The number and proportion of target audience that agree with the campaign message (have positive sentiment)
5 point scale recommended (strongly agree / slightly agree / don't know etc)
14
Evaluation Framework 2.0
All communication activityIncluding low cost, no cost, internal communications and stakeholder engagement activities.
Metric Online/offline
Definition Measurement method
Inputs
Total spend to date Both Aggregate total spend so far £
Content creation Both Infographics, videos etc Volume by type
Volume of press releases
Offline # of press releases sent out # of press releases sent out
Volume of SM releases Online # of releases to owned social media channels
# of releases to owned social media channels
FTE days Offline Total amount of time invested in campaign preparation in terms of person effort
<1, or whole number
Outputs
Estimated total reach Both Aggregate audience reach Absolute number and proportion of target audience
Estimated offline reach Offline Reported audience reach for offline media
Absolute number and proportion of target audience
Reported/estimated online reach
Online The estimated reach as reported by digital platforms
Absolute number and proportion of target audience
Direct contacts Both # of direct on/offline contacts (e.g. eDM etc)
Absolute number and proportion of target audience
Events organised Offline Volume of events # of events
# of attendees
Volume of coverage Both # exposures # press cuts
# broadcasts (break local/national)
Partnerships secured Offline # of partnerships providing amplifying support
Formal sign-up to either actively promote and amplify campaign material or make any form of VIK or financial contribution
15
Evaluation Framework 2.0
Outtakes
Passive engagements/interactions
Online The % of impressions generating an interaction (share/like/retweet)
A 'one-click' interaction
Active engagements/interactions
Online The % of impressions generating an interaction (comment/response/quote)
Something that involves proactive engagement (e.g. typing)
CTR Online Click through rate: the proportion of impressions generating a click-through
%
VTR Online View through rate: the proportion of impressions meeting a minimum view-through percentage
%
Dwell time Online The average length of time spend on a campaign site
Minutes and seconds
Bounce rate Online % of site visitors that navigate no further than the landing page
%
Prompted campaign recognisers
Offline The proportion of target audience that recalls seeing the campaign when prompted
Absolute number and proportion of target audience
Stated/intended behaviour change
Offline The proportion of target audience that claim they will act in accordance with campaign aim
Absolute number and proportion of target audience
EOI/applicant conversion ratio
Both The proportion of EOIs that go on to be applicants
%
Applicant/recruit conversion
Both The proportion of applicants that go on to be employed
%
Reputation perception
Offline See later guidance See later guidance
Response rate Both % of contacts to respond % of contacts to respond
Cost per outcome
N/A The unit cost of raising awareness £
Cost/applicant Both The unit marketing cost per applicant £
Cost/EOI Both Total spend / EOIs £
Cost per completion/registration
Both Unit cost of registration/completion £
Sentiment Either Degree to which a message has been positively or negatively received
Valid for manual assessment of press coverage or a limited (defined) number of stakeholders. Not currently valid for mass automated approach.
16
Evaluation Framework 2.0
Outtakes/outcomes4
Agreement (# and %) Offline The number and proportion of target audience that agree with the campaign message (have positive sentiment)
5 point scale recommended (strongly agree / slightly agree / don't know etc)
Completion/registration rate
Both The proportion of contacts/impressions that go on to complete sign-up/registration
%
Expressions of interest (EOI)
Both The number of people actively expressing interest in applying
Absolute number and proportion of target audience
Unprompted campaign issue awareness (# and %)
Offline The number and proportion of target audience that has unprompted campaign issue awareness
Absolute number and proportion of target audience
Outcomes
Behaviour change (#, %)
Offline The number and proportion of target audience that has changed behaviour
Absolute number and proportion of target audience
Current ROI N/A Unit benefit multiplied by # behaviour changes
£ and X:Y
Recruits Offline The number of people successfully recruited
Absolute number and proportion of target audience
Expressions of interest (EOI)
Both The number of people actively expressing interest in applying
Absolute number and proportion of target audience
Advocacye.g. agreement with value for money statement
Offline The number and proportion of target audience that agree with the campaign message (have positive sentiment)
5 point scale recommended (strongly agree / slightly agree / don't know etc)
Applications Both Unit cost of registration/completion
Absolute number and proportion of target audience
Attitudinal change Both Degree to which people's attitude has changed in favour of the campaign
5 point scale recommended (strongly agree / slightly agree / don't know etc)
4 The input/output/outtake/outcome model is helpful for understanding where metrics fit in the communication activity cycle. Tracking appropriate metrics is more important than defining which category they sit in. Depending on the aims of a campaign, for example, raising awareness can be either an outtake or an outcome
17
Evaluation Framework 2.0
18
Evaluation Framework 2.0
..
Less
ons
lear
ned
Com
mun
icat
ions
ob
ject
ives
Audi
ence
Key
suc
cess
es
Your
less
ons
lear
nt s
houl
d be
mea
ning
ful –
thes
e sh
ould
incl
ude
item
s th
at w
ould
be
usef
ul fo
r you
to re
mem
ber a
nd y
our c
olle
ague
s to
be
awar
e of
for f
utur
e ac
tivity
. Thi
s co
uld
be a
par
ticul
ar s
ucce
ss a
fter t
ryin
g so
met
hing
new
, or r
efle
ctio
ns o
n so
met
hing
that
cou
ld h
ave
gone
bet
ter.
Rem
embe
r, if
som
ethi
ng w
ent w
rong
, it’
s m
ore
usef
ul to
als
o th
ink
abou
t ste
ps y
ou c
ould
take
nex
t tim
e to
miti
gate
this
, rat
her t
han
just
say
ing
som
ethi
ng d
idn’
t wor
k.
Gol
den
exam
ple :
Hol
ding
a b
usin
ess
stak
ehol
der e
vent
in th
e m
orni
ng ra
ther
than
the
afte
rnoo
n en
able
d m
ore
seni
or re
pres
enta
tives
to a
ttend
.
2. O
utpu
ts(d
eliv
ered
–w
hat y
our a
ctiv
ity d
eliv
ered
and
aud
ienc
e re
ache
d)Th
e ai
m h
ere
is to
refle
ctth
e vo
lum
e of
you
r aud
ienc
e yo
u’ve
reac
hed
via
the
wor
k yo
u di
d in
yo
ur in
puts
sec
tion.
Incl
ude
KP
Is a
long
side
the
outp
uts
belo
w.
Med
ia
Exa
mpl
es in
clud
e:
cove
rage
ach
ieve
d (b
roke
n do
wn
by n
atio
nal,
regi
onal
, tra
de a
nd
broa
dcas
t)
key
mes
sage
pen
etra
tion
%
of c
over
age
incl
udin
g m
inis
teria
l/spo
kesp
erso
n qu
ote
Dig
ital
Exa
mpl
es in
clud
e:
aver
age
impr
essi
ons/
reac
h ac
ross
con
tent
(by
chan
nel)
w
ebsi
te/G
OV
.UK
pag
e/co
nsul
tatio
n vi
sits
(tot
al a
nd u
niqu
e)If
utili
sing
pai
d fo
r act
ivity
, sep
arat
e th
is a
nd o
rgan
ic o
utpu
ts.
Exte
rnal
affa
irsE
xam
ples
incl
ude:
co
mm
unic
atio
ns d
eliv
ered
and
aud
ienc
e re
ache
d (n
umbe
r of
lette
rs/e
mai
ls/p
acks
issu
ed, c
alls
mad
e, s
take
hold
ers
reac
hed)
ev
ents
suc
cess
fully
del
iver
ed/a
ttend
ance
supp
ortiv
e st
atem
ent s
ecur
ed fr
om p
artn
ers/
stak
ehol
ders
Inte
rnal
Exa
mpl
es in
clud
e:
audi
ence
reac
hed
(blo
g/st
ory
view
s)
even
ts s
ucce
ssfu
lly d
eliv
ered
/atte
ndan
ce
Act
ivity
title
Dat
es th
is e
valu
atio
n co
vers
Rel
ated
obj
ectiv
e an
d pr
ogra
mm
e
Your
obj
ectiv
es s
houl
d be
as
SM
AR
T as
pos
sibl
e –
spec
ific,
mea
sura
ble,
ach
ieva
ble,
real
istic
and
timel
y.G
olde
n ex
ampl
es: ‘
Incr
ease
reco
gniti
on a
mon
gst 1
8-34
s of
the
econ
omic
opp
ortu
nitie
s of
***
by
5% b
y D
ec 2
019
(vs
Jan
2019
bas
elin
e of
20%
)’ ‘T
o dr
ive
resp
onse
s to
***
cons
ulta
tion/
gree
n pa
per a
nd g
ener
ate
500
resp
onse
s’.
Som
etim
es it
is h
ard
to m
ake
your
obj
ectiv
es p
erfe
ctly
SM
AR
T if
you
don’
t hav
e su
ppor
ting
track
ing
data
. An
exam
ple
of a
n al
mos
t SM
AR
T ob
ject
ive
mig
ht b
e:
‘Rai
se a
war
enes
s of
***
anno
unce
men
t, ge
nera
ting
***
piec
es o
f nat
iona
l cov
erag
e an
d **
* ad
voca
cy s
tate
men
ts’.
This
sec
tion
shou
ld d
etai
l the
thin
gs y
ou’re
mos
t pro
ud o
f! It
coul
d be
thin
gs li
ke:
•m
eetin
g (o
r eve
n ex
ceed
ing)
you
r KP
Is (k
ey p
erfo
rman
ce in
dica
tors
)•
apa
rticu
larly
impa
ctfu
l pie
ce o
f cov
erag
e•
gain
ing
advo
cacy
from
real
ly k
ey s
take
hold
ers,
or m
ovin
g a
noto
rious
ly n
egat
ive
stak
ehol
der i
nto
a m
ore
posi
tive
posi
tion
•po
sitiv
e fe
edba
ck o
n an
inte
rnal
sta
ff ev
ent
4. O
utco
mes
(wha
t was
the
resu
lt)Th
e ai
m h
ere
is to
refle
ctth
e ov
eral
l res
ult o
f you
r act
ivity
as
it re
late
s to
a c
hang
e of
atti
tude
or
beha
viou
r e.g
. did
you
r tar
get a
udie
nce
adop
t an
inte
nded
ser
vice
? In
clud
e K
PIs
.
Th
is s
pace
is to
sho
w h
ow y
ou m
ade
prog
ress
and
met
you
r obj
ectiv
es. T
his
coul
d be
sho
wn
thro
ugh
advo
cacy
, atti
tude
cha
nge
thro
ugh
pre
and
post
ac
tivity
pol
ling,
sta
ff su
rvey
resu
lts, o
nlin
e re
gist
ratio
ns o
r for
m
com
plet
ions
.
Rem
embe
r, as
wel
l as
deta
iling
advo
cacy
, mov
ing
stak
ehol
ders
from
a
nega
tive
to n
eutra
l pos
ition
is s
till a
suc
cess
.
3. O
utta
kes
(res
pons
es to
con
tent
–w
hat y
our t
arge
t aud
ienc
e th
inks
, fe
els
and
does
)Th
e ai
m h
ere
is to
enc
apsu
late
you
r tar
get a
udie
nce’
s re
spon
se to
you
r act
ivity
–w
ere
they
po
sitiv
e ab
out i
t, di
d th
ey ta
ke a
ctio
n as
a re
sult
of y
our c
omm
unic
atio
ns?
Incl
ude
KP
Is.
Med
ia
Exa
mpl
es in
clud
e:
% s
entim
ent o
f cov
erag
e (p
ositi
ve a
nd/o
r neu
tral)
re
spon
ses
(and
any
cov
erag
e th
at in
clud
ed th
ird p
arty
reac
tion)
Dig
ital
Exa
mpl
es in
clud
e:
soci
al m
edia
eng
agem
ent (
follo
ws,
like
s, s
hare
s, c
omm
ent/r
eplie
s,
retw
eets
by
chan
nel)
av
erag
e en
gage
men
t rat
e (b
y ch
anne
l)
shar
es o
f con
tent
from
key
sta
keho
lder
s
use
of o
ffici
al #
men
tions
ac
tions
(clic
k th
roug
hs, c
ost p
er c
licks
, dow
nloa
ds, s
ubsc
riptio
ns)
se
ntim
ent (
via
man
ual d
ippe
d sa
mpl
e ch
eck)
av
erag
e tim
e on
site
Exte
rnal
affa
irsE
xam
ples
incl
ude:
st
akeh
olde
r sen
timen
t (po
sitiv
e en
gage
men
t, nu
mbe
rs o
f pos
itive
st
atem
ents
, adv
ocac
y st
atem
ent –
refe
r to
the
advo
cacy
gui
de fo
r gu
idan
ce o
n th
is)
st
akeh
olde
r fee
dbac
k (s
urve
y fe
edba
ck a
bout
eve
nt)
Inte
rnal
Exa
mpl
es in
clud
e:
resp
onse
s/fe
edba
ck (l
ikes
and
com
men
ts o
n bl
ogs/
new
s, s
urve
y fe
edba
ck a
bout
eve
nt)
en
gage
men
t (cl
ick
thro
ughs
)
surv
ey re
sults
(aw
aren
ess,
reca
ll)
1. In
puts
(pre
para
tion
–th
e w
ork
you
do b
efor
e or
dur
ing
the
activ
ity)
The
aim
her
e is
to re
flect
on th
e w
ork
you’
ve p
ut in
to a
n ac
tivity
. You
sho
uld
also
incl
ude
wor
k th
at m
ay n
ot h
ave
resu
lted
in a
n ou
tput
/out
take
/out
com
e e.
g. a
n op
-ed
that
was
n’t e
vent
ually
in
clud
ed in
a p
ublic
atio
n. U
se th
e le
sson
s le
arnt
to e
xpla
in w
hy s
omet
hing
did
n’t w
ork.
Med
iaE
xam
ples
incl
ude:
co
nten
t cre
atio
n (p
ress
not
ices
, min
iste
rial/s
poke
sper
son/
stak
ehol
der
quot
es, s
peec
hes,
op-
eds,
Q&
As,
med
ia a
nd tr
ail s
crip
ts)
pr
e-en
gage
men
t act
ivity
(brie
fing
jour
nalis
ts a
nd/o
r med
ia o
utle
ts,
brie
fing
OG
Ds
med
ia te
ams,
org
anis
ing
inte
rvie
ws/
med
ia ro
unds
)
Dig
ital
Exa
mpl
es in
clud
e:
pl
anni
ng a
nd s
ched
ulin
g (c
onte
nt c
alen
dar d
evel
oped
, eng
agem
ent p
lan
prod
uced
, cha
nnel
s ch
osen
)
cont
ent d
evel
opm
ent (
web
pag
e de
velo
pmen
t, m
essa
ge te
stin
g,
audi
ence
seg
men
tatio
n fo
r pai
d-fo
r act
ivity
)
cont
ent c
reat
ion
(soc
ial m
edia
pos
ts d
rafte
d,
info
grap
hics
/vid
eos/
quot
epic
scr
eate
d, G
OV
.UK
pag
es c
reat
ed,
cons
ulta
tion
page
cre
ated
, ad
copy
pro
vide
d fo
r pai
d fo
r act
ivity
, tra
cked
lin
ks c
reat
ed w
hich
sho
uld
be s
epar
ated
by
chan
nel e
.g. ‘
4 Tw
itter
in
fogr
aphi
cs c
reat
ed’ i
f rel
evan
t)
pre-
enga
gem
ent (
influ
ence
r/par
tner
s id
entif
ied
and
enga
ged,
dig
ital
tool
kits
sha
red,
OG
Ds
digi
tal t
eam
s br
iefe
d)If
utili
sing
pai
d fo
r act
ivity
, sep
arat
e it
from
org
anic
inpu
t met
rics.
Exte
rnal
affa
irsE
xam
ples
incl
ude:
pl
anni
ng (s
take
hold
er m
appi
ng, s
take
hold
er m
anag
emen
t pla
n, c
all l
ist
prep
ared
)
cont
ent c
reat
ion
(tool
kits
, pac
ks, c
ase
stud
ies
prod
uced
, dra
fted
emai
ls,
call
scrip
t dra
fted,
lette
r dra
fted)
even
ts o
rgan
ised
(ven
ue s
ecur
ed, a
ttend
ees
invi
ted,
roun
dtab
les)
pr
e-en
gage
men
t act
ivity
(sta
keho
lder
s id
entif
ied,
OG
D e
xter
nal a
ffairs
te
am b
riefe
d, s
take
hold
ers
brie
fed)
Inte
rnal
Exa
mpl
es in
clud
e:
plan
ning
(stra
tegi
c na
rrat
ive
draf
ted)
ev
ents
org
anis
ed (s
taff
brie
fing,
dro
p in
s, s
tand
ups
, web
chat
)
cont
ent c
reat
ed (b
log/
new
s st
ory
draf
ted,
scr
een
info
grap
hics
de
velo
ped,
vid
eo c
reat
ed, c
ase
stud
y dr
afte
d, in
trane
t ban
ner c
reat
ed)
pr
e-en
gage
men
t (st
akeh
olde
rs b
riefe
d, s
urve
y is
sued
)
Com
mun
icat
ions
, par
tner
ship
s an
d go
vern
ance
Like
you
r obj
ectiv
es, t
ry to
be
as d
etai
led
as p
ossi
ble
whe
n de
scrib
ing
you
audi
ence
. Thi
s co
uld
be in
term
s of
age
, sec
tor,
geog
raph
ic lo
catio
n et
c. T
his
will
mak
e it
easi
er
whe
n ev
alua
ting
whe
ther
you
reac
hed
this
aud
ienc
e th
roug
h yo
ur a
ctiv
ity.
Gol
den
exam
ple:
Ins
tead
of s
ayin
g ‘g
ener
al p
ublic
’, ar
e yo
u ac
tual
ly a
imin
g fo
r ‘18
-24
year
old
s’ o
r ‘18
-24
year
old
s w
orki
ng in
a p
artic
ular
sec
tor’?
This
gui
de w
ill ta
ke y
ou th
roug
h w
hat y
ou
coul
d in
clud
e fo
r eac
h se
ctio
n in
an
eval
uatio
n re
port
for a
low
cos
t/no
cost
act
ivity
. Not
all
met
rics
will
be
rele
vant
to e
very
type
of a
ctiv
ity
–ju
st u
se th
e on
es a
pplic
able
to y
our w
ork.
Fo
r fur
ther
gui
danc
e co
ntac
t the
insi
ght a
nd
eval
uatio
n te
am.
Example dashboard5
5 The full example of this template, and a blank version for you to populate can be found at:
19
Evaluation Framework 2.0
..
Less
ons
lear
ned
Com
mun
icat
ions
ob
ject
ives
Audi
ence
Key
suc
cess
es
Your
less
ons
lear
nt s
houl
d be
mea
ning
ful –
thes
e sh
ould
incl
ude
item
s th
at w
ould
be
usef
ul fo
r you
to re
mem
ber a
nd y
our c
olle
ague
s to
be
awar
e of
for f
utur
e ac
tivity
. Thi
s co
uld
be a
par
ticul
ar s
ucce
ss a
fter t
ryin
g so
met
hing
new
, or r
efle
ctio
ns o
n so
met
hing
that
cou
ld h
ave
gone
bet
ter.
Rem
embe
r, if
som
ethi
ng w
ent w
rong
, it’
s m
ore
usef
ul to
als
o th
ink
abou
t ste
ps y
ou c
ould
take
nex
t tim
e to
miti
gate
this
, rat
her t
han
just
say
ing
som
ethi
ng d
idn’
t wor
k.
Gol
den
exam
ple :
Hol
ding
a b
usin
ess
stak
ehol
der e
vent
in th
e m
orni
ng ra
ther
than
the
afte
rnoo
n en
able
d m
ore
seni
or re
pres
enta
tives
to a
ttend
.
2. O
utpu
ts(d
eliv
ered
–w
hat y
our a
ctiv
ity d
eliv
ered
and
aud
ienc
e re
ache
d)Th
e ai
m h
ere
is to
refle
ctth
e vo
lum
e of
you
r aud
ienc
e yo
u’ve
reac
hed
via
the
wor
k yo
u di
d in
yo
ur in
puts
sec
tion.
Incl
ude
KP
Is a
long
side
the
outp
uts
belo
w.
Med
ia
Exa
mpl
es in
clud
e:
cove
rage
ach
ieve
d (b
roke
n do
wn
by n
atio
nal,
regi
onal
, tra
de a
nd
broa
dcas
t)
key
mes
sage
pen
etra
tion
%
of c
over
age
incl
udin
g m
inis
teria
l/spo
kesp
erso
n qu
ote
Dig
ital
Exa
mpl
es in
clud
e:
aver
age
impr
essi
ons/
reac
h ac
ross
con
tent
(by
chan
nel)
w
ebsi
te/G
OV
.UK
pag
e/co
nsul
tatio
n vi
sits
(tot
al a
nd u
niqu
e)If
utili
sing
pai
d fo
r act
ivity
, sep
arat
e th
is a
nd o
rgan
ic o
utpu
ts.
Exte
rnal
affa
irsE
xam
ples
incl
ude:
co
mm
unic
atio
ns d
eliv
ered
and
aud
ienc
e re
ache
d (n
umbe
r of
lette
rs/e
mai
ls/p
acks
issu
ed, c
alls
mad
e, s
take
hold
ers
reac
hed)
ev
ents
suc
cess
fully
del
iver
ed/a
ttend
ance
supp
ortiv
e st
atem
ent s
ecur
ed fr
om p
artn
ers/
stak
ehol
ders
Inte
rnal
Exa
mpl
es in
clud
e:
audi
ence
reac
hed
(blo
g/st
ory
view
s)
even
ts s
ucce
ssfu
lly d
eliv
ered
/atte
ndan
ce
Act
ivity
title
Dat
es th
is e
valu
atio
n co
vers
Rel
ated
obj
ectiv
e an
d pr
ogra
mm
e
Your
obj
ectiv
es s
houl
d be
as
SM
AR
T as
pos
sibl
e –
spec
ific,
mea
sura
ble,
ach
ieva
ble,
real
istic
and
timel
y.G
olde
n ex
ampl
es: ‘
Incr
ease
reco
gniti
on a
mon
gst 1
8-34
s of
the
econ
omic
opp
ortu
nitie
s of
***
by
5% b
y D
ec 2
019
(vs
Jan
2019
bas
elin
e of
20%
)’ ‘T
o dr
ive
resp
onse
s to
***
cons
ulta
tion/
gree
n pa
per a
nd g
ener
ate
500
resp
onse
s’.
Som
etim
es it
is h
ard
to m
ake
your
obj
ectiv
es p
erfe
ctly
SM
AR
T if
you
don’
t hav
e su
ppor
ting
track
ing
data
. An
exam
ple
of a
n al
mos
t SM
AR
T ob
ject
ive
mig
ht b
e:
‘Rai
se a
war
enes
s of
***
anno
unce
men
t, ge
nera
ting
***
piec
es o
f nat
iona
l cov
erag
e an
d **
* ad
voca
cy s
tate
men
ts’.
This
sec
tion
shou
ld d
etai
l the
thin
gs y
ou’re
mos
t pro
ud o
f! It
coul
d be
thin
gs li
ke:
•m
eetin
g (o
r eve
n ex
ceed
ing)
you
r KP
Is (k
ey p
erfo
rman
ce in
dica
tors
)•
apa
rticu
larly
impa
ctfu
l pie
ce o
f cov
erag
e•
gain
ing
advo
cacy
from
real
ly k
ey s
take
hold
ers,
or m
ovin
g a
noto
rious
ly n
egat
ive
stak
ehol
der i
nto
a m
ore
posi
tive
posi
tion
•po
sitiv
e fe
edba
ck o
n an
inte
rnal
sta
ff ev
ent
4. O
utco
mes
(wha
t was
the
resu
lt)Th
e ai
m h
ere
is to
refle
ctth
e ov
eral
l res
ult o
f you
r act
ivity
as
it re
late
s to
a c
hang
e of
atti
tude
or
beha
viou
r e.g
. did
you
r tar
get a
udie
nce
adop
t an
inte
nded
ser
vice
? In
clud
e K
PIs
.
Th
is s
pace
is to
sho
w h
ow y
ou m
ade
prog
ress
and
met
you
r obj
ectiv
es. T
his
coul
d be
sho
wn
thro
ugh
advo
cacy
, atti
tude
cha
nge
thro
ugh
pre
and
post
ac
tivity
pol
ling,
sta
ff su
rvey
resu
lts, o
nlin
e re
gist
ratio
ns o
r for
m
com
plet
ions
.
Rem
embe
r, as
wel
l as
deta
iling
advo
cacy
, mov
ing
stak
ehol
ders
from
a
nega
tive
to n
eutra
l pos
ition
is s
till a
suc
cess
.
3. O
utta
kes
(res
pons
es to
con
tent
–w
hat y
our t
arge
t aud
ienc
e th
inks
, fe
els
and
does
)Th
e ai
m h
ere
is to
enc
apsu
late
you
r tar
get a
udie
nce’
s re
spon
se to
you
r act
ivity
–w
ere
they
po
sitiv
e ab
out i
t, di
d th
ey ta
ke a
ctio
n as
a re
sult
of y
our c
omm
unic
atio
ns?
Incl
ude
KP
Is.
Med
ia
Exa
mpl
es in
clud
e:
% s
entim
ent o
f cov
erag
e (p
ositi
ve a
nd/o
r neu
tral)
re
spon
ses
(and
any
cov
erag
e th
at in
clud
ed th
ird p
arty
reac
tion)
Dig
ital
Exa
mpl
es in
clud
e:
soci
al m
edia
eng
agem
ent (
follo
ws,
like
s, s
hare
s, c
omm
ent/r
eplie
s,
retw
eets
by
chan
nel)
av
erag
e en
gage
men
t rat
e (b
y ch
anne
l)
shar
es o
f con
tent
from
key
sta
keho
lder
s
use
of o
ffici
al #
men
tions
ac
tions
(clic
k th
roug
hs, c
ost p
er c
licks
, dow
nloa
ds, s
ubsc
riptio
ns)
se
ntim
ent (
via
man
ual d
ippe
d sa
mpl
e ch
eck)
av
erag
e tim
e on
site
Exte
rnal
affa
irsE
xam
ples
incl
ude:
st
akeh
olde
r sen
timen
t (po
sitiv
e en
gage
men
t, nu
mbe
rs o
f pos
itive
st
atem
ents
, adv
ocac
y st
atem
ent –
refe
r to
the
advo
cacy
gui
de fo
r gu
idan
ce o
n th
is)
st
akeh
olde
r fee
dbac
k (s
urve
y fe
edba
ck a
bout
eve
nt)
Inte
rnal
Exa
mpl
es in
clud
e:
resp
onse
s/fe
edba
ck (l
ikes
and
com
men
ts o
n bl
ogs/
new
s, s
urve
y fe
edba
ck a
bout
eve
nt)
en
gage
men
t (cl
ick
thro
ughs
)
surv
ey re
sults
(aw
aren
ess,
reca
ll)
1. In
puts
(pre
para
tion
–th
e w
ork
you
do b
efor
e or
dur
ing
the
activ
ity)
The
aim
her
e is
to re
flect
on th
e w
ork
you’
ve p
ut in
to a
n ac
tivity
. You
sho
uld
also
incl
ude
wor
k th
at m
ay n
ot h
ave
resu
lted
in a
n ou
tput
/out
take
/out
com
e e.
g. a
n op
-ed
that
was
n’t e
vent
ually
in
clud
ed in
a p
ublic
atio
n. U
se th
e le
sson
s le
arnt
to e
xpla
in w
hy s
omet
hing
did
n’t w
ork.
Med
iaE
xam
ples
incl
ude:
co
nten
t cre
atio
n (p
ress
not
ices
, min
iste
rial/s
poke
sper
son/
stak
ehol
der
quot
es, s
peec
hes,
op-
eds,
Q&
As,
med
ia a
nd tr
ail s
crip
ts)
pr
e-en
gage
men
t act
ivity
(brie
fing
jour
nalis
ts a
nd/o
r med
ia o
utle
ts,
brie
fing
OG
Ds
med
ia te
ams,
org
anis
ing
inte
rvie
ws/
med
ia ro
unds
)
Dig
ital
Exa
mpl
es in
clud
e:
pl
anni
ng a
nd s
ched
ulin
g (c
onte
nt c
alen
dar d
evel
oped
, eng
agem
ent p
lan
prod
uced
, cha
nnel
s ch
osen
)
cont
ent d
evel
opm
ent (
web
pag
e de
velo
pmen
t, m
essa
ge te
stin
g,
audi
ence
seg
men
tatio
n fo
r pai
d-fo
r act
ivity
)
cont
ent c
reat
ion
(soc
ial m
edia
pos
ts d
rafte
d,
info
grap
hics
/vid
eos/
quot
epic
scr
eate
d, G
OV
.UK
pag
es c
reat
ed,
cons
ulta
tion
page
cre
ated
, ad
cop y
pro
vide
d fo
r pai
d fo
r act
ivity
, tra
cked
lin
ks c
reat
ed w
hich
sho
uld
be s
epar
ated
by
chan
nel e
.g. ‘
4 Tw
itter
in
fogr
aphi
cs c
reat
ed’ i
f rel
evan
t)
pre-
enga
gem
ent (
influ
ence
r/par
tner
s id
entif
ied
and
enga
ged,
dig
ital
tool
kits
sha
red,
OG
Ds
digi
tal t
eam
s br
iefe
d)If
utili
sing
pai
d fo
r act
ivity
, sep
arat
e it
from
org
anic
inpu
t met
rics.
Exte
rnal
affa
irsE
xam
ples
incl
ude:
pl
anni
ng (s
take
hold
er m
appi
ng, s
take
hold
er m
anag
emen
t pla
n, c
all l
ist
prep
ared
)
cont
ent c
reat
ion
(tool
kits
, pac
ks, c
ase
stud
ies
prod
uced
, dra
fted
emai
ls,
call
scrip
t dra
fted,
lette
r dra
fted)
even
ts o
rgan
ised
(ven
ue s
ecur
ed, a
ttend
ees
invi
ted,
roun
dtab
les)
pr
e-en
gage
men
t act
ivity
(sta
keho
lder
s id
entif
ied,
OG
D e
xter
nal a
ffairs
te
am b
riefe
d, s
take
hold
ers
brie
fed)
Inte
rnal
Exa
mpl
es in
clud
e:
plan
ning
(stra
tegi
c na
rrat
ive
draf
ted)
ev
ents
org
anis
ed (s
taff
brie
fing,
dro
p in
s, s
tand
ups
, web
chat
)
cont
ent c
reat
ed (b
log/
new
s st
ory
draf
ted,
scr
een
info
grap
hics
de
velo
ped,
vid
eo c
reat
ed, c
ase
stud
y dr
afte
d, in
trane
t ban
ner c
reat
ed)
pr
e-en
gage
men
t (st
akeh
olde
rs b
riefe
d, s
urve
y is
sued
)
Com
mun
icat
ions
, par
tner
ship
s an
d go
vern
ance
Like
you
r obj
ectiv
es, t
ry to
be
as d
etai
led
as p
ossi
ble
whe
n de
scrib
ing
you
audi
ence
. Thi
s co
uld
be in
term
s of
age
, sec
tor,
geog
raph
ic lo
catio
n et
c. T
his
will
mak
e it
easi
er
whe
n ev
alua
ting
whe
ther
you
reac
hed
this
aud
ienc
e th
roug
h yo
ur a
ctiv
ity.
Gol
den
exam
ple:
Ins
tead
of s
ayin
g ‘g
ener
al p
ublic
’, ar
e yo
u ac
tual
ly a
imin
g fo
r ‘18
-24
year
old
s’ o
r ‘18
-24
year
old
s w
orki
ng in
a p
artic
ular
sec
tor’?
This
gui
de w
ill ta
ke y
ou th
roug
h w
hat y
ou
coul
d in
clud
e fo
r eac
h se
ctio
n in
an
eval
uatio
n re
port
for a
low
cos
t/no
cost
act
ivity
. Not
all
met
rics
will
be
rele
vant
to e
very
type
of a
ctiv
ity
–ju
st u
se th
e on
es a
pplic
able
to y
our w
ork.
Fo
r fur
ther
gui
danc
e co
ntac
t the
insi
ght a
nd
eval
uatio
n te
am.
Example dashboard5
5 The full example of this template, and a blank version for you to populate can be found at: www.gcs.civilservice.gov.uk//guidance/evaluation/
20
Evaluation Framework 2.0
Calculating return on investment (ROI)GCS recommends using the following five step process:
1. Objectives
These should be focused on quantifiable behavioural outcomes (such as the number of direct foreign investments generated or the number of teachers recruited).
2. Baseline
Establish the status quo or expectation for the metrics in question if we do nothing.
3. Trend
A forecast of how the baseline will naturally move over the period of measurement. There has been an 8% reduction in adult smoking rate over the last 15 years, so we expect that the next year would see a 0.5% reduction, all other things being equal.
4. Isolation
Exclude or disaggregate other factors that will affect the outcome you are measuring to make sure that the change observed has been caused by the campaign. Recruitment campaigns will normally want to allow for the underlying rate of employment. Communication activity that is accompanied by a tax or legislative change should try to apportion the total observed affect between these different methods of government policy implementation.
5. Externalities
Account for any consequential or collateral effects of your campaign, positive and negative. For example, reducing theft offences may have positive consequences for the insurance industry (reducing their costs and therefore premium levels generally) and negative effects on other types of crime (which some criminals may take up as a substitute).
Assumptions: please note that assumptions are not just acceptable in calculating ROI, but in many cases will be necessary. Assumptions should be clearly identified, reasonable and, where possible, justified. Part of post-campaign evaluation will involve refining assumptions and considering their validity.
21
Evaluation Framework 2.0
Return on investment: a worked exampleThe Department for Health and Social Care are running a campaign costing £3 million to reduce the volume of inappropriate A&E attendances for low- urgency cases. The campaign aims to divert people to a GP surgery where they can be better and more efficiently handled. (Please note this is a fictitious example to demonstrate the process and level of detail expected only).
1. Objectives. The campaign will reduce the number of inappropriate NHS England A&E attendances by 3% or 85,000 in 2018/19 compared to the 2017/18 baseline of 2,865,377.
2. Baseline. The appropriate baseline for comparison here is the previous 12 months of operational data or observations. In 2017/18 there were 23,878,145 A&E attendances in England. 12% of these were found to be ‘inappropriate’ (did not require A&E attendance as could have been handled by a GP or pharmacist). The baseline for inappropriate attendances is 2,865,377 (23,878,145 × 0.12).
3. Trend. Over the past three years we have seen a steady increase in A&E attendances of 2% year-on-year, driven by population growth and other factors. We can forecast 24,355,708 attendances (23,878,145 x 1.02) in 2018/19. The rate of inappropriate attendance has remained broadly constant at 12%. The trend-adjusted baseline for inappropriate attendances is therefore 2,922,685 (24,355,708 × 0.12).
4. Isolation. The NHS is also starting to provide and promote out-of-hours GP surgery appointments. The rate of inappropriate A&E attendance is 4.5 percentage points higher than average at times when GP surgeries are not currently open. We assume that the new offer of out-of-hours service by GPs will reduce the total number of inappropriate attendances by 2.25 percentage points (half of the total observed affect because this only affects half the hours in day). We expect this to independently reduce the number of inappropriate attendances observed by 65,760 (2,922,685 × 0.0225) to 2,856,925.
5. Externalities. Aside from the direct cost benefit of optimising points of treatment across NHS frontline services (in the conclusion), there are indirect benefits or positive externalities that should be considered in this case.
Inappropriate attendances rarely have to be treated, so there will be a negligible cost for this, and operational overheads will remain as a fixed cost. A reduction in inappropriate A&E attendance of 3% is approximately equivalent to 3% uplift in staffing resource (which can be redeployed to urgent cases). The total annual cost of A&E operation is £2.7 billion and staffing makes up 30% of this, which is equal to £810 million. 3% of £810 million is equal to £24.3 million. The indirect benefit of optimising A&E attendance, or the effective ‘opportunity cost’ of not optimising staff resource, is £24.3 million.
22
Evaluation Framework 2.0
Conclusion:The average cost of an A&E attendance is £148. The average cost of a GP appointment is £46. Therefore every potential A&E attendance that is redirected to a GP reflects a saving to the NHS of £102 (£148 - £46).
If 85,000 cases are redirected in this way the health service overall will be £8,670,000 (£102 × 85,000) better off.
The positive externalities generated also create £24.3 million of value for the public sector and society.
The total benefit of this campaign, or return on investment, will be £32.97 million. For every £1 spent on this campaign, society will be £11 better off. This is commonly expressed as a ratio, in this case 11:1.
These results can be validated after the campaign has run by comparing the actual number of inappropriate A&E attendances with the isolated trend-adjusted baseline of 2,856,925.
23
Evaluation Framework 2.0
Measuring and managing reputationThe reputation of an organisation is now well established as an indicator of organisational success. Positive reputations are associated with supportive behaviours from stakeholders, while negative reputations are associated with less support or even hostile responses from stakeholders. Despite good evidence on this link, there is still much confusion about how to best measure and manage reputation, with many seemingly competing models and approaches.
In this guide we build upon more than 20 years of research of the John Madejski Centre for Reputation to provide a simple guide to measuring and managing reputation.5 Measuring reputation is a good form of organisational listening and can help to demonstrate performance and guide strategic action. The guide will conclude by providing an integrated reputation management framework to use when considering reputation management issues.
We suggest that practitioners should ask three questions when considering current and future reputation management projects:
• Reputation with whom?
• Reputation for what?
• Reputation for what purpose?
Reputation with whom?Organisations have different reputations with different groups and individuals. Although all people’s perceptions may be informative, it is important to consider which stakeholders are most critical.
To choose meaningfully it is vital to consider what the purpose of your organisation is and which stakeholders it serves. When considering stakeholders it is important to consider if they are vulnerable or powerful, have demands that are legitimate (socially accepted and expected) or are time critical. A common issue facing many organisations is that urgent issues often take priority over important ones, and attention and resource is given to stakeholders that are making the most noise (often through social media).
5 This draws on a number of key publications and adapt the models and approaches presented within them to the context of reputation management in the public sector. Key publications include: Money, K., Saraeva, A., Garnelo-Gomez, I., Pain, S. and Hillenbrand, C. (2017) ‘Corporate reputation past and future: a review and integration of existing literature and a framework for future research’, Corporate Reputation Review, 20 (3-4)Ghobadian, A., Money, K. and Hillenbrand, C. (2015) ‘Corporate responsibility research: past – present – future’, Group and Organisation Management, 40 (3), pages 271-294Money, K., Hillenbrand, C., Henseler, J. and da Camara, N. (2013) ‘Exploring unanticipated consequences of strategy amongst stakeholder segments: the case of a European Revenue Service’, Long Range Planning, 45 (5-6)Money, K., Hillenbrand, C., Hunter, I. and Money, A.G. (2012) ‘Modelling bi-directional research: a fresh approach to stakeholder theory’, Journal of Strategy and Management, 5 (1)Money, K. and Hillenbrand, C. (2006) ‘Using reputation measurement to create value: an analysis and integration of existing measures’, Journal of General Management, 32 (1)MacMillan, K., Money, K., Downing, S. and Hillenbrand, C. (2004) ‘Giving your organisation SPIRIT: an overview and call to action for directors on issues of corporate governance, corporate reputation and corporate responsibility’, Journal of General Management, 30 (2)
24
Evaluation Framework 2.0
It is therefore important to consider aspects of organisational purpose, stakeholder need, and legitimacy before responding to stakeholders and in managing and measuring reputation. At its best, choosing to measure reputation with a particular group can help to give them a voice, allow your organisation to listen, and to guide and justify the actions of your organisation.
Reputation for what?Reputation is defined as a perception of character. But in practice reputation is often measured as an aggregate of stakeholders’ trust, admiration and respect for an organisation. This is sometimes referred to as emotional appeal.
This aggregate measure of reputation is appealing because it allows organisations that are different (e.g. the armed services and Amazon) to be compared in the extent that they are trusted. There are many other measures of reputation that link to factors such as:
• organisational characteristics (e.g. products, leadership, financial performance)
• relationships (e.g. customer service, listening, the appropriate use of power)
• third party influence (e.g. if important third parties recommend your organisation)
However, they can more usefully be defined and measured as the causes of reputation that make stakeholders trust, admire or respect your organisation (see Figure 1).
Measuring such factors and causally linking them to reputation through multivariate statistical analysis is important in helping to
develop a theory of change. For example, if good service experience is found to be a key factor driving trust in the HMRC, more so than other factors, it would seem reasonable to focus activities on service if it was cost effective.
Reputation for what purpose?In reputational terms it is useful to answer this question in terms of what stakeholder behaviour or attitude you are seeking to maintain or change through the activities of your organisation.
These behaviours can be usefully considered to be consequences of reputation. Essentially, if your organisation is trying to influence a particular behaviour. what is it trying to cause? This could include engaging in activities such as volunteering, paying taxes or engaging in a healthier lifestyle.
You may ask yourself the question “What is my organisation trying to cause?” It is likely that you will identify different stakeholder behaviours in different contexts. The more specifically you can define a behaviour (time, place, duration), the more reliably you will be able to predict it. In this way it is important to consider the three types of behaviour outlined in the guide: start (e.g. recruitment, volunteering), stop (e.g. smoking, speeding, drinking) and maintain (current positive behaviours such as sharing information or paying tax).
As a general rule maintaining behaviours is easier than starting or stopping behaviours, which can be more difficult if it involves breaking long established habits.
MeasurementIt is important to measure not only reputation, but also its causes and
25
Evaluation Framework 2.0
consequences in a way that allows for robust statistical linkages between these factors, e.g. through multivariate techniques such as regression.
In practical terms this often means using established measurement scales that are on five or seven points (see footnote five for sample scale items that can be used). In this way you will be able to identify which stakeholder experiences link to reputation and its associated consequences.
It is also important to include measures that benchmark your organisation against other organisations, especially in terms of aspects of trust, admiration and respect. Historical comparisons to other organisations in terms of the start, stop, and maintain criteria may also be useful.
26
Evaluation Framework 2.0
Cau
ses
Str
ateg
ic a
ctio
ns(O
utpu
ts)
Sta
keho
ld e
xper
ienc
es/
ob
serv
atio
nsS
take
hold
fee
ling
s/b
elie
fsS
take
hold
beh
avio
urs/
inte
ntio
ns/e
nd s
tate
s
Go
od
will
/int
ang
ible
ass
ets
(Out
take
s)K
ey p
erfo
rman
ce in
dic
ato
rs(O
utco
mes
)
Rep
utat
ion
Co
nseq
uenc
es
Mai
ntai
ning
: beh
avio
urs
/in
tent
ions
/ e
nd s
tate
s(e
.g. c
omm
itmen
t, co
mpl
ianc
e,
turn
over
, eng
agem
ent,
wel
lbei
ng,
satis
fact
ion)
Func
tio
nal d
rive
rse.
g. p
rodu
ct q
ualit
y, w
orkp
lace
en
viro
nmen
t, fin
anci
al s
ound
ness
, co
rpor
ate
soci
al re
spon
sibi
lity)
See
Fo
mbr
un a
nd o
ther
s, 2
015;
Wal
sh a
nd
othe
rs, 2
009
Rep
utat
ion
as c
og
niti
veJu
dgem
ent a
nd d
iffer
entia
tor(e
.g.
corp
orat
e pe
rson
ality
dim
ensi
ons
such
as
mod
ern,
trad
ition
al, c
hic)
See
Dav
is a
nd
othe
rs, 2
001;
Mon
ey a
nd o
ther
s, 2
012
Rep
utat
ion
as e
mo
tio
nal
judg
emen
t and
com
para
tive
conc
ept(e
.g.
emot
iona
l app
eal,
trus
t, ad
mira
tion,
re
spec
t) S
ee F
ombr
un a
nd o
ther
s, 2
015
Feed
bac
k lo
op
of
lear
ning
(inp
uts)
Res
earc
h in
form
atio
n, c
alcu
late
d re
turn
on
inve
stm
ent,
linki
ng
to fi
nanc
ial i
ndic
ator
s, s
take
hold
er e
xpec
tatio
ns
Rel
atio
nal d
rive
rs(e
.g. b
enefi
ts, c
omm
unic
atio
ns
(incl
udin
g lis
teni
ng, u
se o
r ab
use
of
pow
er, s
hare
d va
lues
) See
Mac
mil
lan
and
othe
rs, 2
004;
Mon
ey a
nd
othe
rs, 2
012
Mo
tiva
tio
nal d
rive
rs(e
.g. a
cqui
ring
mat
eria
l wea
lth a
nd s
tatu
s, d
efen
ding
so
met
hing
of v
alue
, bon
ding
with
ot
hers
, lea
rnin
g an
d ha
ving
pur
pose
)S
ee L
awre
nce
and
Noh
ria, 2
004;
M
oney
and
Pai
n, 2
016)
Thi
rd p
arty
infl
uenc
e d
rive
rs(e
.g. a
dver
tisin
g, p
ublic
rela
tions
, pee
r-to
-pee
r in
fluen
ce, w
ord
of m
outh
) See
Ebe
rle a
nd
othe
rs, 2
013;
Li a
nd o
ther
s, 2
013;
Dys
on a
nd
Mon
ey, 2
017)
Sto
pp
ing
: beh
avio
urs
/int
enti
ons
/
end
sta
tes
(e.g
. ove
r-co
nsum
ptio
n, a
ntis
ocia
l be
havi
our,
self
harm
ing
beha
viou
r)
Sta
rtin
g: b
ehav
iour
s /i
nten
tio
ns /
en
d s
tate
s(e
.g. c
o-op
erat
ing,
con
sum
ing,
en
gagi
ng, p
ro-s
ocia
l beh
avio
urs,
re
silie
nce,
wel
lbei
ng, n
ew re
latio
nshi
ps)
27
Evaluation Framework 2.0
Introduction to the government Data Ethics FrameworkThe Data Ethics Framework7 guides the design of appropriate data use in government and the wider public sector. This guidance is aimed at anyone working directly or indirectly with data in government, including data practitioners (statisticians, analysts and data scientists), policymakers, operational staff and those helping produce data-informed insight.
1. Start with a clear user need. Using data in more innovative ways has the potential to transform how government works. We must always be clear about what we are trying to achieve for users, both citizens and civil servants.
2. Be aware of relevant legislation and codes of practice. You must have an understanding of the relevant laws and codes of practice that relate to the use of data. When in doubt, you should consult relevant experts.
3. Use data that is proportionate to the user need. The use of data must be proportionate to the user need. You must use the minimum data necessary to achieve the desired outcome.
4. Understand the limitations of the data. Data used to inform policy and service design in government must be well understood. It is essential to consider the limitations of data when assessing if it is appropriate to use it for a user need.
5. Use robust practices and work within your skillset. Insights from new technology are only as good as the data and practices used to create them. You must work within your skillset recognising where you do not have the skills or experience to use a particular approach or tool to a high standard.
6. Make your work transparent and be accountable. You should be as open as possible about the tools, data and algorithms you used to conduct your work. This allows other researchers to scrutinise your findings and citizens to understand the new types of work we are doing.
7. Embed data use responsibly. It is essential that there is a plan to make sure insights from data are used responsibly. This means those teams understand how findings and data models should be used and monitored with a robust evaluation plan.
7 Written by Dr Sarah Gates, Data Policy, DCMS. Sarah can be contacted at [email protected] for further information.
28
Evaluation Framework 2.0
DELIVERING WORLD-CLASS COMMUNICATIONS
28
Evaluation Framework 2.0
Library of further resources
The Government Communication Service (GCS) provides a series of guides, frameworks and tools to support government communicators in their work.
All these are available on gcs.civilservice.gov.uk
Handbook
Campaign highlights
Campaign Solutions Framework
Communications Services Framework
Collaboration Toolkit
Communications and behaviour change
Competency framework
Delivering excellence in partnership
Design102
Diversity and inclusion strategy
Modern media operations
Modern Communications Operating Model (MCOM)
OASIS campaign guide
Professional standards
Propriety guidance
Recruitment guidance
Recommended reading list
7 trends in leading-edge communications
GDS Social Media Playbook
Style guide
The IC Space
Previous Government Communication Plans
GC
S M
od
elO
rgan
isat
ion/
po
licy
ob
ject
ives
Co
mm
unic
atio
n O
bje
ctiv
es
Cam
pai
gn e
valu
atio
n an
d fu
rthe
r in
sigh
t to
info
rm fu
ture
pla
nnin
gg
On-
goin
g in
sigh
t to
info
rm fu
ture
pla
nnin
gg
Sta
ges
:In
put
sO
utp
uts
Out
take
sO
utco
mes
Org
anis
atio
nal
Out
com
es
Thin
gs y
ou
need
to
do
trac
k an
d/o
r ac
hiev
e
Wha
t d
o yo
u d
o b
efor
e an
d d
urin
g th
e ac
tivity
(e.g
):
·p
lann
ing
·p
rep
arat
ion
·p
re-t
estin
g ·
pro
duc
tion
Wha
t is
del
iver
ed/t
arge
t au
die
nce
reac
hed
(i.e
..):
·d
istr
ibut
ion
·ex
po
sure
·re
cep
tion
·re
ach
Wha
t th
e ta
rget
aud
ienc
e th
ink,
feel
or
cons
ider
doi
ng
(i.e.
):
·aw
aren
ess
·un
der
stan
din
g ·
inte
rest
·en
gag
emen
t ·
pre
fere
nce
·su
pp
ort
The
resu
lt of
you
r act
ivity
on
the
targ
et a
udie
nce
(e.g
.):
·im
pac
t ·
influ
ence
·
effe
cts:
-at
titud
e -
beha
viou
r
The
quan
tifiab
le im
pact
on
the
orga
nisa
tion
goal
s/ K
PIs
(e
.g.):
·re
venu
e ·
cost
s re
duc
tion
·co
mp
lyin
g a
ctio
ns ·
rete
ntio
n, ·
rep
utat
ion.
Met
rics
&
Mile
sto
nes
Sel
ect
the
right
met
rics
from
the
fram
ewor
k to
hel
p y
ou m
easu
re a
nd e
valu
ate
the
per
form
ance
of
your
in
tegr
ated
com
mun
icat
ion
activ
ities
.S
elec
t the
righ
t bus
ines
s K
PIs
to
trac
k pe
rform
ance
of
your
In
tegr
ated
com
mun
icat
ion
activ
ities
ag
ains
t you
r org
anis
atio
ns g
oals
.M
etho
ds
Use
a m
ix o
f qua
litat
ive
& q
ualit
ativ
e m
etho
ds (e
.g. s
urve
ys, i
nter
view
feed
back
, foc
us g
roup
s, s
ocia
l med
ia a
naly
tics,
trac
king
etc
).