The Patient - Patient-Centered Outcomes Research (2021) 14:435–445 https://doi.org/10.1007/s40271-020-00476-x
ORIGINAL RESEARCH ARTICLE
Developing a Framework for Public Involvement in Mathematical and Economic Modelling: Bringing New Dynamism to Vaccination Policy Recommendations
Sophie Staniszewska1 · Edward M. Hill2 · Richard Grant3 · Peter Grove4 · Jarina Porter3 · Tinevimbo Shiri5 · Sue Tulip3 · Jane Whitehurst3 · Claire Wright6 · Samik Datta7 · Stavros Petrou8,9 · Matt Keeling2
AbstractObjectives The Mathematical and Economic Modelling for Vaccination and Immunisation Evaluation (MEMVIE) pro-gramme aimed to explore, capture and support the potential contribution of the public to mathematical and economic mod-elling, in order to identify the values that underpin public involvement (PI) in modelling and co-produce a framework that identifies the nature and type of PI in modelling and supports its implementation.Methods We established a PI Reference Group, who worked collaboratively with the academic contributors to create a deliberative knowledge space, which valued different forms of knowledge, expertise and evidence. Together, we explored the key steps of mathematical and economic methods in 21 meetings during 2015–2020. These deliberations generated rich discussion, through which we identified potential points of public contribution and the values that underpin PI in modelling. We iteratively developed a framework to guide future practice of PI in modelling.Results We present the MEMVIE Public Involvement Framework in two forms: a short form to summarise key elements, and a long form framework to provide a detailed description of each potential type of public contribution at each stage of the modelling process. At a macro level, the public can contribute to reviewing context, reviewing relevance, assessing data and justifying model choice, troubleshooting, and interpreting and reviewing outcomes and decision making. The underpinning values that drive involvement include the public contributing to the validity of the model, potentially enhancing its relevance, utility and transparency through diverse inputs, and enhancing the credibility, consistency and continuous development through scrutiny, in addition to contextualising the model within a wider societal view.Discussion and Conclusion PI in modelling is in its infancy. The MEMVIE Framework is the first attempt to identify poten-tial points of collaborative public contribution to modelling, but it requires further evaluation and refinement that we are undertaking in a subsequent study.
Extended author information available on the last page of the article
Key Points for Decision Makers
Public involvement (PI) in modelling is feasible and desirable.
The Mathematical and Economic Modelling for Vaccina-tion and Immunisation Evaluation (MEMVIE) Public Involvement Framework can enhance the quality, validity and relevance of models.
PI in modelling can contribute to the decisions policy makers have to make.
Public involvement (PI) has become increasingly embed-ded within health research, both nationally and interna-tionally. In the UK, PI is defined by INVOLVE as research that is undertaken ‘with’ or ‘by’ patients or members of the public, rather than ‘to’, ‘about’ or ‘for’ them [1]. This active involvement is different from patients or the public participating as passive research subjects in clinical tri-als or research studies with little contribution to identi-fying need, design, conduct or interpretation [2]. In this study, we use PI to refer to inputs from the public rather than patients, acknowledging these can be different [3]. Recently, the focus has been on the concept of ‘co-pro-duction’ of research between researchers and the public [4]. We drew on the concept of co-production, which we acknowledge is a rare approach in modelling. The notion of co-production is founded on a number of elements or principles that are relevant in research [5]. In identifying these, Heaton et al. [5] use the term ‘users’ to refer to the service context. For consistency, we use public con-tributors. First, in the process of co-production, public contributors are regarded as active agents and not merely passive subjects or recipients of services (or research). Second, there is greater than usual equality in the rela-tions between public contributors and professionals, with services becoming more people driven, with their knowl-edge and experience being valued on a par with that of professionals. Third, public contributors and professionals recognise that they can achieve more by working together than they can apart; both also find their relationship to be reciprocal and mutually beneficial. Fourth, public con-tributors’ increased participation transforms the ways in which public services (or research) are designed and deliv-ered, developing capacity for public contributors’ present and emerging needs to be met. Fifth, the participation of public contributors in the co-production of services (or research) is encouraged and facilitated by networks and organisations that support their involvement.
Ideally, the practice of PI is based on an understand-ing of the values that underpin it as they often reveal key motivations. These values may vary among individu-als within a team, making it important to explore them before and during a study. Gradinger et al. [6] identified a range of value systems and value clusters underpinning involvement that includes normative values focused on moral, ethical and or political concerns associated with PI in research (empowerment, rights, ethics); substantive values that focus on concerns about the consequences of PI in research (quality/relevance, validity, reliability) and process values that focus on concerns about the conduct of PI in research (respect/trust, openness and honesty).
PI has become embedded in many areas of health research, where values and different approaches have been explored. However, there are areas where PI is much less common, including complex areas such as mathematical and economic modelling. A definition of a mathematical model is a “mathematical framework representing vari-ables and their interrelationships to describe observed phenomena or predict future events”. The epidemiologi-cal component captures the health-related outcomes [7]. The health economic components of the model capture the costs and, ultimately, the cost effectiveness [8]. Mod-elling provides decision makers with the best available evidence to reach a decision [9]. In 2013, after analysis of the available cost-effectiveness models, the Joint Com-mittee for Vaccination and Immunisation (JCVI), an inde-pendent expert advisory body for vaccinations in the UK, advised that a novel vaccine to prevent meningococcal group B disease would not be cost effective at any price [10]. Stakeholders, including patient groups, submitted evidence and the models were rerun to include additional disease-specific costs and health benefits [11]. This led to the introduction of the vaccine for UK babies in 2015, although a health petition called for access for older chil-dren [12]. Scrutiny of the cost-effectiveness analysis of the MenB vaccine questioned whether cost-effectiveness methods reflected the health characteristics that the pub-lic value when prioritising health spend. The adequacy of methodology is still debated and the Government has not formally adopted the recommendations from a working group that addressed this question [13]. However, involv-ing patients and the public in all stages of health economic analysis could enhance models, improve confidence in and ultimately accelerate decision making. PI in modelling could also play a pivotal role in highlighting key areas for further research in modelling methods.
The UK spends in excess of £200 million per year on vaccines and vaccine delivery [14]. Epidemiological and economic models are important in forming judgements about the introduction of, or a change to, immunisation pro-grammes. The MEMVIE study, commissioned by the UK Department of Health in 2015, provides second-opinion modelling. Model outputs are subsequently presented to the JCVI, the key programme decision maker. While modelling can be a very effective tool, views differ about the extent to which models truly capture the entirety of a concept or a context, as they can be limited by the availability and qual-ity of data, and assumptions made. As George Box said, “essentially, all models are wrong, but some are useful” [15], reflecting a realistic view of the limits of modelling. Although Box did not mention PI, our extrapolation of Box’s view is that models can benefit from additional sources of input, such as PI, to help strengthen them. Different forms of experiential and community knowledge can complete the
437Developing a Framework for Public Involvement in Mathematical and Economic Modelling
‘world view’ that may be required to create a ‘less wrong model’. In commissioning MEMVIE, the Department of Health in the UK recognised the role the public could have in creating models that are more relevant, appropriate and acceptable from a public perspective. The investigation of the role of PI to inform such modelling was novel at the time of the commissioning of the contract by the Department of Health in 2015.
In scoping the literature, it was clear that mathematical and economic modelling has not embraced the potential of active PI, either philosophically, in terms of perceiving any value, or methodologically in practice. It became apparent that in order to embed PI within the MEMVIE study, we needed to explore where, when and how the public could contribute to modelling, understand the values that drive this involvement and develop a framework that could sup-port MEMVIE and others in their implementation of PI in modelling.
2 Aims
1. To explore and capture the potential areas of contribu-tion of the public to mathematical and economic model-ling.
2. To identify the values that underpin PI in modelling.3. To co-produce a PI framework that identifies the nature
and type of PI in modelling and supports its implementa-tion.
3 Methods
PI can be described as a form of social practice and can be informed by evidence that guides best practice. We drew on the learning from the RAPPORT study that emphasised the importance of high-quality relationships, enabled a sense of reciprocity and provided enough time for exploration and discussion [16]. As this study focused on working with pub-lic contributors as partners in research, rather than subjects of research, ethical approval was not required [17]. However, we drew on principles of good ethical practice in PI [18] and GRIPP2 to report the PI in MEMVIE [see electronic supple-mentary material (ESM 1)] [19]. We are one research team, each contributor with different skills and experiences. There are two groups; in one group, the skills are centred on aca-demic profile, and, in the other group, their skills arise from practical living experiences and a wider societal view. The two groups together focussed their attention on the research question. We have called these groups public and academic contributors.
4 Key Stages of Framework Development
We identified the following key stages needed to develop the MEMVIE framework. These were iterative rather than linear, depending on what emerged from discussions.
Recruitment of public contributors We used an informal interview process, intended to identify individuals who had relevant health knowledge, experience and learning, potentially gained through different channels (includ-ing previous PI projects and their community roles). We drew on the Warwick Medical School Public Involvement UNTRAP (Universities/User Teaching and Research Action Partnership) Network, which includes patient con-tributors from diverse backgrounds who are not academ-ics. The informal interview explored how potential public contributors felt about working in a complex area such as modelling. Public contributors needed to be comfortable and confident when asking questions about methods. There would be many sessions focused on learning about the methods and potentially several moments of uncertainty when people realised it may take a while to understand a method. They also needed to be comfortable with having no prescribed route map as we embarked on our explora-tory journey together.
Establishing a deliberative knowledge space The devel-opment of the Public Reference Group meetings (which included both the public and academic contributors) reflected the concept of a deliberative knowledge space [20, 21]. Creating a deliberative knowledge space allowed participants to consider and discuss relevant information from multiple points of view, drawing on their own differ-ent expertise, knowledge and values as the lens through which to view a disease or a method [21]. The first few meetings were exploratory as we discussed how we would examine PI in modelling. As we progressed, the meet-ings started to follow a more logical path through learning (about the diseases and modelling methods) and discus-sion. As public contributors built up their knowledge, they gradually developed their capacity to identify potential PI contributions [20]. Discussions were facilitated (by the PI lead, SS) to ensure that a diversity of views from people with different perspectives were included and that every-one’s contribution was valued and listened to. We ensured meeting places were relaxing and conducive to discussion, moving venue during the project to ensure a higher-quality environment.
Understanding modelling methods An essential activ-ity, which happened in almost every PI Reference Group meeting, was the explanation given by researchers about the methods. Presentation of methods ensured that public contributors felt comfortable asking questions and discuss-ing modelling aspects, enabling them to reflect upon and
438 S. Staniszewska et al.
explore the methods presented. Specific sessions on par-ticular topics were often repeated as learning could be lost in between meetings, as meetings took place between four and five times per year over the 5-year period.
4.1 Emergence of the Framework
Creating a deliberative knowledge space enabled possible areas of public contribution to modelling to emerge, as public contributors were seen as active agents, with their knowledge equally valued and with power shared in how decisions were made in what to include in the framework. Both sides found their relationship to be reciprocal and mutually beneficial. We recognised that the early discus-sions of PI in mathematical and economic modelling rep-resented a ‘liminal’ space, an in-between place or a place of transition, where there were no conventions of what PI in modelling might look like but one where we valued the potentially transformative contribution a co-production approach could bring [21]. This liminality initially felt uncertain and very uncomfortable for both public mem-bers and the research team, and facilitation by the PI lead
(SS) was important in reassuring everyone that this feeling was normal. As meetings progressed, a dialogue emerged, within which it was possible to identify potential areas of public contribution to modelling.
We captured ideas by recording meetings rather than taking detailed notes. This provided the group with a more detailed record of their discussion if we wished to use these. However, these recordings did not provide research data in the conventional sense from which thematic analy-sis could be derived. Rather, from these discussions and reflections, we started to build up a list of areas of possi-ble contribution, which gradually built up into categories that eventually formed the framework. The PI lead (SS) captured the categories identified in the discussion, which we reviewed at each meeting, with the public contributors refining and discussing new categories with the academic contributors each time. This process took 5 years, partly because of the co-production element, but was necessary to create the appropriate knowledge space for exploration in such a complex liminal space [20, 21]. As the categories in the framework became clearer, the public contributors suggested we group the categories, mapped to the key
Fig. 1 The left-hand side of the framework represents the epidemio-logical components of modelling, while the middle section repre-sents the health economic components of the modelling. Each phase is labelled Ep (epidemiological) or Ec (economic), corresponding to the associated section of the long-form framework (Appendix 2). The framework should be read across horizontally, i.e. Ep1, then Ec1,
Ep2, Ec2, and so on. The legend on the right-hand side denotes the five types of PI contribution identified by the reference group. Pat-terned backgrounds illustrate the regions of the model pathway where the associated PI contribution type predominately features. PI public involvement, QALYs quality-adjusted life-years, GP General Practi-tioner
439Developing a Framework for Public Involvement in Mathematical and Economic Modelling
steps of modelling, to facilitate the use of the framework by the modelling community (Fig. 1).
4.2 Identification of Values
Values are important in PI as they can motivate individuals in different ways [6]. As we conceptualised PI in modelling as taking place within a liminal space, one that had not yet been explored and where things were unclear [21], we ena-bled the values to emerge during the discussions to ensure they were the result of a co-production process. Once we had identified the values, we then compared them with those identified by Gradinger and colleagues as a form of reflec-tion rather than validation [6].
5 Results
We structure our study outcomes as follows. First, we pre-sent the characteristics of the Public Reference Group con-tributors and the academic contributors, with some quotes about process. Second, we report the values that our meet-ings identified as underpinning PI in modelling. Third, we consider key aspects of practical framework implementation. Fourth, we present the short form of the framework, which summarises the main components to enable an overview (Fig. 1). The final section presents the long-form framework, a detailed description of potential PI contribution (reported in full in ESM 2). It is important to emphasise that, at this stage, the MEMVIE framework has not undergone a full evaluation, although one is being conducted within our follow-up study, namely MEMVIEER (Mathematical and Economic Modelling for Vaccination and Immunisation Evaluation and Emergency Response). Therefore, at present it is unclear which framework elements are vital to PI in modelling and which are optional. Prior to full evaluation, we would encourage teams to select and evaluate the aspects they feel are of most relevance to their specific project.
5.1 Characteristics of the Public Involvement Reference Group Contributors and the Academic Contributors
The public contributors included eight individuals at the start of the project, with a further public contributor joining in the third year. They were recruited through the Warwick University UNTRAP Network and all contributors were members of that network. They represented a range of ages, ethnicities and cultural backgrounds, different careers/pro-fessions/employment, a range of ‘family’ experiences, dif-ferent lifestyles, different health issues within family/friend-ships, and of different ages and from different generations. Five public contributors remained at the end of the study.
These individuals had a range of expertise that included a biology background, health visiting, nursing, community development, and experience of patient organisation input into the cost-effectiveness modelling of meningitis vaccines. There was also a diversity in ages and caring responsibili-ties. The academic contributors included two mathematical and economic modellers at any one time, an economist and a social scientist. All contributors were comfortable with complex topics, a deliberative approach and being within an often uncertain, liminal space [21]. The Principal Investiga-tor recruited, supported, facilitated and organised PI involve-ment, capturing outputs and reporting impact.
5.2 Values
PI Reference Group discussions identified a range of val-ues associated with the process of modelling and the use of model outputs (Table 1). There was recognition that the public bring forms of knowledge (such as experiences in the wider context of life) that supplement evidence from peer-reviewed papers. The expertise and broader societal perspec-tives that the public bring can add to the validity of a model, potentially enhancing its relevance, utility and transparency through diverse inputs. The concept of co-production to guide PI in modelling was recognised as relevant and impor-tant. The Reference Group identified the potential for PI to contribute to the credibility and consistency of the model through this scrutiny and continuous improvement through development. Additionally, the group attributed importance to contextualising the model within a wider societal view and understanding the connections to other services and con-texts. The Reference Group identified tacit values that may underpin PI in modelling and are important to acknowledge, such as the belief that vaccination is a public good.
We compared these values with those identified by Gradinger and colleagues [6]. We identified significant alignment, particularly with substantive values focusing on the consequences of PI in research, such as effective-ness and research quality, and on process system values, those concerned with conduct of PI in research. Regarding normative values, it was recognised that the output of cost-effectiveness modelling ultimately decided whether the pub-lic would have access to sometimes lifesaving interventions and that certain methods used could create ethical dilemmas. For example, placing greater emphasis on long-term health effects by using a lower discount rate can make or break whether a vaccine is deemed a cost-effective spend of public funds. A recent example of this is extending the availabil-ity of the HPV vaccine to boys in the UK, which was only deemed cost effective when longer-term health benefits were considered. While the group are not decision makers, they wanted to ensure that a full picture of the costs and benefits of interventions are considered. These may not always be
440 S. Staniszewska et al.
Tabl
e 1
A v
alue
s fra
mew
ork
for P
I in
mod
ellin
g
Valu
esW
hy w
e ar
e in
volv
ing
publ
ic in
mod
ellin
gFu
rther
com
men
ts
Nat
ure
of k
now
ledg
eTh
e pu
blic
brin
g di
ffere
nt fo
rms o
f kno
wle
dge,
exp
ertis
e an
d so
ciet
al p
er-
spec
tives
that
can
com
plem
ent t
he m
odel
ling
team
kno
wle
dge
The
publ
ic p
rovi
de a
diff
eren
t per
spec
tive,
incl
udin
g ex
perie
ntia
l kno
wle
dge
base
d on
spec
ific
expe
rtise
, a p
atie
nt c
arer
vie
w o
f the
dis
ease
and
com
mu-
nity
kno
wle
dge,
and
a w
ider
wor
ld v
iew.
The
pub
lic c
an a
lso
inte
rpre
t and
ev
alua
te re
sear
ch e
vide
nce
in d
iffer
ent w
ays.
As s
uch,
they
offe
r a u
niqu
e an
d im
porta
nt c
ontri
butio
n to
mod
ellin
gC
olla
bora
ting
thro
ugh
co-p
rodu
ctio
nTh
e pr
inci
ple
of c
o-pr
oduc
tion
is im
porta
nt in
cre
atin
g a
mod
el. I
t rec
og-
nise
s tha
t the
pub
lic a
re e
qual
pla
yers
in th
e te
amC
o-pr
oduc
tion
repr
esen
ts a
key
way
in w
hich
PI i
s ena
cted
. NIH
R IN
VO
LVE
co-p
rodu
ctio
n gu
idan
ce (H
icke
y et
al.
[4])
can
info
rm th
e m
odel
ling
pro-
cess
. Thi
s has
key
impl
icat
ions
for t
he b
alan
ce o
f pow
er w
ithin
the
team
Rele
vanc
eTo
hel
p en
sure
the
mod
el is
mor
e re
leva
nt to
pat
ient
s and
wid
er so
ciet
yM
odel
s sho
uld
be re
leva
nt fo
r the
pop
ulat
ion
they
aim
to se
rve.
Tha
t bei
ng
the
case
, out
puts
from
mod
ellin
g an
alys
is c
an p
rovi
de a
wor
thw
hile
con
tri-
butio
n to
the
evid
ence
bas
eC
redi
bilit
y th
roug
h sc
rutin
yTo
eng
age
with
, hea
r the
pub
lic v
oice
and
resp
ond
to it
. Thi
s can
enh
ance
th
e ex
tent
to w
hich
the
mod
el is
trus
ted
and
belie
ved
in a
nd im
prov
es
cred
ibili
ty
The
publ
ic h
ave
bette
r con
fiden
ce a
nd tr
ust i
n th
e de
velo
pmen
t and
app
li-ca
tion
of th
e m
odel
, par
tly b
ecau
se it
has
bee
n de
velo
ped
with
pub
lic
invo
lvem
ent a
nd sc
rutin
yC
onsi
stenc
y th
roug
h sc
rutin
yTh
e m
odel
nee
ds to
be
cons
isten
t in
term
s of h
ow d
ecis
ions
abo
ut v
ari-
able
s, in
puts
and
qua
lity
of d
ata
are
mad
eA
dditi
onal
scru
tiny
of v
aria
bles
, dat
a an
d as
sum
ptio
ns le
ads t
o in
crea
sed
cons
isten
cy w
ith re
gard
to th
e m
odel
inpu
tsVa
lidity
It is
impo
rtant
to re
cogn
ise
that
val
idity
nee
ds to
be
judg
ed w
ithin
the
con-
strai
nts o
f the
avai
labl
e da
ta th
at c
an b
e di
scus
sed
and
eval
uate
d. P
I can
he
lp d
evel
op c
omm
unity
val
idity
Com
mun
ity v
alid
ity c
an b
e th
ough
t of i
n th
e w
ider
soci
etal
con
text
, alo
ng-
side
oth
er fo
rms o
f val
idity
. The
mod
el th
us re
pres
ents
the
clos
est w
e ca
n ge
t to
real
ity a
nd it
is c
ogen
t and
soun
d fro
m a
pub
lic v
iew.
Pub
lic in
volv
e-m
ent m
akes
the
flaw
s and
lim
itatio
ns o
f a m
odel
mor
e tra
nspa
rent
Util
ityTh
e m
odel
nee
ds to
hav
e pr
actic
al a
pplic
abili
ty. P
I can
hel
p en
sure
this
The
publ
ic h
elps
ens
ure
that
the
oper
atio
nalis
atio
n of
the
mod
el is
bas
ed o
n an
d cr
eate
s pra
ctic
al a
pplic
abili
ty. I
n ot
her w
ords
, it w
ould
wor
k in
the
real
w
orld
Con
text
ualis
atio
nTh
e m
odel
nee
ds to
sit w
ithin
a w
ider
soci
etal
con
text
and
pub
lic in
volv
e-m
ent a
llow
s us t
o id
entif
y th
ese
soci
etal
fact
ors.
They
are
impo
rtant
be
caus
e th
ey m
ay c
ontri
bute
to th
e co
nten
t val
idity
of t
he m
odel
The
mod
el a
ccou
nts f
or w
ider
soci
al fa
ctor
s tha
t may
impa
ct o
n its
succ
ess
by b
eing
awar
e of
them
. The
pub
lic p
rese
nt a
diff
eren
t vie
w T
he re
sear
ch-
ers a
nd m
odel
lers
can
tap
into
the
wid
er li
ved
expe
rienc
eC
onne
ctio
nTh
e m
odel
nee
ds to
con
side
r con
nect
ions
to o
ther
con
text
s and
serv
ices
The
mod
el si
ts w
ithin
a w
ider
con
text
, lin
ked
to so
ciet
al in
stitu
tions
such
as
scho
ols,
heal
th a
nd so
cial
car
e se
rvic
es a
nd th
e po
litic
al c
onte
xt o
f ser
vice
pr
ovis
ion.
The
pub
lic c
ould
hav
e a
pote
ntia
l rol
e in
vac
cine
pro
gram
me
impl
emen
tatio
nTr
ansp
aren
cyTh
e m
odel
nee
ds to
be
trans
pare
nt in
mea
ning
, in
its d
evel
opm
ent,
and
in
its c
hoic
e an
d us
e of
dat
a/in
puts
Publ
ic in
volv
emen
t can
hel
p m
ake
the
proc
ess o
f mod
el d
evel
opm
ent
trans
pare
nt a
nd a
id u
nder
stan
ding
of t
he re
lativ
e str
engt
hs a
nd w
eakn
esse
s of
the
mod
el. P
I als
o pr
ovid
es st
akeh
olde
r inv
olve
men
t in
som
ethi
ng th
at
dire
ctly
affe
cts t
hem
and
thei
r chi
ldre
nD
iver
sity
of i
nput
We
can
enha
nce
the
dive
rsity
of p
ublic
vie
ws t
hat s
hape
the
mod
elW
e re
cogn
ise
that
div
ersi
ty in
vie
ws i
s im
porta
nt a
nd a
ctiv
e pu
blic
invo
lve-
men
t is a
pur
pose
ful a
ctiv
ity th
at a
ims t
o cr
eate
div
ersi
ty in
thin
king
and
he
lps m
itiga
te a
gain
st co
nser
vativ
e or
est
ablis
hed
view
s and
way
s of w
ork-
ing
or ‘g
roup
thin
k’C
ontin
uous
impr
ovem
ent
Each
mod
el n
eeds
con
tinuo
us im
prov
emen
t dur
ing
its d
evel
opm
ent a
nd
ther
eafte
rB
y ad
optin
g a
mod
el o
f con
tinuo
us im
prov
emen
t, it
is p
ossi
ble
to id
entif
y an
d co
mm
ent o
n ga
ps. P
I can
pro
vide
sugg
estio
ns fo
r pol
icy
mak
ers
conc
erni
ng w
ider
rele
vanc
e, fu
ture
info
rmat
ion
and
data
requ
irem
ents
and
id
eas f
or fu
ture
dev
elop
men
t
441Developing a Framework for Public Involvement in Mathematical and Economic Modelling
well represented under the current cost-effectiveness rules, particularly for marginalised groups.
5.3 Evaluation of Process
The use of the concept of a deliberative knowledge space and think-aloud techniques provided an open, non-prescrip-tive dialogue that enabled ideas and thoughts to emerge. The public contributors were able to challenge the data, the basis for the collection of data and the interpretation of that data, thinking outside of the box in a safe space where modellers could rework their thinking.
One of our public contributors stated that the process “was more akin to embarking on a long journey on a com-plex ship, with no captain, with an unknown destination with no operating manual and a very mixed and unfamiliar crew who were trying to figure out how this ship works, how each crew member fits in whilst trying to write the outline of an operating manual for future passengers on a similar journey”.
The meetings enabled thematic development over time as the Reference Group contributors worked with the aca-demic contributors on continuous iterations of the emerging framework. We held 21 MEMVIE meetings over 5 years, each lasting between 2–3 h, with email contact in-between with the group commenting on documents.
Two quotes illustrating the public contributors’ journey and a third view from a researcher are presented in Table 2.
5.4 Implementing the Mathematical and Economic Modelling for Vaccination and Immunisation Evaluation (MEMVIE) Framework
As PI in modelling for developing policy vaccination recom-mendations is in its early stages, the modelling community may need support to implement the MEMVIE Framework. The Public Reference Group suggested adopting a ‘What, How, When, Where, with Whom’ approach that a modelling team could use to plan for PI in their study (see ESM 3). A brief summary is provided below.
We encourage teams to define PI before they consider how to undertake it. A suggestion of co-production as an approach to PI is made [4, 5]. However, as co-production is difficult with some aspects of modelling, it may be that teams need to recognise the current limits to co-production and perhaps draw on the ACTIVE Framework for patient and public involvement (PPI) in systematic reviews [22], which recognises a range of levels of public input in differ-ent scenarios.
We encourage teams to use the concept of a delibera-tive knowledge space in which to hold discussion, to ‘think aloud’ and to draw on standards [20, 21] (such as UK INVOLVE standards) [23] and the developing PI evidence PI
pub
lic in
volv
emen
t, N
IHR
Nat
iona
l Ins
titut
e fo
r Hea
lth R
esea
rch
Publ
ic in
volv
emen
t in
rese
arch
is o
ften
unde
rpin
ned
by a
set o
f val
ues,
ofte
n ta
cit.
It is
impo
rtant
to id
entif
y th
e fu
ll ra
nge
of v
alue
s of r
elev
ance
at t
he st
art o
f a P
IPI a
ctiv
ity, i
n or
der t
o id
entif
y po
ssib
le m
otiv
atio
n an
d pe
rcep
tion
and
how
thes
e m
ight
var
y ac
ross
a te
am. I
n th
is ta
ble,
we
prov
ide
a su
mm
ary
of th
e va
lues
that
wer
e id
entifi
ed d
urin
g th
e M
EMV
IE st
udy
and
whi
ch u
nder
-pi
nned
our
wor
k
Tabl
e 1
(con
tinue
d)
Valu
esW
hy w
e ar
e in
volv
ing
publ
ic in
mod
ellin
gFu
rther
com
men
ts
Ethi
cal v
alue
sTh
e et
hics
asp
ects
of a
mod
el n
eed
to b
e re
cogn
ised
and
und
ersto
odW
e re
cogn
ise
ther
e m
ay b
e di
ffere
nt e
thic
al is
sues
that
nee
d to
be
cons
id-
ered
. Cer
tain
met
hods
may
als
o cr
eate
eth
ical
dile
mm
as. W
e al
so n
eed
to
cons
ider
wid
e et
hica
l iss
ues s
uch
as d
oing
no
harm
Reco
gnis
ing
taci
t ass
umpt
ions
Expl
ore
the
taci
t ass
umpt
ions
of t
he e
ntire
team
We
reco
gnis
e th
at p
ublic
mem
bers
invo
lved
in a
mod
ellin
g pr
ojec
t are
like
ly
to su
ppor
t the
con
cept
of v
acci
natio
n as
a p
ublic
goo
d
442 S. Staniszewska et al.
base to guide practice. The need for teams (particularly researchers) to undergo training in PI was seen as vital to developing appropriate skills that support the process. Facil-itation of PI is key and a PI lead role is considered essential. Similarly, the need for the public contributors to undergo training in the basic methods of health economic evaluation was also seen as crucial. Furthermore, the values that are relevant to a particular project need to be identified (Table 1) as they will be embedded through discussion and can help understand differences in view and perspective.
We encourage teams to be flexible according to the needs of the project, but recognise that by definition the ‘complete modelling team’ should include both researchers and public contributors. Evaluating process and outcome is crucial. One method proposed by our Reference Group was an impact log to capture key contributions after each meeting or key inter-action. The log then forms the basis of a narrative model, which can provide a qualitative ‘story of model develop-ment’ in a way that replicates the quantitative elements with narrative, capturing key decisions, key assumptions, values and other aspects of discussion that public contributors feel are important. This qualitative, narrative model could be reported alongside the quantitative model.
5.5 The Short‑Form MEMVIE Framework
Due to the complexity and length of the MEMVIE frame-work, the Reference Group identified the need for a short form, or summary, of the framework (see Fig. 1).
The framework breaks down the epidemiological and economic components of the modelling. Each phase is labelled Ep (epidemiological) or Ec (economic), correspond-ing to the associated section of the long-form framework shown in ESM 2, and should be read across horizontally. It is important to emphasise that this figure reflects one repre-sentation of modelling in order to demonstrate the potential for PI. Modelling can be flexible and iterative as a method. Through Reference Group discussion, we identified a range of potential macro areas of contribution for PI in modelling.
The five overarching types of PI contribution (right-hand column, Fig. 1) are Reviewing Context and Relevance, Assessing Data and Justifying Model Choice, Troubleshoot-ing, Interpreting and Reviewing Outcomes and Decision Making. Each of the components of these overarching areas is described in detail in the long-form framework.
5.6 The Long‑Form MEMVIE Framework
Each of these model steps described in the short-form frame-work have a more detailed set of potential PI contributions presented in the long form, explained in ESM 2. The pub-lic Reference Group felt it was important to differentiate between the academic modelling team activity, the nature of the PI contribution and the components that support PI input. As with the short form, the framework should be read horizontally across, i.e. Ep1, then Ec1, Ep2, Ec2, and so on. We provide here an overview of the five overarching types of PI contribution.
The initial stage is concerned with reviewing the context and relevance of the model and determining the state struc-ture; that is, determining what should go into the model, with inputs from disease-specific experts and through refer-ence to relevant literature. A contribution that the MEMVIE Public Reference Group made at this stage, when consider-ing the transmission of pneumococcal bacteria among the population, was identifying the potential for uncertainty because the study used to identify mixing patterns in the population was conducted 10 years previously and mixing patterns may have changed due to broader societal change [24]. The modelling team had not identified this aspect at that point.
Within assessing data and justifying model choice activ-ity (PI contribution), a farther-reaching benefit of PI is the Reference Group querying the robustness of a tool or ques-tionnaires that have been used to collect data. For example, our Reference Group requested to see the NATSAL survey questionnaire [25] to judge the quality of data that MEMVIE was drawing on. The troubleshooting activity provided an
Table 2 Public contributor and academic contributor views
(Public contributor) “We had no picture of what PI involvement would look like, and no route map to guide our journey. We also had no idea of the constraints we might need to work within. The researchers we were working with had no idea of what they wanted from us, or even if we could add any anything useful to their model. The first year really felt like working in the dark, not even sure what we were trying to achieve”
(Public contributor) “A key breakthrough was the pictorial representation of the Epidemiological and Economic Model shown in Fig 1. For the first time we understood modelling as a process and provided a framework through which we could start to organise and structure our contribu-tions”
(Academic contributor) "When I joined midway through the duration of the MEMVIE project, I had not had any previous exposure to public involvement as part of the research process. I found it extremely beneficial to have an additional forum to describe our modelling process, dis-cuss model assumptions and examine data. From my perspective, being given the opportunity to convey the work to public members through reasoned discourse, ensured justification of modelling aspects, aiding model integrity and validity. In addition, public involvement generated broader discussion surrounding data curation and data collection (such as questionnaire content), producing recommendations that can be used to inform future developments"
443Developing a Framework for Public Involvement in Mathematical and Economic Modelling
opportunity for the modellers to host a series of thinking-aloud sessions while the models were under construction. Any preliminary findings from the models were presented to the group and the influence of some of the factors thought to be contributing to uncertainty were discussed in more detail.
The interpreting outcomes stage allowed the modellers to present their models and associated outcomes in their entirety. Both the economic and epidemiological models were merged together and it was an opportunity for the group to review the models and consider the sensitivity of the results in relation to the parameters used, and to critique the model assumptions. Another step for the PI group to consider was the impact of ‘rules’, such as the time horizon considered for the evaluation and the discount rate used, and how these rules affect the cost effectiveness of the interven-tion under consideration.
The final stage was decision making. While this phase would most likely be enacted by an independent committee of experts, PI contribution at this stage could be in the form of a report detailing the findings of the group, which could be considered alongside the model or presented by public representation on vaccination committees themselves.
6 Discussion and Conclusion
The MEMVIE study has been a key source of policy advice to the Department of Health in England, creating math-ematical and economic models in a number of clinical areas, including human papillomavirus (HPV), pneumo-coccal and influenza. We used each of these projects as opportunities to build PI into modelling, a highly techni-cal process often hidden from public or patient scrutiny, and yet with the potential to impact their health provision and health status in significant ways. We were not able to identify an existing framework for collaborative PI in modelling, and therefore we believe MEMVIE is the first international framework for PI in vaccination modelling with the intention of informing policy. The development of the MEMVIE framework demonstrates the feasibility of involving the public in the deliberation and decisions that inform models, providing that the context for implementa-tion is appropriate.
Through the MEMVIE study, we were also able to iden-tify the enablers that can support PI in modelling; for exam-ple, the creation of a deliberative knowledge space. Under-standing each other’s values was vital to recognising the different perspectives public contributors can bring, which need to be explored at the start of and during the process [6]. Planning for the implementation of PI in a study was seen as important and emerged from recognising that PI in
modelling is novel and requires careful facilitation from a PI lead to support the entire team of academic and public contributors working together.
The idea of a deliberative knowledge space enabled the technical and practical–experiential forms of knowledge to come together in a context where effective contribu-tions could be made, informed by many deliberative ses-sions learning about methods. Through the deliberations, we identified the need to develop a narrative model that could complement the quantitative model. This provided the ‘story’ of model development, making accessible the often inaccessible equations and graphs. We will refine the idea of a narrative model in our next study, MEM-VIEER, using a reflection and impact log to build up our understanding of contribution, guided by the MEMVIE framework.
As the primary beneficiaries of healthcare modelling, the public have a moral right to be involved as key valued and impacted stakeholders. Public contribution has the poten-tial to ‘ground’ models in the reality of lived experience and behaviour and add to its interpretation and understand-ing. As Williams and Popay [26] say of the public “they are themselves bearers of considerable lay knowledge of their own life worlds and through their own ‘stocks of knowl-edge at hand’, were skilled interpreters and translators of the ‘external’ evidence provided by professional experts in dialogue with whom they could share in the decision making about what was to be done”.
Through deliberative discussion, challenging assumptions and enhancing the richness of the dialogue, model validity is enhanced, with models created that sensibly account for the complexity of a disease and of societal response. Extending the approach by Box and Draper [15], in the context of PI in mathematical and economic modelling, PI could ‘help models be less wrong and more useful’.
While the MEMVIE modelling study provides useful policy advice, the MEMVIE PPI framework has yet to be evaluated or used in practice. This is the intention in our next study, MEMVIEER, but other modelling teams may wish to use all or part of the framework to guide their work. We rec-ognise that PI in modelling is not a linear process, with some phases undertaken in a different order to that illustrated by Fig. 1. We fully expect the framework to evolve over time as teams implement it and report their framework refine-ments. We would encourage them to publish their outcomes to strengthen the PI evidence base in modelling, building high-quality practice that enhances the validity and utility of models, making models more accessible and account-able to the public, and creating significant patient and pub-lic health benefits in addition to furthering public trust in recommendations.
444 S. Staniszewska et al.
Supplementary Information The online version of this article (https ://doi.org/10.1007/s4027 1-020-00476 -x) contains supplementary mate-rial, which is available to authorized users.
Declarations
Conflict of interest CW is employed by the Meningitis Research Foun-dation, which has received grants and conference sponsorship from Pfizer, GlaxoSmithKline, and Sanofi Pasteur. Sophie Staniszewska, Edward M. Hill, Richard Grant, Peter Grove, Jarina Porter, Tinevimbo Shiri, Sue Tulip, Jane Whitehurst, Samik Datta, Stavros Petrou, and Matt Keeling these authors declare no conflict of interest.
Funding MEMVIE was funded by the UK Department of Health and the National Institute for Health Research (NIHR). SS, EMH, SP and MK are supported by the NIHR (Policy Research Programme, Infec-tious Disease Dynamic Modelling in Health Protection, Grant number 027/0089). SP also receives support as an NIHR Senior Investigator. SS is part funded by the NIHR Applied Research Collaboration (ARC) West Midlands, the NIHR Health Protection Research Unit (HPRU) Gastrointestinal Infections, and the NIHR HPRU Genomics and Ena-bling data. This report is independent research funded by the NIHR (Policy Research Programme, Infectious Disease Dynamic Modelling in Health Protection, 027/0089). The views expressed are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Ethics approval Not applicable.
Consent to publication All authors confirm their consent for publica-tion.
Data availability statement Data sharing was not applicable to this article as no datasets were generated or analysed during this study.
Code availability Not applicable.
Author’s contribution SS: PPI lead for the MEMVIE study. Organised the PPI meetings, facilitated the PPI discussions as knowledge spaces for co-production, contributed expertise on PPI evidence, co-produced the PPI framework, lead author and lead on reviewer responses. EMH: epidemiological modeller who participated in PPI meetings, contrib-uting expertise regarding mathematical modelling and intersections with PPI, co-produced the PPI framework, conceptualised Fig. 1, and contributed to writing of the manuscript and responding to reviewer comments. RG: public contributor, contributed experiential expertise to the PPI discussions, co-produced the framework and supporting appen-dices, and contributed to writing of the manuscript and responding to reviewer comments. PG: policy maker who commissioned the study, supported the exploration of PPI and the capture of its contribution, attended PPI discussions, and contributed to conceptual development, writing of the manuscript and responding to reviewer comments. JP: public contributor, contributed experiential expertise to the PPI dis-cussions, co-produced the framework and supporting appendices, and contributed to writing of the manuscript and responding to reviewer comments. TS: health economic who participated in PPI meetings, con-tributed expertise regarding economic modelling and intersection with PPI, co-produced the PPI framework, and contributed to writing of the manuscript and responding to reviewer comments. ST: public con-tributor, contributed experiential expertise to the PPI discussions, co-produced the framework and supporting appendices, and contributed to writing of the manuscript and responding to reviewer comments. JW: public contributor, contributed experiential expertise to the PPI dis-
cussions, co-produced the framework and supporting appendices, and contributed to writing of the manuscript and responding to reviewer comments. CW: patient group representative, contributed experiential expertise to the PPI discussions, contributed a broader charity perspec-tive to wider economic discussions, co-produced the framework and supporting appendices, and contributed to writing of the manuscript and responding to reviewer comments. SD: epidemiological model-ler who participated in PPI meetings, contributing expertise regard-ing mathematical modelling and intersections with PPI, co-produced the PPI framework, and contributed to writing of the manuscript and responding to reviewer comments. SP: health economic who partici-pated in PPI meetings, contributed expertise regarding economic mod-elling and intersection with PPI, co-produced the PPI framework, and contributed to writing of the manuscript and responding to reviewer comments. MK: epidemiological modeller who participated in PPI meetings, contributing expertise regarding mathematical modelling and intersections with PPI, co-produced the PPI framework, and con-tributed to writing of the manuscript and responding to reviewer com-ments.
Open Access This article is licensed under a Creative Commons Attri-bution-Non-commercial 4.0 International License, which permits any non-commercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Com-mons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regula-tion or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creat iveco mmons .org/licen ses/by-nc/4.0/.
References
1. INVOLVE. Briefing notes for researchers: Public involvement in NHS, public health and social care. Eastleigh: INVOLVE; 2012.
2. National Institute for Health Research. Going the Extra Mile: Improving the nation’s health and wellbeing through public involvement in research. The final report and recommendations to the Director General Research and Development/Chief Medi-cal Officer Department of Health of the ‘Breaking Boundaries’ strategic review of public involvement in the National Institute for Health Research. National Institute for Health Research; 2015.
3. Stafinski T, Street J, Menon D. OP114 the public’s role in under-standing the value of health technologies. Int J Technol Assess Health Care. 2018;34(Suppl 1):43–4. https ://doi.org/10.1017/S0266 46231 80014 11.
4. Hickey G, Brearley S, Coldham T, Denegri S, Green G, Stan-iszewska S, et al. Guidance on co-producing a research project. Southampton: INVOLVE; 2018.
5. Heaton J, Day J, Britten N. Collaborative research and the co-production of knowledge for practice: an illustrative case study. Implement Sci. 2015;11:20. https ://doi.org/10.1186/s1301 2-016-0383-9.
6. Gradinger F, Britten N, Wyatt K, Froggatt K, Gibson A, Jacoby A, et al. Values associated with public involvement in health and social care research: a narrative review. Health Expect. 2013;18(5):661–75.
7. Eykhoff P. System identification; parameter and state estimation. Chester: Wiley; 1974.
445Developing a Framework for Public Involvement in Mathematical and Economic Modelling
8. Petrou S, Gray A. Economic evaluation using decision analyti-cal modelling: design, conduct, analysis, and reporting. BMJ. 2011;342:d1766.
9. WHO. Cost effectiveness and strategic planning (WHO-CHOICE). 2020. https ://www.who.int/choic e/en/. Accessed 15 Feb 2020.
10. JCVI interim position statement on the use of Bexsero meningo-coccal B vaccine in the UK. An interim position statement about the use of Bexsero meningococcal B vaccine in the UK released by JCVI. Department of Health and Social Care; 24 July 2013. https ://www.gov.uk/gover nment /publi catio ns/jcvi-inter im-posit ion-state ment-on-the-use-of-bexse ro-menin gococ cal-b-vacci ne-in-the-uk. Accessed 27 Jul 2020.
11. Christensen H, Trotter CL, Hickman M, Edmunds WJ. Re-evalu-ating cost effectiveness of universal meningitis vaccination (Bex-sero) in England: modelling study. BMJ. 2014;349:g5725.
12. Petition: Give the meningitis B vaccine to ALL children, not just newborn babies. https ://petit ion.parli ament .uk/archi ved/petit ions/10807 2. Accessed 27 Jul 2020.
13. Cost-effectiveness methodology for Immunisation Programmes and Procurements (CEMIPP) The government’s decision and summary of consultation responses. Department of Health and social Care; 2019. https ://asset s.publi shing .servi ce.gov.uk/gover nment /uploa ds/syste m/uploa ds/attac hment _data/file/80785 6/CEMIP P_Consu ltati on_Respo nse_1.pdf. Accessed 27 July 2020.
14. Datta S, Pink J, Medley G, Petrou S, Staniszewska S, Underwood M, et al. Assessing the cost-effectiveness of vaccination strategies for adolescent girls and boys. BMC Infect Dis. 2019;19:552. https ://doi.org/10.1186/s1287 9-019-4108-y.
15. Box GEP, Draper NR. Empirical model-building and response surfaces. Wiley; 1987. p 424. ISBN 0471810339.
16. Wilson P, Mathie E, Poland F, Keenan J, Howe A, Munday D, et al. How embedded is public involvement in mainstream health research in England a decade after policy implementation? A real-ist evaluation. J Health Serv Res Policy. 2018;23(2):98–106.
17. Health Research Authority/INVOLVE (2016). Public involvement in research and research ethics committee review. www.invo.org.
18. Pandya-Wood R, Barron DS, Elliott J. A framework for public involvement at the design stage of NHS health and social care research: time to develop ethically conscious standards. Res Involv Engagem. 2017;3:6.
19. Staniszewska S, Brett J, Simera I, Seers K, Mockford C, Goodlad S, et al. GRIPP2 reporting checklist: tools to improve reporting of patient and public involvement in research. BMJ. 2017;358:j3453. https ://doi.org/10.1136/bmj.j3453 .
20. National Consumer Council and INVOLVE. Deliberative public engagement: Nine Principles. London: National Consumer Coun-cil; 2008.
21. Elliott E, Williams G. Developing public sociology through health impact assessment. Sociol Health Illn. 2008;30(7):1101–16.
22. Campbell P, Struthers C, Synnot A, Nunn J, Hill S, Goodare H, et al. Development of the ACTIVE framework to describe stake-holder involvement in systematic reviews. J Health Serv Res Policy. 2019;24(4):245–55.
23. NIHR. National Standards for Public Involvement. Southamp-ton. NIHR; 2018. https ://www.invo.org.uk/wp-conte nt/uploa ds/2019/02/71110 _A4_Publi c_Invol vemen t_Stand ards_v4_WEB.pdf. Accessed 27 Jul 2020.
24. Mossong J, Hens N, Jit M, Beutels P, Auranen K, Mikolajczyk R, et al. Social contacts and mixing patterns relevant to the spread of infectious diseases. PLoS Med. 2008;5(3):e74. https ://doi.org/10.1371/journ al.pmed.00500 74.
25. Natsal. The National Survey of Sexual Attitudes and Lifestyles. https ://www.natsa l.ac.uk/home.aspx. Accessed 27 Jul 2020.
26. Williams G, Popay J. Lay health knowledge and the concept of the lifeworld. In: Scambler G, editor. Habermas, critical theory, and health. London: Routledge; 2001.
Authors and Affiliations
Sophie Staniszewska1 · Edward M. Hill2 · Richard Grant3 · Peter Grove4 · Jarina Porter3 · Tinevimbo Shiri5 · Sue Tulip3 · Jane Whitehurst3 · Claire Wright6 · Samik Datta7 · Stavros Petrou8,9 · Matt Keeling2
1 Division of Health Sciences, Warwick Medical School, University of Warwick, Coventry CV4 7AL, UK
2 The Zeeman Institute for Systems Biology and Infectious Disease Epidemiology Research, School of Life Sciences and Mathematics Institute, University of Warwick, Coventry CV4 7AL, UK
3 Warwick Medical School, University of Warwick, Coventry CV4 7AL, UK
4 Department of Health, London, UK5 Liverpool School of Tropical Medicine, Liverpool L3 5QA,
UK
6 Meningitis Research Foundation, Bristol BS1 1LT, UK7 Population Modelling, National Institute of Water
and Atmospheric Research Ltd (NIWA), Wellington, New Zealand
8 Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford OX2 6GG, UK
9 Warwick Clinical Trials Unit, Warwick Medical School, University of Warwick, Coventry CV4 7AL, UK
