Population Health Management
NHS England and NHS Improvement
PHM Analyst Academy26th February
09:00 -09:30 Arrival and Registration
09:30 – 09:45 Welcome & introduction to the day (Mentimeter) M Mohammed, SU and J Dellar, PHE
09:45 – 10:30 An apple a day: Logic models & Qual Methods P Mason, SU
10:30 – 10:45 (Ex-ante) Design stage evaluations / impact assessments vs
(Ex-post) summative evaluations
A Hood, SU
10:45 – 11:00 Coffee /Tea break
11:00 – 12:30 Modelling approaches (RIGHT Framework) A Hood, SU
12:30 – 13:15 Lunch
13:15 - 14:30 Experimental study designs
- Randomised controlled trials
- Cluster randomised trials
- Stepped wedge designs
P Seamer, SU
14:30 – 14:45 Coffee/Tea Break
14:45 - 15:45 Observational study designs
- Interrupted time series analysis
(exercise)
- Matched cohort design
- Synthetic controls
P Seamer, SU
15:45 - 16:00 Mentimeter questions
Review and look ahead to Session 4
J Dellar, PHE and M A Mohammed, SU
midlandsphmacademy.nhs.uk
Population Health Management
NHS England and NHS Improvement
What’s your theory of change?
A brief introduction to logic models
Paul Mason
Every action has a causal theory
5
“If we do x, then we’ll get y”
Sometimes explicit; usually implicit
“If we deliver our training package, then we will
improve the skills of care homes staff...
If staff are more skilled, then they will be more able to
cope in the event of a crisis...
If staff are more able to cope in a crisis, then there will
be fewer unplanned admissions to hospital....
If there are fewer unplanned admissions, then more
people will die in a setting of their choice. They will have
a better death; we will make better use of resources.”
‘An apple a day keeps the doctor
away…’
Giving free fruit to primary schools in order
to improve health
What’s the causal story? How do you
(logically) get from action to outcome?
Pulling out this ‘theory of change’ is essential for evaluation – and for designing initiatives
We learn via advances in theory
So the unit of analysis for evaluation
should be ‘the theory’
Results then refine, reject, (provisionally)
support the theory
Apples
Delivered
Apples
Eaten
Vitamin
Levels
Raised
Health
Outcomes
Improved
Interpretation
❌ ❌ ❌ ❌ Implementation Failure
☑ ❌ ❌ ❌Engagement Failure
(first causal link)
☑ ☑ ❌ ❌Theory Failure
(early causal link)
☑ ☑ ☑ ☑ Consistent with theory
☑ ☑ ☑/❌ ☑Theory Failure
(later causal link)
☑ ☑ ☑/❌ ☑/❌Partial Theory Failure
Works in some contexts
☑ ☑ ❌ ☑Theory Failure
(different causal path)
“An apple a day...” (Ref: Funnell & Rogers, 2013)
Logic models are one tool for representing these theories. There are different approaches; all share common elements*
* Terminology varies but basic concepts remain constant
Inputs
Resources used
Activities
Things done (measured by outputs)
Outcomes
Effects of activities
Impacts
Broader societal ‘goods’
Simple logic model for refurbishing a house (the lazy way)…
Inputs
£
Time
Activities
Source and
manage
experts
Outcomes
Improved
domestic
environment
Impacts
Increased
house value
Improved area
Inputs
£
Time
Activities
Book place to
stay, find
things to do
together
Outcomes
Reduced
stress (?)
Increased
happiness
Impacts
Improved
family
functioning
…or going on a family holiday
“If we deliver our training package, then we
will improve the skills of care homes staff...
If staff are more skilled, then they will be more
able to cope in the event of a crisis...
If staff are more able to cope in a crisis, then
there will be fewer unplanned admissions to
hospital....
If there are fewer unplanned admissions, then
more people will die in a setting of their choice.
They will have a better death; we will make
better use of resources.”
This shows the
theory that
connects activity…
…to outcomes…
…to impacts
•Impacts are the final effects that you are working towards – e.g. increased life expectancy, reduced health inequality, more sustainable services, etc
•Normally expressed at a high level. Triple / quadruple aim a useful framework
•Changes at this level only indirectly attributable to your intervention –you ‘contribute to’, rather than ‘cause’. Contextual factors a significant influence
Work back: what change do you ultimately want to see?
•These are the changes that you are trying to make / that would (logically / evidence suggests…) result from your activities
•Can be broken down into:
• Intermediate outcomes – changes in knowledge / awareness / skills / access
• Outcomes – changes in behaviour / condition / status
•Language suggesting change is therefore important: ‘reduced, increased, improved, better, worse’
Being more detailed: what outcomes do you need to achieve your impact?
•The things you do (e.g. establish apple scheme, etc)
•Measured by outputs (e.g. # people eating apples; # apples eaten)
•You don’t need to be detailed – just the main activities - the logic model is not a plan
•What ‘mechanism’ links activities to outcome? A causal ingredient?
What will you do to achieve these outcomes?
Often at this point, you’ll find
yourself working backwards
and forwards
•Should be fairly straightforward: these are the resources you have to do the things you do
•Usually measured in £
•For most programmes, cash funding is the largest element – but maybe there are in-kind inputs too, e.g. if partners have assigned staff to your programme, if you have lots of volunteers, if you are given ‘free’ facilities, etc..
And what resources will you use?
“In an Integrated Care System, NHS organisations, in partnership with local councils and others, take collective responsibility for managing resources, delivering NHS standards, and improving the health of the population they serve.”
NHS England
Inputs
Resources used
Activities
Things done (measured by outputs)
Outcomes
Effects of activities
Impacts
Broader societal ‘goods’
Take a step back and reflect on:
1: Assumptions in the model:o Practical (e.g. shows significant reliance on recruitment of…)
o Evidential (e.g. implied connection between activity x and effect y)
o Contextual (e.g. that there is no significant change in regulation of x)
Can this be used in programme planning? Is this showing risks to be managed? Would more evidence help design? What does it mean for evaluation?
2: Your overall theory of change. Policies generally use:o Sticks (beat / regulate things into place)
o Carrots (incentivise / ease the change you want)
o Sermons (eulogise and persuade)
What is the mix in your theory? Does this seem optimal given the task? If not, what is missing and can this be managed?
Parting note 1: Logic models and economic evaluation
Cost
(Economy)
Efficiency (£ per output)
Effectiveness / Benefit (£ per outcome)
Parting note 2: your model should reflect your theory, which might not be linear
Further resources / guidance
1. www.strategyunit.co.uk – search ‘logic model’ for fuller guide to using logic models
2. www.betterevaluation.org
3. HMT ‘Magenta Book’ – good all round guidance on evaluation
Population Health Management
NHS England and NHS Improvement
An incredibly brief introduction to qualitative
methods for evaluation
Paul Mason
Qualitative Vs
Quantitative?
There are many different types of qualitative method / approaches for evaluation…
Individual
interviews
Group
interviews
Focus groupObservation
Document
analysis
Participatory
approaches
Diaries
Social media
analysisQualitative
surveysCase studies
Ethnography
…all underpinned by some common purposes
Understanding things as
they are experienced by
the people involved
(context)
Exploring – what matters, to
whom, why
Explaining (e.g.) why did
events unfold in this way?
Creating (e.g.) what
should we do now?
(Not always, but…) democratising
the evaluation process
Interesting to note what is
not on this list that would
be there for quantitative
method
Choice of method depends on many factors
•The question! First and most important consideration…
•Time and resource. Qualitative methods can be expensive / time consuming
•Ethics. Accessing participants, what you might find, power dynamics (etc)
•Skills. Done well, it looks easy; but so easy to do badly. Analysis of non-quantitative data is a specialist skill too
•Stage of the evaluation process. Drawing out theory? Designing approaches (defining outcomes that matter)? Tracing process / implementation? Explaining effects? Interpreting findings / working up recommendations?
•Related point: where does this fit with quantitative methods?
Population Health Management
NHS England and NHS Improvement
Evaluation and Impact Assessments
Andy Hood
•Before implement - design stage evaluation / ex-ante modelling
•After implement - quantitative (ex-post) evaluation
When should we estimate the effect of a planned change?
31
time
Ex-ante
Ex-post
known
unknown
Intervention
or RiskOutcome
or Impact Applications
• commissioning plans• service plans• opportunity assessment• business case
• roll-out • decommission
• adjust plans / contracts• remedial action
Population Health Management
NHS England and NHS Improvement
Break
Population Health Management
NHS England and NHS Improvement
Design stage evaluationAndy Hood
What it is that makes each of these a model?
What are common characteristics of models?
y ~ x
•Models help us understand things that would otherwise be obscured by the complexity of the real world.•Associations, casual relationships and core dynamics – descriptive / explanatory
•Consequences, forecasts - predictive
•Help us understand what we need to do - prescriptive
•Models make us document our assumptions.
•Models help us test things that would be too costly / risky / impractical / unethical to try in real life.
•Models can act as guides or templates for complicated actions / developments.
Why do we need models in healthcare?
Modelling and simulation techniques for supporting healthcare decision making: a selection framework
A collaboration of 6 universities
•Cambridge Engineering Design Centre, University of Cambridge
•The School of Information Systems, Computing and Mathematics, Brunel University
•Brunel Business School, Brunel University
•The School of Management, University of Southampton
•The Information Engineering Research Group, University of Ulster
•The School of Mathematics, Cardiff University
The RIGHT Frameworkhttps://www-edc.eng.cam.ac.uk/downloads/right.pdf
“This workbook is intended to provide guidance for people who are making decisions in healthcare. It is aimed at anyone who wants to find out more about different modelling and simulation techniques –what they are, when to apply them, and what resources are required to use them. It will not only help decision makers commission more appropriate modelling work, but also assist professional modellers and business consultants to expand their modelling repertoire in order to meet the diverse needs of their clients.
The workbook is not a “how-to-do” guide to modelling and simulation, rather a “what-is-it” introductory guide. That said, the further reading section at the end of the workbook will help locate further details for each technique. The RIGHT research team would also welcome any contact regarding the applications of these techniques.”
Technique characterisation (input requirements)
RIGHT framework: technique selection table
Thinking about a specific project you are involved with and using the RIGHT resources on your tables:
• What stage of the project cycle are you currently (stuck) at?
• What type/s of output are you looking for?
• What constraints are you under - time, costs etc…
When you’ve identified a potential modelling tool, your HOMEWORK is to find about a bit more about that technique and discuss with manager/project lead how you’re going to apply it.
Have a go yourself....(5 mins)
RIGHT Modelling
Framework
Design Stage
Evaluation
•A visual aid to explore how different variables in a system are related.
•A modelling method in its own right – but can also underpin quantitative modelling methods
•Comprised of
•a set of nodes, each representing a quantity or variable
•a set of arrows indicating the influence of one quantity on another
Causal loop diagrams (directed acyclic graphs)
X Y
an increase (decrease) in X tends to cause
an increase (decrease) in Y…
… all other things being equal
Population size
Number of
people with a
mental health
problem
Smoking
prevalance
Population life
expectancy
X Y
an decrease (increase) in X tends to cause
an increase (decrease) in Y…
… all other things being equal
X Y
X influences Y but through Z (i.e. not directly)
Z
Prevalence of
cardio-
vascular
disease
Premature
deaths from
cardio-vascular
disease
Smoking
prevalence
Population life
expectancy
Prevalence of
cardio-
vascular
disease
Premature
deaths from
cardio-vascular
disease
Smoking
prevalence
Population life
expectancy
Incidence of
cancer
Premature
deaths from
cancer
hungerfood
consumptionb
a balancing loop
bank balance interestr
a reinforcing loop
•Balancing and reinforcing loops can be made up of more the 2 nodes
•A loop is
Balancing – if there are an odd number of negative arrows (- - - →)
Reinforcing – if there are no or an even number of negative arrows (- - - →)
Balancing and reinforcing loops
b
r
•Work in groups of 2 or 3
•Join the nodes to indicate influence
•Decide if/where there are balancing or reinforcing loops.
•When discussions concluded, you may want to produce final / tidy version
TIP: In total, we think there are 10 arrows and 3 (+1) loops.
Exercise – Join the dots patients who
need to be
admitted
admissionsaverage
length of stay
admission
threshold
discharge
threshold
unoccupied
beds
all beds
Exercise – Join the dots patients who
need to be
admitted
admissionsaverage
length of stay
admission
threshold
discharge
threshold
unoccupied
beds
all beds
Exercise – Join the dots patients who
need to be
admitted
admissionsaverage
length of stay
admission
threshold
discharge
threshold
unoccupied
beds
all beds
b
bb
Exercise – Join the dots patients who
need to be
admitted
admissionsaverage
length of stay
admission
threshold
discharge
threshold
unoccupied
beds
all beds
b
bb
r
Roemer’s Law
"in an insured population, a hospital bed built is a filled bed“
Shain, M; Roemer, MI (April 1959). "Hospital costs relate to the supply of beds". Modern Hospital. 92 (4): 71–3
Parkinson’s Law
"the number of patients always tends to equality with the number of beds available for them to lie in“
patients who
need to be
admitted
admissionsaverage
length of stay
admission
threshold
discharge
threshold
unoccupied
beds
all beds
b
bb
r
Last year…
•Hip replacements completed – 1,253
•Mean length of stay – 7 days
•Overnight bed occupancy – 80%
•Beds – 30
•Mean waiting time – 91 days
You speak with the service and clinical lead and they say that in 5 years time they expect…
•Demand will have increased by 12%
•Mean length of stay will fall by 14%
•It would like overnight bed occupancy to reduce to 70%
How many more/less beds will we need?
Quantitative modellingan elective inpatient hip replacement service
Current Change parameter Future
Annual admissions 1,253 +12% 1,403
Average LoS 7.0 -14% 6.0
Actual occupied bed
days8,771 8,448
Bed days with
occupancy %10,964 -10% 12,069
Beds required 30 33
Therefore, under this model, we would require an additional 3 beds for our hip replacement service
Last year…
•Hip replacements completed – 1,253
•Mean length of stay – 7 days
•Overnight bed occupancy – 80%
•Beds – 30
•Mean waiting time – 91 days
You speak with the service and clinical lead and they say that in 5 years time they expect…
•Demand will have increased by 12% for all levels of need
•24% of cases have only a moderate level of need. They would like to divert all these cases to receive a non-surgical service in another unit
•Mean length of stay for the non-moderate cases will fall by 14%
•It would like overnight bed occupancy to reduce to 70%
Quantitative modellingan elective inpatient hip replacement service
Last year…
•Hip replacements completed – 1,253•Moderate need - 301•High need - 752•Severe need - 200
•Mean length of stay – 7 days•Moderate need – 3.0 days•High need – 7.0 days•Severe need – 13.0 days
•Overnight bed occupancy – 80%
•Beds – 30
•Mean waiting time – 91 days
You speak with the service and clinical lead and they say that in 5 years time they expect…
•Demand will have increased by 12% for all levels of need
•24% of cases have only a moderate level of need. They would like to divert all these cases to receive a non-surgical service in another unit
•Mean length of stay for the non-moderate cases will fall by 14%
•It would like overnight bed occupancy to reduce to 70%
How many beds will we need?
What assumptions are needed to reach this view?
Quantitative modellingan elective inpatient hip replacement service
Current Change parameter Future
Annual admissions 1,253Shift 'moderate need'
then +12%1,066
Average LoS 7.0-14% for remaining
cases7.1
Actual occupied bed
days8,771 7,575
Bed days with
occupancy %10,964 -10% 10,821
Beds required 30 30
Therefore, under this model, we would require no change in beds for our hip replacement service
Last year…
•Hip replacements completed – 1253•Moderate need - 301•High need - 752•Severe need - 200
•Mean length of stay – 7 days•Moderate need – 3.0 days•High need – 7.0 days•Severe need – 13.0 days
•Overnight bed occupancy – 80%
•Beds – 30
•Mean waiting time – 91 days
You speak with the service and clinical lead and they say that in 5 years time they expect…
•Demand will have increased by 12% for all levels of need
•24% of cases have only a moderate level of need. They would like to divert these cases to receive a non-surgical service in another unit
•Mean length of stay for the non-moderate cases will fall by 14%
•It would like overnight bed occupancy to reduce to 70%
How many beds will we need?•What if we only manage to divert 50% of moderate cases?•What if we live with 80% occupancy?
What assumptions are needed to reach this view?
Quantitative modellingan elective inpatient hip replacement service
Current Change parameter Future
Annual admissions 1,253
Shift 50% of
'moderate need'
then +12%
1,235
Average LoS 7.0-14% for remaining
cases6.5
Actual occupied bed
days8,771 8,009
Bed days with
occupancy %10,964 No change 10,012
Beds required 30 27
Therefore, under this model, we would require 3 fewer beds in our hip replacement service
But what if the question you really need to answer was;
•If we divert moderate need patients from the beginning of year 4, how long will it take before the average waiting time falls below 84 days?
•What proportion of patients will spend more than 15 days in hospital?
•How frequently will occupancy exceed 85%?
What if the length of stay reduction is heavily dependent on the occupancy rate?
In such circumstances, basic rules-based models are not sufficient.
Two options worth considering;
•Systems dynamics modelling
•Discrete event simulation
Type Name Link
Proprietary AnyLogic https://www.anylogic.com/
Simul8 https://www.simul8.com/
Sysdea https://sysdea.com/
Open Source Insight Maker https://insightmaker.com/
Simmer (for R) https://r-simmer.org/
Time dependant
I need to know about the model variables at a number of time points –
not just the start and end state of the model and I expect the changes
to evolve non-linearly over time
⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫
Individual
I need to know about variation between patients (its not sufficient to
know the average effect for groups of patients) and I have patient level
data to populate the model.
⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫
Interactions / dynamic
What happens to one patients strongly and importantly effects what
happens to others.⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫
Stochastic
I need to know about the degree of uncertainty in the model results
(either due to randomness or uncertainty of inputs / parameters) – a
point estimate with / without sensitivity analysis is not sufficient.
⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫
Basic rule based model ⚫
Systems dynamics model ⚫ ⚫ ⚫ ⚫
Discrete event simulation model ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫ ⚫
Discrete event simulation (DES)
Models of sequences of discrete events within a system where future events are a condition of the current system state.
•e.g. patients flowing through a hospital.
Probability distributions used to describe the arrival and treatment of patients.
https://www.youtube.com/watch?v=4BHBJlzv4RA
Systems dynamics modelling (SD)
Used to model nonlinear behaviour in systems over time.
Causal loop models used to describe stocks and flows within a systems.
Calculus used to define the system.
https://www.youtube.com/watch?v=nTD1SL2qp3o
Evidence is best – but not always available. It’s often necessary to rely on expert opinion.
Establish a reference group
•One opinion is better than none, but multiple opinions are better still (https://en.wikipedia.org/wiki/Wisdom_of_the_crowd)
•Be clear what you need to know
•Provide the reference group with useful context
Ask your reference group members to provide •a central estimate (best guess) is good, but ranges are better still•Ask for best guess, low & high estimate – and use triangular distributions or ask for 90% confidence intervals•You can calibrate you reference group member’s views (https://hubbardresearch.com/publications/how-to-measure-anything-book/)
Delphi methods can be used to aggregate multiple views•Ask reference group members for views –share these and ask or second set of estimates (https://en.wikipedia.org/wiki/Delphi_method)
Practical tips on parameritising* a model?
* yes that really is a word!
Population Health Management
NHS England and NHS Improvement
Lunch
Population Health Management
NHS England and NHS Improvement
Experimental Study Designs
Paul Seamer
Population Health Management
NHS England and NHS Improvement
Break
Population Health Management
NHS England and NHS Improvement
Observational Study Designs
Paul Seamer
Population Health Management
NHS England and NHS Improvement
ReflectionsMohammed A Mohammed and Janine Dellar
Population Health Management
NHS England and NHS Improvement
Many thanks for your participation!