Rapid qualitative analysis v the ‘traditional approach’:

early findings and reflections

Beck Taylor, CRF Theme 1

03/05/2023

The team

Home Birth evaluation:• Beck Taylor• Sara Kenyon• Cathy Shneerson

Comparative analysis:• Ian Litchfield• Sheila Greenfield• Louise Bentham

Acknowledgements• Birmingham Women’s Hospital, particularly the

Home Birth Team• Gill Coombes• CLAHRC peer reviewers

Background: the original project• Dedicated Home Birth Service – an innovation• CLAHRC asked “to evaluate”• Identified model and its implementation evolving, not well-

defined• Designed a project based on Evaluability Assessment

methodology– Identify key documents and stakeholders – Determine the programme theory, and whether it has been

implemented as expected, and why– Define the data sources that are available – Areas for evaluation – Recommendations for changes to programme design,

monitoring and evaluation systems, and for potential future evaluation opportunities

Home Birth Exploratory Review• Rationale: model not clear and evolving, a need to capture design

and implementation process

• Aim: to describe the intended and actual service and its implementation over the course of the pilot

• Objectives:– Establish the original ‘programme theory’ – Once each year of the pilot

• Identify how the Service as it stands differs from the original design, and why

• Identify any facilitators or barriers to implementation • Establish what data are available with respect to the HBS, and

how it is being / could be gathered• Identify the areas that staff would like to focus on in order to

develop/improve the HBS

Home Birth Exploratory ReviewMethod:• Data collection:

– Semi-structured interviews with all key stakeholders willing/able to participate (not service users in this initial review) (n=21)

– Focus group with sample of community midwives (n=13)– Key service documents identified by participants (n=9)

• Analysis:– Documents and interview/FG transcripts– Descriptive findings plus recommendations

• Verification:– Member checking of findings (presented in report and discussed

face to face) with strategic team and HBT midwives• Dissemination:

– Report shared with trust, commissioners and participants– Further dissemination planned following longitudinal work

At the design stage…

We determined that timely feedback of value

• Sharing of model/lack of model quickly

• Inform ongoing development and decision-making

• Information not out of date, useful

Was there a way to analyse date and deliver findings more quickly?

Rapid analysisLittle in the literature (that we could find)

James Beebe: Rapid Assessment Process: An Introduction (2001)

Alison Hamilton. Qualitative Methods in Rapid Turn-Around Health Services Research. Presented online for the US Department of Veterans’ Affairs (2013)

Rapid Assessment Process (Beebe, 2001)

Key ideas:

• Speed, rapid turnaround and feedback, fewer resources

• Teams of researchers

• Concurrent data collection and analysis, iterative

• Researchers with less qualitative experience

• Analysis not about coding data – a different approach

• Rapid reduction of data using ‘summary templates’

• Future in-depth, more inductive work may follow RAP

Our Rapid Assessment ProcessKey ideas:

• Speed, rapid turnaround and feedback, fewer resources

• Teams of researchers

• Concurrent data collection and analysis, iterative

• Researchers with less qualitative experience

• Analysis not about coding data – a different approach

• Rapid reduction of data using ‘summary templates’

• Future in-depth, more inductive work may follow RAP

Basic approach

Summary template

Questions arising…

• How does this approach work in practice?• Does it deliver findings more quickly than ‘traditional’

qualitative research?• Does it elicit similar findings to traditional approaches?

If not, how do they differ?• What impact might any differences in findings have?• Is this approach useful?• What might the applications of this approach be?

“Can’t we do it the ‘normal’ way and see how the two compare?”

(Thank you Dr Kenyon!)

Comparative analysis project

• Independent, blinded researcher (Ian Litchfield)

• Thematic analysis, Framework method

• Input from second researcher, Louise Bentham

Comparative analysis projectTwo research questions

Is there a difference in researcher time requirements?Researcher time logTotal hoursCommentary on how this time was used

Are findings different?Independent scrutiny before cross-team data sharing (SG)Comparison of findings generated under key research question

headings (numerical and qualitative)

Impressions so far: time

• RA seems to deliver outputs more rapidly (25-50% less time)

• However, to compare accurately we need both projects to reach the report/recommendations stage, as processes not in parallel

Impressions so far: findings• So far only scoping comparison of RA key

headings and TA themes by SG

• As with time, need full TA ‘report’ to compare accurately, including interpretation and recommendations

• RA seems to identify same key issues as TA

• TA seems to offer more granularity

Comparing the two teams

Rapid Analysis (Beck and Cathy)

Thematic Analysis (Ian and Louise)

Embedded in the field No connection to team or contextBT collected the data Did not collect dataUsing RA for first time – learning time, need to avoid usual practice

Experienced in TA – no method to ‘learn’, doing what comes naturally

Informal reflection day to day No space for informal reflectionMain focus of work over short period Project ‘squeezed in’ among other

commitmentsExperienced qualitative researchers More experienced qualitative

researchers

Important we acknowledge the differences

ReflectionsThe process• Requires time discipline for researchers used to a

traditional approach.• Uncomfortable at times spending so little time with data,

tension with what we feel to be academically rigorous.• Did not use summary templates for all data – switched to

direct entry into matrix. Any advantage to using separate template?

• How would this work for novice researchers? Existing skills definitely useful. Would they get the same results?

• How would this work in larger teams? Would it take longer to synthesise data?

Reflections - applications• Not a replacement for more in-depth approaches

• Advantages and disadvantages – it’s about the context

• Useful for – Rapid findings where time of the essence– Identifying key priorities for action (breadth not depth)– Identifying areas for more in-depth scrutiny

Intended output

• Methodology paper

• Comparing process, time taken and findings between the two approaches

• Key limitations in our approach but ? interesting to editors due to novelty and potential impact/relevance

Questions and comments• Impressions from the group

• Would this be of interest to others?

• What are the key aspects to focus on?

• Is there anything we can do to strengthen the work?

• Where should we aim our outputs?