© NICE 2019. All rights reserved. Subject to Notice of rights.

1 of 26

NATIONAL INSTITUTE FOR HEALTH AND CARE

EXCELLENCE

MEDICINES AND TECHNOLOGIES PROGRAMME

Process and methods statement for the production of NICE

IAPT assessment briefings (IABs)

March 2019

1 Introduction

1.1 Introduction to process and methods statement

1.1.1 This process and methods statement provides an overview of the key stages

and principles used for developing NICE improving access to psychological

therapies (IAPT) assessment briefings (IABs). The statement details the

process applied for topic selection of digitally enabled therapies for anxiety

disorders and depression, and the main stages for IAPT assessment briefing

development. This statement does not cover the process for evaluation in

practice or the development of the IAPT report following evaluation in practice.

This statement is designed to ensure that robust quality assured IABs are

developed for the NHS in an open, transparent and timely way.

1.2 Background to IAPT assessment briefings

1.2.1 The National Institute for Health and Care Excellence (NICE) provides national

guidance and advice to improve health and social care. Further information

about NICE and its work is available on the NICE website.

1.2.2 NICE has been commissioned by NHS England to assess selected, digitally

enabled therapies (self-study online reinforced and supported by the therapist)

for depression and anxiety using ongoing data collection to determine whether

there are improvements in service efficiency, through saving therapist time,

and whether patient outcomes are at least as good as those achieved by

NICE-recommended, non-digital therapy. The NICE and NHS England digital

© NICE 2019. All rights reserved. Subject to Notice of rights.

2 of 26

therapy technology endorsement for IAPT programme has 3 phases: selection

and assessment, developmental support and evaluation in practice.

1.2.3 Selection and assessment –The IAB will include an assessment of evidence

of effectiveness, cost and resource impact, technical standards assessment,

and content assessment conducted by clinical experts. The draft IAB will be

considered by the NICE IAPT expert panel to make a recommendation for

inclusion, or not, in the IAPT programme. Those with a positive

recommendation will be labelled ‘assessed as provisionally suitable for IAPT’.

NICE will lead this phase.

1.2.4 Evaluation in practice – NHS England will allocate suitable products to a set

of local IAPT services that express an interest in using the products and

already achieve clinical outcomes that are not markedly below expectation.

Each service will be responsible for identifying and training therapists in how

to deliver each digitally enabled therapy. Data will be collected as part of the

ongoing data collection for the IAPT programme, and will be reviewed by the

IAPT expert panel on a quarterly basis for 2 years. Analysis will be based on a

before-and-after comparison. NHS England will lead this phase.

1.3 What are IAPT assessment briefings?

1.3.1 IABs consist of a summary of content assessment, technical standards

assessment, clinical effectiveness, and cost and resource impact of a digitally

enabled therapy. The aim of an IAB is to provide information to the expert

panel who will review the IAB and decide whether an assessed digitally

enabled therapy is provisionally suitable for IAPT, needs further development,

or is not suitable for evaluation in practice. Digitally enabled therapies that are

deemed suitable will be assessed in practice in IAPT for up to 2 years. IABs

are not NICE guidance.

1.3.2 The data generated will be analysed to find out whether the technology has

equivalent or better patient outcomes compared to standard care, and has

equal or lower cost impact to standard care, is safe and improves patient

© NICE 2019. All rights reserved. Subject to Notice of rights.

3 of 26

experience. This will be used to inform a decision on whether the technology

should be routinely offered by IAPT.

2 IAPT assessment briefings

2.1 Key audiences

2.1.1 IABs are produced for:

• NICE IAPT expert panel

• IAPT service leads.

2.1.2 They may also be of interest to:

• digital therapy technology owners and developers

• patients and the public, to help inform treatment choices.

2.2 Key activities

2.2.1 The key activities involved in the production of each IAB are:

• identifying and screening the technology

• preparing a technology summary to support prioritisation and topic selection

• describing the technology and its potential use in the evidence-based

treatment pathway as part of a blended model of care

• summarising and critically appraising the available evidence

• quality-assuring available cost benefit analysis

• assessing the clinical content of the technology

• assessing the technical reliability of the technology

• recording the IAPT expert panel considerations and decision relating to

‘assessed as provisionally suitable for use in IAPT’.

2.2.2 The flowchart in figure 1 details the process.

© NICE 2019. All rights reserved. Subject to Notice of rights.

4 of 26

Figure 1: IAB development process

3 Who is involved in producing IABs?

3.1 The medicines and technologies programme

3.1.1 The medicines and technologies programme (MTP) is part of NICE’s Health

and Social Care Directorate. The MTP consists of technical, project and

administrative staff. For IABs, the team is responsible for:

• developing and reviewing processes and methods for producing IABs

• identifying potential topics for IABs in collaboration with the IAPT expert

panel

• developing and preparing draft and final IABs for publication in line with the

agreed process and standards (this includes selecting and critically

appraising the evidence)

• identifying and liaising with external specialist commentators and

technology owners to ensure the content of IABs is relevant and useful

• providing quality assurance of the content of IABs

© NICE 2019. All rights reserved. Subject to Notice of rights.

5 of 26

• ensuring timelines and processes for quality assurance are followed.

3.2 Other NICE teams

3.2.1 The MTP works closely with other teams at NICE to develop IABs. These

include:

• the public involvement programme (PIP) – providing advice on involving

patients, carers and members of the public

• guidance information services – assisting with topic selection and

conducting literature searches

• the medical technologies evaluation programme (MTEP) – ensuring there is

no conflict or overlap with published, planned or proposed NICE medtech

innovation briefings

• the guidelines team – ensuring that there is synergy with published or

planned NICE guidelines, including the review and updating of NICE

guidelines

• the communications team - communicating the programme and inviting

submissions for consideration

• the publishing team – publishing the IABs.

3.3 The NICE IAPT expert panel

3.3.1 The role of the NICE digital IAPT panel is to provide independent expertise in

the selection, assessment and prioritisation of digital technologies designed to

support psychological therapies within the IAPT programme. The members of

the panel represent core areas of expertise relevant to the development of

IABs. The panel is chaired by the National Clinical Director for Mental Health,

NHS England and members include a service user expert, a health economist,

a health service commissioner, a digital technology expert, psychiatrist,

psychologist and research and trial expert. The panel reviews the IAB to

decide if the technology is provisionally suitable for use in IAPT. The decision

is reached by consensus after taking into account content, technical

standards, evidence of effectiveness and resource impact, which are

summarised in the IAB. The panel also reviews data from practice on

© NICE 2019. All rights reserved. Subject to Notice of rights.

6 of 26

technologies assessed as provisionally suitable to decide if the digitally

enabled therapy is suitable for use in mainstream IAPT – see section 6.6.2.

3.4 NHS England

3.4.1 NHS England is the commissioner of this workstream.

3.5 External clinical assessors

3.5.1 The content assessment for each draft IAB will be done by clinical experts with

significant experience in the field of clinical psychology. They are external to

NICE and separate from the NICE IAPT expert panel.

3.6 Technology developers/owners

3.6.1 For a technology to be considered it must be notified by the technology owner

on the IAPT notification form.

3.6.2 When a topic is selected for an IAB, NICE informs the organisation who owns

the digitally enabled therapy that it intends to produce an IAB and the

expected timeframe. NICE invites the organisation to provide information to

support the production of the IAB. The technology owner will be asked to

provide technical information on an ad hoc basis.

3.6.3 The organisation is also invited to provide comments on a draft IAB within an

agreed timeframe. The organisation can comment on factual accuracy and

respond to any specific questions from NICE about the information submitted

to inform the development of the IAB.

3.6.4 The technology owner will also be asked to check facts in the IAB before

publication. This will take place after the IAPT expert panel has made its

recommendation, and after this decision has been fed back to the technology

owner and discussed as needed.

3.6.5 For digitally enabled technologies which enter the evaluation in practice

phase, the organisation will need to comply with predefined data collection

requests and this will be managed by NHS England.

© NICE 2019. All rights reserved. Subject to Notice of rights.

7 of 26

3.6.6 For technologies that are recommended as suitable for the evaluation in

practice phase, the technology owners will be expected to provide unlimited

licences at no cost. The evaluations will involve at least 200 people using the

technology and no upper limit has been set.

3.7 Selected IAPT services

3.7.1 Following the publication of an IAB, selected IAPT services will evaluate

technologies assessed as provisionally suitable. The evaluation will run for up

to 2 years in at least 2 IAPT services. Services will collect data as per the data

specification to assess the safety, clinical effectiveness, patient and therapist

experience and resource impact compared with face-to-face therapy.

3.7.2 The resulting data will be presented to the IAPT expert panel every 6 months

during the evaluation in practice phase. The panel will decide at each 6

monthly time point whether to stop or continue the evaluation.

3.7.3 At the end of the 2 year evaluation in practice phase (or when the panel are

satisfied sufficient data has been collected), the expert panel will decided

whether to recommend that the technology should be routinely offered to

people in IAPT services

4 Conflicts of interest

4.1.1 NICE staff, members of the IAPT expert panel, external clinical assessors, and

clinical experts from IAPT must comply with the NICE conflicts of interest

policy. For more information, please see the policy.

5 Topic identification, screening, prioritisation and

selection

5.1.1 NICE’s MTP is responsible for managing topic identification, prioritisation and

selection for IABs.

© NICE 2019. All rights reserved. Subject to Notice of rights.

8 of 26

5.2 Technology identification

5.2.1 Identification of digitally enabled therapies for consideration for the IAPT

assessment will consist of, but is not limited to, the following sources:

• A review of published literature to identify digital therapy technologies for

anxiety disorder and depression.

• NICE internal teams (medicines technology evaluation programme

engagement team, scientific advise programme, office for market access,

centre for clinical guidelines)

• IAPT services, invited to inform NICE of any digitally enabled therapies that

they use for anxiety disorders and depression.

• An outward communications plan to invite self-submission of digitally

enabled therapy.

• Linking to the engagement work done by MTEP. MTEP regularly meets

with companies producing digital medical technologies, which may include

technologies eligible for the IAPT programme.

• Technologies identified by the NIHR Innovations Observatory who will be

setting up a horizon scanning service for medical technologies, including

digital healthcare technologies.

5.3 Eligibility criteria

5.3.1 NICE will complete an eligibility screen for all identified technologies. The aim

of the screening is to identify those technologies that meet set eligibility criteria

(see below), including having all the elements of an intervention

recommended by NICE and that address a condition currently managed by

the IAPT programme.

5.3.2 This step is important to ensure that the technologies chosen for assessment

are suitable for evaluation in practice, and to manage demand on the

programme and the expectations of technology producers. Technologies

which meet the following eligibility criteria will progress to prioritisation:

© NICE 2019. All rights reserved. Subject to Notice of rights.

9 of 26

• The digital programme (online or an app) delivers a substantial portion of

the therapy but is designed to be used with therapist assistance.

• Designed to treat 1 of the 13 clinical conditions covered by IAPT:

depression, generalised anxiety disorder, social anxiety disorder, panic

disorder, agoraphobia, post-traumatic stress disorder, health anxiety

(hypochondriasis), specific phobia, obsessive-compulsive disorder, body

dysmorphic disorder, irritable bowel syndrome, chronic fatigue syndrome,

or medically unexplained symptoms not otherwise specified.

• The content of the treatment should mirror a NICE recommended

psychological therapy for the relevant condition. For example, in obsessive-

compulsive disorder the programme should not just be a generic CBT

intervention. It would need to include exposure and response prevention).

• Designed to support a model of care where the therapist guides the user

through the programme and regularly reviews the user’s work, clinical

outcomes and risk. To facilitate this process it is expected that the digital

programme will have inbuilt mechanisms that:

a. Support direct two-way communication between the therapist and the

service user to facilitate the effective delivery of the intervention.

b. Enable the regular collection, monitoring and reporting of outcome and

risk data to the service user and the therapist supporting the

intervention.

c. Enable the therapist to view the work that the user is doing with the

programme in order to provide personalised feedback on progress, and

direct the service user to the most relevant content, features and tools of

the programme.

• Designed to treat adults.

• Should use (or the technology owner should be prepared to add) the

outcome measures required by IAPT

• Should have at least 1 published randomised controlled trial. The expert

panel will consider the published evidence available for each technology

according to the table below:

© NICE 2019. All rights reserved. Subject to Notice of rights.

10 of 26

Level Definition Further recommendation

1 Comprehensive RCT evidence of positive effect using therapist-guided care model: trial in relevant population with IAPT standard care as comparator

Meets evidence levels

needed for eligibility.

2 RCT evidence (1 or more) suggesting positive effect in therapist-guided care model. Can include studies where comparator is no treatment (waiting list control)

Meets evidence levels

needed for eligibility.

3 RCT evidence where the technology is used in a self-help model without therapist guidance OR

RCT evidence where population in the study is not the same as expected in IAPT

Available evidence to be

considered by the expert

panel on the individual merits

of each technology.

4 No randomised studies but does have other supporting evidence, such as audit data, observational studies, or non-comparative evidence.

Available evidence to be

considered by the expert

panel on the individual merits

of each technology.

5 No available evidence at all. Not eligible.

• Must be supplied by an organisation committed to keep ownership and

responsibility to maintain and update the technology.

5.3.3 The following exclusion criteria will also be applied to the technology:

• Not designed to treat a condition covered by IAPT.

• A digital treatment which is not a psychological therapy.

• Treatment modality is not in line with NICE guidance.

• Standalone products (for examples mood diaries and leaflets).

• Specifically designed for children.

• No available evidence.

• No clear accountability for the technology.

5.4 Prioritisation

5.4.1 Eligible digitally enabled therapies will be prioritised by the IAPT expert panel

using the following criteria, on an ongoing basis:

© NICE 2019. All rights reserved. Subject to Notice of rights.

11 of 26

• Potential to increase access to psychological therapy.

− The technology is designed to target a small population currently

underserved.

− The technology is designed to manage a condition for which there are

currently limited digital therapy options.

− The technology has the potential to reach a large subpopulation.

• Evidence base.

− There are multiple trials relevant to the technology which can be made

publicly available to summarise and critically appraise.

− There are non-inferiority randomised controlled trials comparing the

technology with face-to-face treatment.

• Resource impact.

− There is evidence of a cost-consequences analysis done by the

technology owner.

− There is potential that the digitally enabled therapy will improve efficiency

compared with current management.

5.4.2 The purpose of the prioritisation criteria is to enable the panel to discuss,

consider and prioritise digitally enabled therapies which are likely to increase

access to psychological therapies. The IAPT expert panel will make the final

decision taking into account panel considerations.

5.4.3 NICE will contact the prioritised technology owners to inform them of the

selected technologies and next steps.

6 Development

6.1.1 NICE will lead on the assessment process for selected topics and coordinate

the outcome of the content assessment, technical standards assessment,

clinical effectiveness and cost and resource impact assessment, and present it

in an IAB to the IAPT expert panel.

© NICE 2019. All rights reserved. Subject to Notice of rights.

12 of 26

6.1.2 The IAB will be written by the IAPT senior technical analyst. Literature

searches will be done by NICE’s guidance information services, in line with

pre-agreed search strategies.

6.1.3 The IAB will include a description of the digital technology and its place in the

current care pathway.

Figure 2 IAB draft to publication

© NICE 2019. All rights reserved. Subject to Notice of rights.

13 of 26

6.2 Content assessment

6.2.1 The content assessment will include:

• a description of the treatment modality, in line with conditions managed by

IAPT services and NICE guidance

• a generic assessment and a condition specific content assessment (based

on an objective framework developed by clinical experts; updated from the

Department of Health publication – The competences required to deliver

effective cognitive and behavioural therapy for people with depression and

with anxiety disorders [see appendix])

• a rating against an established competency framework, from 1 (lowest) to 5

(highest). These scores will be used only as a guide for the assessors when

they reach their overall conclusions on the content of the technology.

• the use of the technology for the condition described, as used within the

blended care model recommended by the owner of the technology

• an assessment of the scalability of the digital technology.

See Section 9: Appendix A for examples of the competency frameworks used.

6.3 Technical standards assessment

6.3.1 The technical assessment will be populated based on the Health Developer

Network version 2.1 of the Digital Assessment Questions, a tool developed by

NHS Digital and currently available to developers in beta form. Technology

owners will self-assess against predefined standards for information

governance.

6.3.2 The following sections of the DAQ will be used in this evaluation:

• Clinical safety

• Data protection

• Security

• Usability and accessibility

• Interoperability

© NICE 2019. All rights reserved. Subject to Notice of rights.

14 of 26

• Technical stability

6.3.3 The self-assessment will be reviewed by an expert third party in digital

technical standards to provide quality assurance of the answers and an

outcome categorisation for each question as follows:

• no concerns identified

• concern identified and remedied

• concern identified, remediation plan is acceptable

• insufficient information provided

• concern identified and no acceptable remediation plan provided

6.5.4 A summary digital technical assessment outcome report will be produced for

each assessed technology for the NICE expert panel to review and consider.

This will include a final outcome (pass or fail), and any technical barriers which

may help the panel in their final recommendations.

6.4 Clinical effectiveness assessment

6.4.1 A full literature search will be performed for each topic, by NICE’s information

services team. The clinical evidence will be screened, summarised and

critically appraised by the IAPT senior technical analyst.

6.4.2 For each identified digital technology NICE will:

• Define standard PICO criteria and develop a research protocol taking into

account the eligibility criteria.

• Request literature reviews to be done by NICE information services team.

The search strategy will be logged.

• Sift results and order and review full papers. The studies that are the best

available quality with outcomes most relevant to the NHS will be selected.

Randomised controlled trials are usually judged to be the best quality

evidence to assess the effectiveness (including cost effectiveness) of an

intervention. However, other study designs (including observational,

© NICE 2019. All rights reserved. Subject to Notice of rights.

15 of 26

experimental or qualitative) may also be used to assess effectiveness.

These may include ways of delivering services, or the experience of people

using services and how this contributes to outcomes.

• Produce a high-level summary of the evidence (such as the number of

studies and study design).

• Extract key study information and clinical-effectiveness data into evidence

summary tables.

6.5 Cost and resource impact assessment

6.5.1 The cost and resource impact section will include the cost consequence of the

digital technology and a description of the technology owner’s business model.

The costs of standard care will be described and the potential impact of the

digital technology on cost and system resources will be stated. The technology

owner will be asked to provide a short statement describing the anticipated

impact of the technology on the health and social care system. Any published

economic evidence will be summarised.

6.6 IAPT expert panel consideration of the draft IAB

6.6.1 The senior technical analyst will present the key findings of the 4 sub-

assessments and the evidence statements to the IAPT expert panel.

6.6.2 The panel have a choice of three outcomes for each digitally enabled therapy

assessed as follows:

• Option 1: The technology is not suitable for the evaluation in practice

phase. The IAB is published on the NICE website and proceeds no further

through this programme

• Option 2: The technology is provisionally suitable for the evaluation in

practice phase. The IAB is published on the NICE website. The technology

owner is directed to NHS England who provide information about next steps

• Option 3: The technology needs further development before it can be

considered for the evaluation in practice phase. The IAB is published on the

© NICE 2019. All rights reserved. Subject to Notice of rights.

16 of 26

NICE website. The technology owner is directed to NHS England who will

provide details of the next steps following completion of development work.

6.7 Timeline

6.7.1 Producing the IAB will take 14 weeks, from the date that a topic is selected by

the IAPT expert panel to publication on the NICE website.

7 Quality assurance of the IAB

7.1.1 Quality assurance of the draft IAB will be provided at several levels. Before the

draft IAB is presented to the IAPT expert panel, the IAB will receive:

• senior technical review

• comment from the technology owner

• sign-off from the programme director.

7.1.2 The final IAB will then move through pre-publication stages:

• review by senior technical analyst

• pre-publication factual accuracy check by the technology owner

• review by NICE editorial team

• sign-off from programme director

• final sign-off from NICE’s publication executive.

8 Publication

8.1.1 The IAB will be published on the NICE IAPT webpage in PDF format following

an internal sign off process and Publication Executive approval.

8.1.2 If information becomes available during the production of an IAB that results in

the expert panel deciding that the technology is not eligible for inclusion in the

programme, the panel can recommend that the technology be notified to the

NICE medical technology evaluation programme to be evaluated for a

medtech innovation briefing (MIB)

© NICE 2019. All rights reserved. Subject to Notice of rights.

17 of 26

9 Appendix A

Rating form for digital therapies

Application/programme

Condition(s)/ problem areas being addressed

Main therapeutic model employed

1. MATCH TO CBT PRINCIPLES (BASED ON MAPPING OF CBT-SPECIFIC CONTENT)

Overall rating of match to CBT principles for interventions for this condition/problem area 5= content is closely mapped 3 = some but not all aspects of principles included 1 = significant proportion of basic principles not covered

Comment

2. GENERIC RATING OF THE APP/PROGRAMME

5= meets this criterion 3 = meets some but not all aspects of this criterion 1 = fails to meet this criterion

There is a clear indication of the target group for whom this app/programme is intended, and this indication is appropriate

Overall usability

There is a clear homepage/dashboard

There are ‘technical’ instructions that give clear guidance on navigation through the site

The text and content us consistently pitched at an appropriate educational level and so likely to be understood by the majority of the target audience

Overall structure facilitates a wide range of learning styles and prior assumptions

The app/programme reviews and also revisits all important aspects of the content from a number of angles so as to ensure that basic principles are well-understood (i.e. there is deliberate use of redundancy so as to account for different learning styles, accommodate clients with different prior assumptions and ‘models’ of their own condition, and different patterns of symptoms and difficulties)

© NICE 2019. All rights reserved. Subject to Notice of rights.

18 of 26

The app/programme covers issues at the right level of depth and appropriately thoroughly (and with enough redundancy to assure (potential) generalisation)

Clarity of introduction to the treatment

There is a clear description of the condition itself

There is a clear and comprehensive explanation of the rationale for treatment

There is a clear indication of conditions for which this is not an appropriate intervention

Managing expectations

The programme makes it clear what sort of results can be expected and does so in a way that will be meaningful to the individual user

Engagement

There are appropriate strategies used to engage users (particularly convincing (?real) case examples)

Adaptation to the individual user

The app/programme allows for adaptation to the specific concerns of the user

Goal setting

The app/programme (usually in conjunction with direct input from the therapist) helps the user set realistic and achievable goals that are meaningful to them and helps them review and adjust goals as the app/programme proceeds

Practice assignments

The app/programme provides a clear rationale for practice assignments/behavioural experiments and stresses their importance

The app/programme offers clear guidance to the user to help them set up appropriate homework tasks

The app/programme guides the user to help them monitor and reflect on outcomes from practice assignments and on this basis identify subsequent assignments

The app/programme (usually in conjunction with direct input from the therapist) helps users identify and reflect on any unexpected difficulties that arise when assignments are carried out, and (if required) helps them construct new assignments that address the problem that arose

Level of therapist contact (help to keep on track)

The app/programme allows users opportunities to contact the therapist (e.g. to deal with queries, uncertainties, non-improvement or deterioration)

The design of the app/programme is such that it assumes the therapist stays in contact throughout the programme (not just at the start)

The design of the app/programme allows the therapist to access the user’s written input/ indictors of progress/ worksheets/ comments and allows the therapist to provide feedback/further assistance as required.

Patient safety (adverse impacts)

There are inbuilt instructions regarding the procedure to follow if the user feels themselves to be at risk (including procedures for contacting the therapist, or emergency services)

© NICE 2019. All rights reserved. Subject to Notice of rights.

19 of 26

There are inbuilt systems that allow therapists to monitor and detect deterioration/ non-response and respond to indicators of risk

There are inbuilt systems that alert therapists to indicators of risk

Training and supervision of the therapist

Guidance on implementation of the app/programme includes explicit instruction regarding the expected level of training, supervision and support for the therapist

Guidance on implementation of the app/programme indicates that the scope of training for the therapist should cover clinical aspects of implementation (and not just training in technical implementation)

OVERALL EVALUATION AND COMMENT

© NICE 2019. All rights reserved. Subject to Notice of rights.

20 of 26

Depression: Beck model

5= content is closely mapped

3 = some but not all aspects of principles included

1 = significant proportion of basic principles not covered

General Rating

Does the app/programme implement CT in a manner that is not overly reductionistic or simplistic, but which nonetheless conveys a set of coherent principles to the user

Does the app/programme help users adapt CT models to the individual case, ensuring that the model works for the user, and not that the user is fitted to the model

Identifying and responding to risk Rating

Does the app/programme include an initial screening to identify individuals at risk of self-harm, and a process for directing them to relevant clinical services

Does the app/programme have a protocol for identifying clients whose risk of self-harm emerges during their use of the app, for alerting therapists to this risk, and for managing this risk by instituting appropriate practical strategies

Knowledge of the cognitive model of depression – taking the app/programme as a whole

Rating

Does the app/programme help users understand the concept of the “cognitive triad” (core beliefs in three areas: self, others, and future)

Does the app/programme help users identify and conceptualise common “processing biases” (e.g. arbitrary inference, selective abstraction, overgeneralisation, magnification/ minimisation etc)

Does the app/programme help users consider the role of interpersonal factors in the development/maintenance of depression, particularly the tendency to elicit reactions from others that confirm self-perceptions/beliefs and lead to self-fulfilling prophecies

Engaging users

Helping users identify/clarify concerns Rating

Does the app/programme help users identify any ‘chief complaints’ or ‘chief problems’ through which depression is manifested

Does the app/programme help users identify their “personal paradigm” (the belief system and information processing strategies through which they construe and interpret their world)

Does the app/programme help users identify the configuration of cognitive problems which contribute to the maintenance and exacerbation of target depressive symptoms

Does the app/programme help users identify the coping mechanisms they use (e.g. stress tolerance, level of functioning and capacity for introspection and self-objectivity)

Explaining the rationale for cognitive therapy Rating

Does the app/programme help users understand the rationale for a focus on cognitions and help them consider this in relation to themselves (e.g. using a recent/specific example to consider the link between the cognitive triad (how they think about themselves, their environment and their future) and feelings, motivation and behaviour)

© NICE 2019. All rights reserved. Subject to Notice of rights.

21 of 26

Does the app/programme help users identify information processing that leads to unhelpful conclusions

Does the app/programme convey the cognitive model in a way which implies that users are thinking inaccurately rather than ‘irrationally’

Establishing priorities for intervention Rating

Does the app/programme help users establish which of their depressive symptoms and associated problems they would like to prioritise for intervention

Does the app/programme help users who feel overwhelmed by problems to identify specific problems and working to identify solutions

Applying cognitive techniques

Key overarching principles Rating

Does the app/programme help users understand the relevance of links between thinking, feeling and behaving in relation to their own difficulties, usually through eliciting concrete examples

Does the app/programme help users examine/explore evidence for their beliefs and assumptions based on their own behavioural experiments

Helping users develop a model of their depression Rating

Does the app/programme help users draw their own conclusions regarding ways of understanding the mechanisms involved in the maintenance of their own depression

Does the app/programme help users discover useful information that can be used to help them gain a better level of understanding (and to help them adopt this strategy for themselves as the basis for exploring their own beliefs)

Does the app/programme help users to create doubt in place of certainty, providing them with the opportunity for re-evaluation and for new learning to occur

Does the app/programme help users develop hypotheses regarding their current situation and to generate potential solutions for themselves

Does the app/programme help users develop a range of perspectives regarding their experience (by examining evidence, considering alternatives, weighing advantages and disadvantages)

Detecting and recording key cognitions (automatic thoughts/ images) Rating

Does the app/programme define ‘cognitions’ in a way which is meaningful for, and relevant to users, and explain the concept of automatic thoughts and images

Does the app/programme help users understand general concept of links between thinking, feeling and behaving by using specific examples of distressing situations

Does the app/programme help users make links between specific environmental events and cues associated with depressive cognitions and mood

Does the app/programme help users detect automatic thoughts and images through devising specific ‘projects’ designed to help elicit and “catch” pertinent cognitions

Does the app/programme help users identify cognitions that are associated with distressing emotions, particularly the ‘hot’ cognitions most likely to have caused the relevant affect

Does the app/programme help users identify the different forms of cognitive biases which tend to support their negative thinking

Does the app/programme help users use and complete relevant written records with the aim of helping them learn to self-appraise the accuracy of their thoughts and beliefs

© NICE 2019. All rights reserved. Subject to Notice of rights.

22 of 26

Examining and reality testing automatic thoughts/ images Rating

Does the app/programme help users to find alternative solutions to problems on the basis of helping them to reconceptualise their difficulties (i.e. by examining the accuracy of the specific thoughts and underlying beliefs that operate in a given situation)

Does the app/programme help users record dysfunctional thoughts and alternative more flexible responses to these thoughts, with the aim of helping them learn to self-appraise evidence and to generate alternative less personally negative interpretations

Identification and modification of dysfunctional assumptions (“intermediate beliefs”) Rating

Does the app/programme help users identify dysfunctional assumptions, including both conditional and unconditional assumptions

Does the app/programme help users consider how assumptions can become self-fulfilling

Does the app/programme help users consider changing/discarding self-defeating assumptions by listing the advantages and disadvantages (benefits and costs) of holding on to these assumptions

Does the app/programme help users examine the long-term effectiveness of assumptions, especially when assumptions currently appear to be working in their favour in the short-term

Does the app/programme help users generate and carry out behavioural experiments which help them modify their assumptions (for example, comparing what they ‘should’ do with what they are able to do, or acting against their assumptions)

Helping users apply and test out change methods using practice assignments Rating

Does the app/programme help users think through the rationale for performing practice assignments (in terms of cognitive therapy), and to identify and problem solve any anticipated difficulties in carrying out tasks

Does the app/programme help users identify practice assignments that test-out cognitions practically, and gain experience in dealing with high levels of emotion

Helping users manage depression-specific cognitions and beliefs

Managing depression-specific cognitions and beliefs Rating

Does the app/programme help users examine and reality-test their thoughts (without assuming that all their pessimistic statements are necessarily invalid)

Does the app/programme help users use self-monitoring and behavioural experiments to deal with excessive self-blame by helping them to be more aware of the frequency and nature of specific self-criticisms, and using appropriate strategies to manage these (e.g. looking at meanings and alternative responses, objectification, learning how to use self-challenging responses)

Does the app/programme help users manage/challenge all-or-nothing thinking using appropriate strategies (e.g. looking for partial gains from adverse events, fostering self-questioning to think about all-or nothing responses)

Does the app/programme help users use reattribution techniques/ strategies to manage excessive self-blame (for example, using pie charts)

Behavioural activation Rating

Does the app/programme help users understand the reciprocal links between underactivity/ failure to engage in activities and their depressive ruminations/ preoccupations

© NICE 2019. All rights reserved. Subject to Notice of rights.

23 of 26

Does the app/programme help users use activity scheduling to help them manage behavioural symptoms such as passivity, avoidance or inertia

Does the app/programme help users revise a graded hierarchy of daily activities, in line with their response to earlier phases of activity scheduling

Does the app/programme help users set realistic expectations of their capacity to complete planned activities and to help them identify beliefs regarding their achievability

Does the app/programme help users use cognitive rehearsal to help them identify potential problems in achieving tasks

Does the app/programme help users record activities undertaken, including ratings of mastery and pleasure

Does the app/programme help users review homework and identify relevant cognitions and beliefs related both to accomplishments and to areas where they were unable to carry out planned assignments

Helping users to manage specific problems frequently associated with depression Rating

Does the app/programme help users identify anxiety symptoms which appear in the context of depression, and suggest appropriate strategies for their management

Does the app/programme help users identify strategies to manage concentration and memory problems

Does the app/programme help users overcome problems of low motivation and avoidance of constructive activity

Does the app/programme help users manage sleep disturbance (for example, education about sleep, teaching relaxation methods, ‘sleep hygiene’ methods)

Helping to guard against relapse Rating

An ability to help users who have recovered identify problematic events which have led them to become depressed or anxious in the past and explicitly to plan ways in which they can implement relevant coping strategies they have learned and access relevant resources

Overall rating based on assessment of specific content

© NICE 2019. All rights reserved. Subject to Notice of rights.

24 of 26

Panic disorder: Clark model

5= content is closely mapped 3 = some but not all aspects of principles included 1 = significant proportion of basic principles not covered

Conveying knowledge of the CBT model Rating

Does the app explain the cognitive model of panic, specifically:

• the importance of a tendency to misinterpret bodily sensations (usually those associated with normal anxiety responses) in a catastrophic manner (i.e. as an indication of an immediately impending physical or mental catastrophe

• hypervigilance (especially to introceptive cues) in response to this tendency, and

• safety-seeking behaviours and patterns of avoidance which maintain these negative interpretations

Does the app translate the cognitive model into treatment, and specifically focus on three main goals:

• helping users to identify their catastrophic interpretations of bodily sensations as part of a maintenance vicious circle

• generating alternative non-catastrophic interpretations and attributions and

• testing the validity of catastrophic and non-catastrophic interpretations by ‘discussion’ and behavioural experiments, including disconfirmation and reversing safety seeking behaviour.

Assessment Rating

Does the app help users assess the detailed pattern of specific panic attacks, and to identify whether panic disorder is the main problem, or whether the pattern of panic is better accounted for by another diagnosis

Does the app help users describe the pattern of their panics, particularly, their frequency and severity and inter-connection between associated somatic sensations, fearful thoughts and safety seeking behaviours

Does the app help users focus on examples of recent and severe panics in order to identify details of relevant negative thoughts, images and somatic reactions

Does the app help users draw links between specific somatic sensations and specific thoughts

Does the app help users to identify safety seeking behaviours aimed at preventing or minimising catastrophic fears and consequences

Does the app help users identify the situations and activities associated with the occurrence of panics in order to help them confront these with appropriate preparation

Does the app help users identify patterns of avoidance (e.g. situations and activities, active and passive avoidance) again with a view to actively confronting them

Does the app help users appraise the client’s own beliefs about the problem, and the likely implications of these on motivation for treatment

Does the app help users assess the attitudes/beliefs and responses of significant others to the problem

Formulating an individualised version of the cognitive model Rating

Does the app help users to construct an individualised version of the cognitive model, which shows how their particular thoughts, sensations and behaviours contribute to the vicious circle of panic

© NICE 2019. All rights reserved. Subject to Notice of rights.

25 of 26

Monitoring Rating

Does the app set up and integrate appropriate monitoring procedures, including questionnaires, panic diaries and in-session ratings of beliefs; does it specify how the outcomes of monitoring may be used

Techniques to help users identify panic-related negative automatic thoughts and images

Rating

Does the app help users to focus on catastrophic misinterpretations (rather than on controlling feared sensations or related factors such as relaxation, hyperventilation and so on)

Does the app help users identify the observations that they use as evidence for their panic-related beliefs.

Does the app help users offer tailored corrective psychoeducation focused on the specific beliefs they hold regarding somatic sensations occurring before and during panics

Does the app help users to identify any examples where panic intensity has been moderated by events that contradict their beliefs, and to help them recognise the significance of these in the context of their individualised understanding

Does the app help users modify images representing feared outcomes using image restructuring (e.g. helping them to consider how realistic the image is, eliciting and reworking imagery in a graduated manner, and encouraging homework practice)

Behavioural experiments Rating

Does the app help users to devise and to conduct behavioural experiments aimed at modifying catastrophic misinterpretation of introceptive cues, ensuring:

• that experiments are relevant to the client, and

• that the outcomes from experiments are used to help the client see how these challenge their beliefs.

Does the app help users conduct a ‘paired associates’ or similar catastrophizing activation behavioural experiment in order to help them discover the operation of the vicious cycle of panic

In users who identify increased breathing as a response to panic (inadvertent or as a safety seeking behaviour) does the app/programme help them conduct a behavioural experiment utilising hyperventilation in order to help the client discover operation of the vicious cycle of panic

Does the app help users to conduct behavioural experiments involving redirected attention from an internal to an external focus in order to help them discover the operation of the vicious cycle of panic

Does the app help users to conduct a “chest pain” behavioural experiment in order to help them discover the operation of the vicious cycle of panic

Does the app help users to use behavioural experiments to in order to help them discover that some safety seeking behaviours induce panic related sensations

Does the app help users conduct behavioural experiments in which they evoke feared sensations without attempts to prevent anticipated catastrophes, and ensure that these are introduced only when the client is ready to undertake them

Adapting behavioural experiments to manage avoidance behaviour Rating

Does the app help users to expose themselves to feared situations or activities, in particular focussing on ways in which this enables them to test specific panic related beliefs

Does the app help users to model (role play) the social consequences of panic attacks which the client fears (such as fainting) with the aim of helping them discover that reactions to these events are less extreme than predicted

© NICE 2019. All rights reserved. Subject to Notice of rights.

26 of 26

Relapse prevention Rating

Does the app help users to plan for relapse prevention by encouraging them to anticipate strategies for the management of potential setbacks

Does the app help users to review whether any remaining panic-related beliefs are considered credible, and to work on these residual beliefs prior to completing the programme

Overall rating based on assessment of specific content