Development and evaluation of clinical practice guideline in

psychiatry

Dr Diptadhi MukherjeeLGBRIMH, TezpurDate: 22/02/16

Outline

• Introduction• Development of guidelines• Evaluation of guidelines• Brief comparison of guidelines• Indian scenario• Conclusion

Introduction• The history of Good Clinical Practice statute traces back to The

Hippocratic Oath- ethical code it is primarily known for its edict to do no harm to the patient.

• Clinical practice guidelines are one of the foundations of efforts to improve health care.

• The modern age of guidelines began with a 1992. Institute of Medicine (IOM) report

• Defined guidelines as “systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances”

• Psychiatry, as a discipline, is relatively new in the field of evidence based medicine.

• Some of the most well-known and widely accepted guidelines are -

APA guidelines by the American Psychiatric Association Texas Medication Algorithm Project Group (TMAP) Patient Outcomes Research Team (PORT) Canadian treatment guidelines by the Canadian Psychiatric

Association CANMAT (Canadian Network for Mood and Anxiety

Treatments) NICE guidelines by National Institute for Clinical Excellence Maudsley guidelines

Why are these guidelines needed?

• Improve the quality of health care.• Reduce the use of unnecessary, ineffective or harmful interventions.• Facilitate the treatment of patients with maximum chance of

benefit, with minimum risk of harm, and at an acceptable cost.

• Principles of good decision making which takes account of patients' preferences and values, clinicians values and experience, the best available evidence and the availability of resources.

National Health and medical Research Council (NHMRC) 1999. A guide to the development, implementation and evaluation of clinical practice guidelines.

Development of CPG• Six CPG development handbooks available.

1. Council of Europe 2. National Health and Medical Research Council of Australia3. National Institute for Health and Clinical Excellence4. New Zealand Guidelines Group5. Scottish Intercollegiate Guideline Network 6. World Health Organization (WHO).

• Target audiences and their use of guidelines• Prioritizing topics for guideline development

• Guideline group composition and group process- Technical process (systematic reviews of relevant evidence) and Social

process (interpretation of the results of the systematic review and development of recommendations)

Sufficient experience Vs Cohesiveness Optimum size for a small group - between eight and ten people, although

groups of larger size have operated effectively

• Consumer involvement- “incorporate patients’ values, preferences, knowledge, or perspectives in CPG recommendations”– incorporating individual patients in guideline development groups;– a ‘one off ’ meeting with patients;– a series of workshops with patients;– incorporating a consumer advocate in guideline development groups

Eccles et al. Implementation Science 2012, 7:60

• Managing conflicts of interest (COI) in guideline development-

Financial COI is the best known type of COI and typically a result of direct financial benefit related to topics discussed or products recommended in guidelines or personal financial interests

Intellectual COI is another type of conflict that is increasingly recognized and results from a guideline group member being invested in her/her intellectual work

• Asserting strength of evidence- Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approaches

• 5 factors can lower the confidence in an estimate of effect and study quality:

Study design and execution (risk of bias) Inconsistency of the evidence across studies Indirectness of the evidence (including concepts of lack of generalizability,

transferability and external validity) Lack of precision of the estimate of the effect Publication bias.• 3 factors can increase the quality of evidence: A strong or very strong association; A dose-effect relationship; All plausible residual confounding may be working to reduce the

demonstrated effect or increase the effect if no effect was observed.

• 4 Factors that influence recommendations-

Quality of the available supporting body of evidence Balance between benefits and undesirable downsides Certainty about or variability in values and preferences of

patients Resource expenditure associated with the management

options

Balance between benefits and undesirable downsides

• When the benefits of recommendation >>> downsides - the recommendation will be strong.

• When the desirable consequences ≈ undesirable consequences - a weaker recommendation is warranted.

• Relative risk/ hazard ratio/ odds ratio Vs absolute effects for a specific population or situation

• Desirable and undesirable effects at baseline risk

Uncertainty or variability of patient values and preferences• Clinicians’ understanding of importance of particular outcomes for

patients can differ from that of the patients• Data about patients’ preferences and values are often limited.• Alternative management strategies- as a trade-off is always

necessary.• Transparent explanation facilitates the interpretation of

recommendations

Costs or resource utilization• Costs are much more variable over time and geographic areas than

are other outcomes• Higher costs will reduce the likelihood of a strong recommendation

in favor of a particular intervention• In considering resource allocation- very specific about the setting to

which a recommendation applies and the perspective they took, i.e., that of a patient, a third party payer or society as a whole.

Wording recommendations

• Inform users of guidelines (e.g., clinicians, patients and their family members, policy makers) about the degree of confidence by specifying the strength of recommendations.

• Phrasing recommendations in an active voice as clear indications what specific action should follow- ‘we recommend . . .’ Vs ‘we suggest . . .’

• Alternatively- ‘clinicians should . . .’ Vs ‘clinicians might . . .’ or ‘we conditionally recommend . . ..’

• Avoid ambiguous phrases- ‘clinically appropriate’ or ‘if necessary.’

Should guideline panels make recommendations in the face of very low-quality evidence?

• Option of not making a recommendation should be included for all guideline panels- ‘insufficient evidence to make a recommendation’ category.

• Higher-quality evidence may never be obtained/ physicians need guidance regardless- transparently lay out the judgments they make.

Evaluation of clinical guidelines

• Numerous guidelines in every disciplines

• Great variability exists in the quality of clinical practice

guidelines

• Need to discriminate high-quality from lower-quality

guidelines.

• 24 appraisal instruments of practice guidelines

• The AGREE instrument-

• Based on the Cluzeau instrument• It has been validated. • It uses a numerical scoring scale- easier to compare scores.• easy-to-use, and transparent instrument• internationally developed and widely accepted, • “It can possibly serve as a basis for an instrument to evaluate

the methodological quality of clinical pathways. “

Françoise Cluzeau

• The AGREE Instrument is designed to assess guidelines developed by local, regional, national or international groups or affiliated governmental organizations. These include:1. New guidelines2. Existing guidelines3. Updates of existing guidelines

• The AGREE Instrument is intended to be used by the following groups:i) By policy makers to help them decide which guidelines could be

recommended for use in practice.ii) By guideline developers to follow a structured and rigorous

development methodology and as a self-assessment tool to ensure that their guidelines are sound.

iii) By health care providers who wish to undertake their own assessment before adopting the recommendations

Structure and content of the AGREE Instrument AGREE consists of 23 key items organized in six domains. Each domain is

intended to capture a separate dimension of guideline quality.• Scope and purpose (items 1-3) is concerned with the overall aim of the

guideline, the specific clinical questions and the target patient population.• Stakeholder involvement (items 4-7) focuses on the extent to which the

guideline represents the views of its intended users.• Rigour of development (items 8-14) relates to the process used to gather

and synthesize the evidence, the methods to formulate the recommendations and to update them.

• Clarity and presentation (items 15-18) deals with the language and format of the guideline.

• Applicability (items 19-21) pertains to the likely organizational, behavioral and cost implications of applying the guideline.

• Editorial independence (items 22-23) is concerned with the independence of the recommendations and acknowledgement of possible conflict of interest from the guideline development group.

Document-• Recommend that you read the guideline and its accompanying

documentation fully before you start the appraisal. Number of appraisers• We recommend that each guideline is assessed by at least two appraisers

and preferably four as this will increase the reliability of the assessment. Response scale• Each item is rated on a 4-point scale ranging from 4 ‘Strongly Agree’ to 1

‘Strongly Disagree’, with two mid points: 3 ‘Agree’ and 2 ‘Disagree’. Comments• There is a box for comments next to each item. You should use this box to

explain the reasons for your responses.

Calculating domain scores

Overall assessment• A section for overall assessment is included at the end of the instrument. • Contains a series of options ‘Strongly recommend’, ‘Recommend (with provisos

or alterations)’, ‘Would not recommend’ and ‘Unsure’.

SCOPE AND PURPOSE

1. The overall objective(s) of the guideline is (are) specifically described.• Potential health impact of a guideline on society and populations of

patients- Rational prescribing of antidepressants in a cost-effective way2. The clinical question(s) covered by the guideline is(are) specifically

described.• Are selective serotonin reuptake inhibitors (SSRIs) more cost-effective than

tricyclic antidepressants (TCAs) in treatment of patients with depression?3. The patients to whom the guideline is meant to apply are specifically

described.• A guideline on the management of depression only includes patients with

major depression, according to the DSM--5 criteria, and excludes patients with psychotic symptoms and children.

STAKEHOLDER INVOLVEMENT

4. The guideline development group includes individuals from all the relevant professional groups.

• Information about the composition, discipline and relevant expertise of the guideline development group should be provided.

5. The patients’ views and preferences have been sought.• For example, the development group could involve patients’

representatives, information could be obtained from patient interviews, literature reviews of patients’ experiences could be considered by the group

6. The target users of the guideline are clearly defined.• For example, the target users for a guideline on headache may include

general practitioners, neurologists, psychiatrist, and psychologist7. The guideline has been piloted among target users.• For example, a guideline may have been piloted in one or several primary

care practices or hospitals. This process should be documented

RIGOUR OF DEVELOPMENT

8. Systematic methods were used to search for evidence.• Search terms used, databases of systematic reviews9. The criteria for selecting the evidence are clearly described.• Including /excluding criterias10. The methods used for formulating the recommendations are clearly

described.• Areas of disagreement and methods of resolving them should be

specified- for example, a voting system, formal consensus techniques (e.g. Delphi, Glaser techniques).

11. The health benefits, side effects and risks have been considered in formulating the recommendations.

• These may include: survival, quality of life, adverse effects, and symptom management or a discussion

12. There is an explicit link between the recommendations and the supporting evidence.

• Recommendation should be linked with a list of references on which it is based

13. The guideline has been externally reviewed by experts prior to its publication.

14. A procedure for updating the guideline is provided.• For example, a timescale has been given, or a standing panel receives

regularly updated literature searches and makes changes as required.

CLARITY AND PRESENTATION

15. The recommendations are specific and unambiguous.• Recommendation should provide a concrete and precise description of

which management is appropriate in which situation and in what patient group, as permitted by the body of evidence.

16. The different options for management of the condition are clearly presented.

• For example, a recommendation on the management of depression may contain the following alternatives:

• a. Treatment with TCA/ b. Treatment with SSRI/ c. Psychotherapy/ d. Combination of pharmacological and psychological therapy

17. Key recommendations are easily identifiable.• For example, they can be summarized in a box, typed in bold, underlined

or presented as flow charts or algorithms.18. The guideline is supported with tools for application.• For example, a summary document, or a quick reference guide,

educational tools, patients’ leaflets, computer support, and should be provided with the guideline.

APPLICABILITY

19. The potential organizational barriers in applying the recommendations have been discussed.

• Recommendations may require changes in the current organization of care within a service or a clinic which may be a barrier to using them in daily practice.

20. The potential cost implications of applying the recommendations have been considered.

• For example, there may be a need for more specialized staff, new equipment, expensive drug treatment. These may have cost implications for health care budgets. There should be a discussion of the potential impact on resources in the guideline.

21. The guideline presents key review criteria for monitoring and/or audit purposes.

• Clearly defined review criteria that are derived from the key recommendations in the guideline- like follow up of MDD- HAM-D < 7 for 6 months

EDITORIAL INDEPENDENCE

22. The guideline is editorially independent from the funding body.• Explicit statement that the views or interests of the funding body have not

influenced the final recommendations. If it is stated that a guideline was developed without external funding, then should answer ‘Strongly Agree’.

23. Conflicts of interest of guideline development members have been recorded.

• For example, this would apply to a member of the development group whose research on the topic covered by the guideline is also funded by a pharmaceutical company. There should be an explicit statement that all group members have declared whether they have any conflict of interest

Limitations of the AGREE instrument-

• Does not include criteria addressing how the guideline topic was selected

• Does not include criteria addressing how the it will be implemented.

• Does not assess whether systematic methods were used to appraise the research evidence used to support guideline recommendations

• Not possible to set thresholds for the scores to classify a clinical practice guideline as ‘good’ or ‘bad’.

• NICE guideline had the highest methodological quality according to AGREE and the highest scores in five out of six domains.

• [‘One explanation might be that this guideline was developed as part of a national policy within an established guideline programme adequately resourced by the health authorities.’]

• Followed by the second edition of the American Psychiatric Association (APA) guideline (total score=71)the Royal Australian and New Zealand College of Psychiatrists guideline total score= 62)

• Most (19) of the 24 guidelines did not include contributions from key stakeholders such as patients or relatives.

• Only three guidelines considered health-economic effects of the treatment options or other cost issues (AU, FI, GB1)

• 5 guidelines referred to particular cultural, ethnic or socioeconomic issues either in diagnostic assessment or treatment planning (AU, DK, GB1, SG, US1).

• Only 6 guidelines were reviewed externally by reviewers not involved in the guideline development.

• Only 6 guidelines (AU, DK, FI, GB1, NO, US1) gave background information and detailed recommendations for specific mental health community treatment.

• Large variations in the type and frequency of psychosocial interventions recommended.

• A majority of guidelines recommended some kind of family support or family involvement

• Half had recommendations for psycho-educational interventions and vocational rehabilitation. However, recommendations concerning psychosocial interventions were generally not detailed.

Indian Psychiatric Society (IPS) guidelines

• Indian Psychiatric Society task force on clinical practice guidelines for psychiatrists in India proposed clinical practice guidelines for psychiatrists in India

• Based on the national workshop held at Jaipur in 2004.• A detailed literature review, mentions elaborate list of the Indian studies• Include specific recommendations about treatment of the acute,

continuation/stabilization, maintenance/stable phase of various mental illnesses.

• Include several recommendations about pharmacological management, group and individual therapy, vocational rehabilitation and specific psychosocial interventions

• Distinct advantage of being easy to use and comprehend- clearly convey the duration of treatment and other important issues/ algorithms given at the end of chapters further simplify the user’s understanding/ a point-wise listing of the guidelines

Indian J Psychiatry 49(4), Oct-Dec 2007

But……• Influence of Indian studies on the final drafting of the guidelines is not

clear; the guidelines seem to be a direct adaptation of the Western literature.

• Studies occur separately as a table or in the bibliography, without any mention in the text.

• Depth of recommendations- lacking strength• Cost and resource- guideline failed to conceptualize this important aspect

in the IPS guidelines- no mention of the cost of therapy of any of the illnesses;

• Supposed to serve as guiding principles for the Indian psychiatrists- not been properly reflected in the drafting of the IPS guidelines. – Drugs or psychotherapeutic techniques which are not available in

India or are available at very few centers. – Do not give enough emphasis to the available resources

But….

• Timely review/ update not done

• Specificity or ease of operationalizing- almost didactic in format,

making them more difficult to operationalize

• Contributor bias

• Lack a structured format- not uniform

• Ethnicity, psychosocial and cultural considerations- not done

Conclusion

• ??Unified guideline

• Scope of vast improvement in Indian scenario

• Guideline dissemination and implementation

• Specific mental health guidelines could be of considerable

importance in changing mental health treatment and

professional performance.

References

• Institute of Medicine Committee to Advise the Public Health Service on Clinical Practice Guidelines. Clinical Practice Guidelines: Directions For A New Program. Washington DC: National Academy Press, 1990.

• Vlayen J. et al. A systematic review of appraisal tools for clinical practice guidelines: multiple similarities and one common deficit. International Journal for Quality in Health Care 2005; Volume 17, Number 3: pp. 235–242

• http://www.implementationscience.com/content/7/1/61• Gaebel W et al. Schizophrenia practice guidelines: international survey and

comparison. British Journal Of Psychiatry ( 2 0 0 5 ) , 1 8 7, 2 4 8-2 5 5• AGREE Collaboration. Development and validation of an international

appraisal instrument for assessing the quality of clinical practice guidelines: the AGREE project. Qual Saf Health Care 2003; 12: 18–23.

• Goel D, Trivedi JK. Clinical practice guidelines for psychiatrists: Indian Psychiatric Society guidelines vs. international guidelines: A critical appraisal. Indian J Psychiatry 49(4), Oct-Dec 2007

THANK YOU