A Co-Existing Disorders (CED) Skills Framework and

Work Based Assessment (WBA) Approach to

Enhancing Workforce Development

Klare Braye (Matua Raki) and Dr Tom Flewett (CCDHB)

Acknowledgments

CCDHB CADS- Capital and Coast District Health Board Community Alcohol and Drug Service

Matua Raki-National Addiction Workforce Development

Pilot site clinicians, services and tangata whaiora

Overview

Issues: Building CED capabilityApproach: Literature review, Skills

development & assessment, Pilot, EvaluationFindings: Efficacy and utility Implications: Wider implementationConclusions

Issue of CED Skills Development

Requirement for services and clinicians to be CED capable

Questions raised about:How does that happen?How could it happen better?How do we know it is

happening/happened?

CED Capability

Competencies - relate to the individual Skills - practical application of the work we

do Skills development - acquisition and

enhancement of skills (in conjunction with appropriate knowledge and attitudes)

Skills measurement - quantifying/ qualifying skills development

Approach to the CED Skills Development Pilot Project Objective

• To improve the skills of clinicians from a wide variety of disciplines in assessment and treatment of clients with co-existing disorders

Method• Identify the skills required • Align these with existing competencies• Use workplace based assessments (WBAs) as training and

assessment tools Pilot Evaluation

• To assess validity, feasability and utility

Selected literature re: CED Competencies/Skills:

• International competencies (Graham &White 2011; www.ccsa.ca 2010)

• Challenges of linking competencies to clinical care (Jones et al., 2011) thus the move to a skills focus.

• Practical implications of implementing competencies (Mulder et al., 2010)

Competencies A number of competencies already exist:

– Aotearoa New Zealand Addiction Specialty Nursing Competency Framework (DANA)

– CanMeds Framework – Competency Based Fellowship Programme-CBFP (RANZCP)– Alcohol and Drug Practitioner Competencies (dapaanz)– Let’s Get Real: real skills for real people working in mental health

and addiction (Let’s Get Real, Real skills plus CAMHS, Real skills plus Seitapu)

– Problem Gambling Foundation Competencies– Nga Kaiakatanga Hauora mo Aotearoa Health Promotion

Competencies for Aotearoa New Zealand– Smoking Cessation Competencies for New Zealand– Takarangi Framework-Cultural competencies

Copyright © 2006 The Royal College of Physicians and Surgeons of Canada. http://rcpsc.medical.org/canmeds.

Reproduced and adapted with permission.

Collaborator Scholar

Communicator

HealthAdvocate

Professional M

anagerClinical Expert

Communicator

Role Competencies• Demonstrate the ability to work respectfully with clients, families,

carers, and carer groups• Demonstrate the ability to use interpersonal skills to improve client

outcomes

More specific Learning Outcomes• Demonstrate an ability to work collaboratively and respectfully

with consumer and carer reps, other health professionals and other agencies to improve patient outcomes

• Develop therapeutic relationships with clients, carers and relevant others

Te Ariari o te Oranga – a clinical framework

Seven key principles– Cultural considerations– Well-being– Engagement– Motivation– Assessment– Management– Integrated care

Examples of Identified CED Skills Undertake a comprehensive assessment Complete a risk assessment Implement a brief intervention Screen for mental illness/substance use

disorder Implement a relapse prevention strategy Appropriate management and support in

the provision of opioid substitution treatment

SKILLS Foundation Capable Enhanced

Brief Interventions Knowledge of brief interventions

Able to apply simple brief intervention strategies

Application of brief interventions

Demonstrates skills in utilising brief interventions in a variety of settings

Comprehensive mental health, substance use and gambling assessment

Knowledge of DSM-IV/ICD-10 for MH, SUDs (inc. Subs induced) and PG

Assessment of recent and lifetime mental health symptoms/problems (patterns of use and tx)

Assessment of recent and lifetime substance use Assessment of recent and lifetime gambling behaviour (patterns of use and tx)

Assessment of mental health, substance use and gambling history in the context of psychological and physical functioning, symptomatology and withdrawal history

Demonstrates knowledge and application of DSM-IV/ICD-10 criteria for mental health, substance use disorders (including substance-induced states) and Pathological Gambling

Monitoring and testing of substances, alcohol and medications

Knowledge of common substance testing procedures and laboratory investigations

Knowledge of common blood/urine tests

Application of substance testing procedures and laboratory investigations.

Comprehensive knowledge of methods for substance testing

Demonstrates knowledge and interpretation of common substance testing procedures and laboratory investigations

Miller’s Pyramid for Assessment

Does

Shows How

Knows How

Knows

Observed interviews with feedback

Written Exams eg MCQ

Complex written exams

Workplace based assessment

Assessment

Miller GE 1990. The assessment of clinical skills/competence/performance. Academic Medicine, 65 Supp 563-567

Selected literature re: Work Based Assessments

• Significant correlation between scores from the CEX and exam scores (Searle 2008)

• WBAs as an essential element of British psychiatrists training

• Need for good planning and understanding around the WBAs prior to initiation ‘(Menon, Winston and Sullivan 2012)

• Highlights validity measuring complexities and points to the value of utility (Holsgrove 2010)

Workplace Based Assessments to measure acquisition of skills

• ACE: Assessment of Clinical Expertise• CBD: Case Based Discussion• Mini-CEX-Mini Clinical Evaluation Exercise• MSF: Multi Source Feedback• MDT/Case Conference• Logbook• Client Satisfaction Questionnaire

The need for piloting and evaluation• Assess the:

• Utility (Skills framework/WBA)• Validity (NZ/CEP context)• Inter-rater reliability (disciplines/sites)• Generalisability (disciplines/services)• Feasibility (resources)

• Win the confidence of stakeholders• Provide opportunity for adaptation

and alteration

Evaluation Objectives• Validity

• Inter-rater reliability

• Generalisability

• Feasibility

• Utility

Selected Literature Re: Workforce Development• Workforce development typically is of individuals ,

through education and training , in a knowledge transfer structure (Allsop &Helfgott 2002, Roche 2009)

• Training as a stand-alone event, results in limited or non-sustained change (Arthur et al. 1998, Baer et al. 2004, Bennett et al. 2007, Roche 2002)

• Requires systems wide approach that supports the transfer of skills and knowledge Eg appropriate supervisory, peer and organisational support post training (Roche 2002, Cromwell 2004, Lim & Morris 2006, Taylor 2000)

Pilot Phase Two sites Invited enhanced practitioners/assessors Voluntary application for clinicians Involved assessor training and workbook Assessor and clinician evaluation

Assessor ParticipantsWellington Assessors:

counsellor (2)social worker (2)nurse (3)psychiatrist(1)

Christchurch Assessors:nurse (7)

At initiation n=16

At completion n=11

Service Type DHB 9 7

NGO 7 4

Addiction 7 4

Mental Health 9 7Professional discipline Support worker 2 2

Cultural worker 1 1

Counsellor 6 3

Social worker 3 2

Nurse 3 2

Psychiatrist 1 1

Self reported/perceived level of CED competence

Foundation 8

Capable 6

Capable/Enhanced 2

Wellington Clinician Participants

Study Design Written survey

– Qualitative and quantitative responses Training package Skills Framework guiding document CEP skills framework Each of the work based assessments The pilot overall

Focus groups– Separate assessor and clinician focus groups

Barriers and benefits Value and utility of the framework and WBAs Assessor/clinician relationships The pilot project overall

Key Findings Skills Framework Reflected a diverse range of skills across services

and disciplines From its original form, requires further

development Clinicians would like feedback on their level of

capability Clinicians will straddle a number of skills across

the range of foundation, capable and enhanced Link the framework and WBAs to employment

and PD opportunities/plans

Key Findings Work Based Assessment

Benefits clinician and the assessor

CBDs and ACEs most time consuming but beneficial

Summative and formative assessment

Structured approach to CED supervision

Clarify the assessor/supervisor role

Respond to client feedback

Encourage reflective practice

A compulsory element to completion

Encourages sustained behaviour change and practice Pilot shows applicability and benefits to clinical practice:

– Addiction and mental health clinicians – Across disciplines and professional bodies– NGOs and DHBs – Assessors and clinicians skills development– Across service types/networking

A willingness to up skill and develop CEP capability– Resourcing intensity cannot be underestimated– Support required from key stakeholders and management

Implications

Conclusion The skills framework and associated WBAs have

value and utility as a tool to enhance CEP capability They encourage implementation of best clinical

practice and supervision Whilst resource intensive they benefit all parties,

with sustained change Would be best serviced to complement, not

supersede formal knowledge building and existing forms of professional development

Training to be from a whole systems perspective, within the context of a robust workforce development plan to support effective practice.

ReferencesAllsop, S. J., & Helfgott, S. (2002). Whither the drug specialist? The workforce development needs of drug specialist staff and agencies. Drug and Alcohol review, 21, 215-222.Arthur, W., Jr., Bennett, W., Jr., Stanush, P. L., & McNelly, T. L. (1998). Factors that influence skill decay and retention: A quantitative review and analysis. Human Performance, 11, 57–101.Baer, J.S., Rosengren, D.R., Dunn, C., Wells, E.A., Ogle, R., & Hartzler, B. (2004). An evaluation of workshop training in motivational interviewing for addiction and mental health clinicians. Drug and Alcohol Dependence, 73, 99-106. Bennett, G. A., Moore, J., Vaughan, T., Rouse, L., Gibbins, J.A., Thomas, P., James, K., & Gower, P. (2007). Strengthening Motivational Interviewing skills following initial training: A randomised trial of workplace-based reflective practice. Addictive Behaviors, 32, 2963-2975. Bhugra, D., Malik, A. & Brown N. (2007) “Workplace-based Assessments in psychiatry” published College Seminar Series of the Royal College of Psychs., UK. CCSA. Competencies for Canada’s Substance Abuse Workforce. www.ccsa.ca/eng/priiorities/workforce/competencies/pages/default.aspx. Retrieved on 16/7/2012.Cromwell, S. E., & Kolb, J. A. (2004). An examination of work-environment support factors affecting transfer of supervisory skills training to the workplace. Human Resource Development Quarterly, 15(4), 449-471.Holsgrove, G. Reliability issues in the assessment of small cohorts. General Medical Council Guidance Paper. 2010Friedman, Mark (2005),Trying Hard is Not Good Enough:How to Produce Measurable Improvements for Customers and Communities. Trafford Publ.Canada p81.Graham, H. & White, R. (2011) Comorbidity Competencies: Skills Indicators by support recovery from comorbidity in Tasmania, University of Tasmania: AustraliaJones, M.D., Rosenburg,A., Gilhooly, J. and Carraccio C. Competencies, outcomes and controversy-Linking professional activities to competencies to improves resident education and practice. Academic Medicine. 2011; vol 86. 2:1-5Lim, D. H., & Morris, M. L. (2006). Influence of trainee characteristics, instructional satisfaction, and organizational climate on perceived learning and training transfer. Human Resource Development Quarterly, 17, 85-115.Menon, S., Winston,M. & Sullivan G. Workplace-based assessment: attitudes and perceptions among consultant trainers and comparison with those of trainees. The Psychiatrist Online Jan. 2012. 36:16-24;Miller GE (1990). The assessment of clinical skills/competence/performance. Academic Medicine, 65 Supp 563-5672 Mulder, H. ten Cate, O., Daalder, R. & Berkverns, J. Building a competency-based workplace curriculum around entrustable professional activities: The case of physician assistant training. Med. Teach 2010;32(10):e453-9.Roche, A. M. (2009). New horizons in AOD workforce development. Drugs: education, prevention and policy. 16 (3), 193 - 204. Roche, A. M. (2002). Workforce Development Issues in the AOD Field: A Briefing Paper for the Inter-Governmental Committee on Drugs. Unpublished Report. National Centre for Education and Training on Addiction. Searle, G. Is Cex good for psychiatry? An evaluation of workplace-based assessment. Psychiatric Bulletin. 2008; 32:271-273Taylor, M. C. (2000). Partners in the transfer of learning: A qualitative study of workplace literacy programs. Proceedings of the 41st Annual Adult Education Research Conference (pp. 459-461). Vancouver: University of British Columbia.Walters, S. T., Matson, S. A., Baer, J.S., & Ziedonis, D. M. (2005). Effectiveness of workshop training for psychosocial addiction treatments: A systematic review. Journal of Substance Abuse Treatment, 29, 283-293. The Royal college of physicians and surgeons of Canada. 2006. http://rcpsc.medical.org/canmeds

Competency Framework References Alcohol and Drug Treatment Workforce Development Advisory Group. 2001. Practitioner Competencies for Alcohol & Drug Workers in Aotearoa – New Zealand. ALAC Occasional Publication: No 13. Wellington: Alcohol Advisory Council of New Zealand. DAPAANZ (Drug and Alcohol Practitioners Association Aotearoa New Zealand). 2008. ALAC Alcohol and Drug Practitioner Competencies, Revised 2008. Wellington: DAPAANZ. Health Promotion Forum. 2000. Nga Kaiakatanga Hauora mo Aotearoa Health PromotionCompetencies for Aotearoa New Zealand. Auckland: Health Promotion Forum. Le Va Pasifika. 2009. Real skills plus Seitapu: working with Pacific peoples. Auckland: Le Va Pasifika Te Pou o Te Whakaaro Nui The National Centre of Mental Health Research, Information and Workforce Development. Matua Raki. 2009.Takarangi Competency Framework. Wellington: Matua Raki National Addiction Workforce Development. Matua Raki 2011. Aotearoa New Zealand Addiction specialty nursing (knowledge and skills) competency framework (Addiction nursing framework) Wellington: Matua Raki National Addiction Workforce Development. Ministry of Health. 2008. Let’s get real: real skills for people working in mental health and addiction. Wellington: MoHMinistry of Health. 2007. Smoking Cessation Competencies for New Zealand. Wellington: Ministry of Health. Nursing Council of New Zealand. 2007. Competencies for nurse assistants and enrolled nurses. Wellington: Nursing Council of New Zealand. Nursing Council of New Zealand. 2007. Competencies for registered nurses. Wellington: Nursing Council of New Zealand. Problem Gambling Foundation NZ. 2009. Problem Gambling Foundation Competencies. Auckland: PGFNZPublic Health Association of New Zealand. 2007. Generic Competencies for Public Health in Aotearoa-New Zealand. New Zealand: Public Health Association of New Zealand. Royal Australian and New Zealand College of Psychiatrists 2012. Competency Based Fellowship Programme-CBFP The Royal college of physicians and surgeons of Canada. 2006. CanMeds Framework. http://rcpsc.medical.org/canmedsThe Werry Centre. 2008. Real skills plus CAMHS: A competency framework for the infant, child and youth mental health and alcohol and other drug workforce. Auckland: The Werry Centre for Child and Adolescent Mental Health Workforce Development.

“Culture change does not happen in complex organisations through one or two training

sessions... it happens with little events that happen thousands of times in the life of an

organisation”

(Friedman, 2005:81)