London, England

7-8

July

2

01

1International Congress onProfessional andOccupational Regulation

Applying what evidence there is to a new continuing competence programme: An individually tailored, high-trust approach.

Anne Goodhead and Steve Osborne, New Zealand Psychologists Board

Promoting Regulatory Excellence

Presentation Outline

1. Introduction to our Continuing Competence Programme (CCP)

2. Rationale

3. Development process

4. Principles

5. Evidence for our approach

6. Where to from here

The CCP

Introduced April 2009.

Compulsory for all active practitioners to participate.

Just completed our second year, including audits of participation.

The CCP (six step) model

1Against Core Competencies2With Supervisor's involvement

Self Reflective Review1,2

Documenteach step

Strengths and Weaknesses

Learning Objectives

Learning Activities

Self-directed Learning

Plan

Year-end Review of Benefits2

Rationale

The HPCA Act requires that the Board be satisfied that a practitioner is competent before issuing a practising certificate.

Complaints data suggest older and more experienced practitioners are more likely to attract complaints, consistent with research in other disciplines.

Our aim is to lift the competence of all.

How can we lift performance?

Best practice guidelines.

Newsletters to raise awareness of pertinent issues.

Well-established culture of supervision at all levels of practice.

And most recently, mandatory CCP.

Development of the CCP

Consultation - widespread support.

Pilot study helped refine the model.

Informed by literature on CPD.

Newsletter updates on instructions, also conference and stakeholder group presentations.

Audit has informed further refinement, as has our new learning outcomes survey.

CCP principles

Lift competence standards overall.

Individually tailored: Practitioners benchmark themselves against competence standards, then strive to improve on own areas of need.

Assure the public of high standards.

Flexible: To encompass the diversity of practice.

CCP principles

Valid and relevant for practitioners.

Support self-regulation, and the natural incentives for high quality practice.

Increase self-efficacy.

Why try to lift overall standard?Need to counter the trend of knowledge

and performance declining with years of experience. Choudhry 2005 review

Traditional input models to CME and CE known to have only a small impact. Marinopoulos et al 2007, Mansouri and Lockyer 2007, Forsetlund et al (Cochrane review) 2009, Davis and Galbraith 2009

Research on output models of CPD “sparse” but seen as way forward. PARN 2008

Will the CCP lift the overall standard?

It is informed by what works:• Improved by multiple media, mixed

interactive. Mansouri and Lockyer 2007

• Multiple events on single topic. Davis and Galbraith 2009, Mazmanian et al 2009

• Practitioner assessed learning needs. Mazmanian and Davis 2002

• Integrated with practice.

Will the CCP lift the overall standard?

What works (continued):• Self-awareness, self-reflection, and

self-evaluation. Kaslow et al 2007

• Seek feedback. Eva and Regehr 2008

• Guided self-assessment. Duffy and Holmboe 2006

• Evaluation of outcomes - participant satisfaction, knowledge, behaviour, patient safety. Tian et al 2007

And what of individual improvement?

CCP requires a Self Reflective Review against prescribed Core Competencies.

Eva and Regehr caution against self-evaluation. (2005, 2006) We require supervisor or colleague involvement - but is this enough?

Face validity: Set goals, plans, and evaluate at end of the CCP year.

Learning outcomes, CCP vs. opportunistic

Does CCP assure safety of the public?Meeting legislative obligations.

Consumer and policy “watchdogs” on public safety groups promote a similar approach to professional development. Citizens Advocacy Centre (2006), Institute of Medicine (2010)

CCP includes the quality assurance step of evaluating outcomes and seeking feedback re impact on practice.

IS the CCP valid and relevant?

Intention is that each psychologist shapes their programme to be meaningful, useful, relevant, and valid for them.

Avoids tokenism.

Does not block opportunistic professional development or that driven by unanticipated professional challenge.

Does the CCP facilitate self-regulation?

Our pilot study suggests most psychologists do a considerable amount of PD without being directed to.

The majority hold high ethical standards and strive for excellence.

Natural incentives reward high standards.But, a US survey showed ~ 20% did little

if not mandated. Neimeyer,2009,2010a,2010b

Does CCP support self-efficacy?

Our pilot study indicated amount of PD was correlated with confidence in being up-to-date with psychology practice.

Undertaking the SRR also appeared to be associated with more confidence and higher professional self esteem.

Do the benefits outweigh the costs?

In an ideal world:

- cost-benefit analysis.

- demonstrate cost-effective.

- compared to status quo.

- compared to other CE models.

But hard to measure this in PD generally - Few studies, evidence weak, hard to research. Brown et al 2002

Costs

Personal/Employer costs: • Bureaucratic costs of record keeping.

• Opportunity costs in supervision.

• For some, increased PD.

• Organisational costs (e.g., employee’s time).

• Stress.

Costs

For the Board:• Labour costs of development and

instruction.

• The financial, time, and other resource costs of the annual audit.

• Potential loss of goodwill.

Benefits

Potential impact on client outcomes.

Mixed reports whether it changes what individuals would have done anyway.

Long term impact unknown (but we’re hopeful).

Benefits

Meeting statutory obligations.

Feedback from audit participants and others…

Audit participant feedback How burdensome or time consuming have you found the recording of your CCP?

1 = highly burdensome 10 = low burden

Audit participant feedback

Has the CCP helped you focus your professional development activities?

1 = No 10 = Yes

Audit participant feedback

Has CCP participation helped you to ensure that you are competent?

1 = No 10 = Yes

Audit participant feedback

Have you found the CCP useful?

1 = not useful 10 = very useful

Where to from here?

Continue to fine tune, have a framework we can tweak:

Now offering more structured way of recording to reduce confusion and time (optional template).

Clarifying role of supervisor, goal setting, and the end of year review.

Attitudes remain mixed, ranging from enthusiastic to resentful.

Where to from here?

Further research?

We would be interested in any related research that you know of.

Thank you!

And thanks to Dr Lois Surgenor for

her assistance with the statistical analysis!

Research and development work by…

Anne GoodheadPsychology AdvisorNew Zealand Psychologists BoardPO Box 10-626Wellington 6143New Zealand

anne.goodhead@nzpb.org.nzwww.psychologistsboard.org.nz

Speaker Contact Information

Steve Osborne MSc ClinPsych MASPPB MIPGA

Chief Executive and RegistrarNew Zealand Psychologists BoardPO Box 10-626Wellington 6143New Zealand

steve.osborne@nzpb.org.nzwww.psychologistsboard.org.nz

Bibliography• Brown, C., C. Belfield, et al. (2002). "Cost effectiveness of continuing professional

development in health care: a critical review of the evidence." British Medical Journal 324: 652-655.

•• CAC - refer to Swankin et all below.

• Choudhry, N., R. Fletcher, et al. (2005). "Systematic Review: The relationship between clinical experience and quality of health care." Annals of Internal Medicine 142(4): 260 -273.

•• Davis, D. and R. Galbraith (2009). "Continuing Medical Education Effect on Practice

Performance." Chest 135(3): 42S-48S.•• Duffy, F. D. and E. Holmboe (2006). "Self-assessment in lifelong learning and improving

performance in practice: Physician know thyself." JAMA 296(9): 1137-1139.•• Eva, K. and G. Regehr (2005). "Self-assessment in the Health Professions: A

reformulation and research agenda." Academic Medicine 80(10): S46-S54.•• Eva, K. and G. Regehr (2008). ""I'll never play professional football" and other fallacies of

self-assessment." Journal of Continuing Education in the Health Professions 28(1): 14-19.•• Forsetlund, L., A. Bjorndal, et al. (2009). "Continuing education meetings and workshops:

effects on professional practice and health care outcomes (review)." The Cochrane Library(2).

• Institute of Medicine (2010). Redesigning continuing education in the health professions. The National Academies Press. Washington, DC, Downloaded from:

• http://www.nap.edu/catalog/12704.html

• Professional Associations Research Network (PARN) (2008). Approaches to Continuing Professional Development (CPD) Measurement. International Accounting Education Standards Board. New York, International Federation of Accountants. Downloaded from: http://www.ifac.org

•• Kaslow, N., N. Rubin, et al. (2007). "Recognising, assessing and intervening with problems of

professional competence." Professional Psychology: Research and Practice 38(5): 479-492.

• Marinopoulos, S., Dorma,T. et al (2007)”Effectiveness of Continuing Medical Education, Evidence report/technology assessment no 149”. Rockville, MD: Agency of Healthcare Research and Quality.

••• Mazmanian, P. (2009). "Continuing Medical Education Costs and Benefits: Lessons for

competing in a changing health care economy." J of Continuing Education in the Health Professions 29(3): 133-134.

••• Mazmanian, P. and D. Davis (2002). "Continuing Medical Education and the physician as a

learner." JAMA 288(9): 1057-1060.••• Monsouri, M. and J. Lockyer (2007). "A meta-analysis of continuing medical education

effectiveness." Journal of Continuing Education in the Health Professions 27(1): 6-15.••

• Neimeyer, G. and J. Taylor (2009). "Continuing education in psychology: Outcomes, evaluations and mandates." Professional Psychology: Research and Practice

• 40(6): 617-624.•• Neimeyer, G., J. Taylor, et al. (2010). "Continuing education in psychology: Patterns of

participation and aspects of selection." Professional Psychology: Research and Practice 41(4): 281-287.

• Neimeyer, G., J. Taylor, et al. (2010). "Continuing education in psychology: Patterns of participation and perceived outcomes among mandated and nonmandated psychologists." Professional Psychology: Research and Practice 41(5): 435-441.

••• Regehr, G. and K. Eva (2006). "Self-assessment, Self-direction, and the Self-regulating

Professional." Clinical Orthopaedics and Related Research Number 449: 34-38.•• Regehr, G. and M. Mylopoulos (2008). "Maintaining competence in the field: Learning about

practice, through practice, in practice." J of Continuing Education in the Health Professions 28(S1): S20-S23.

••• Swankin, D., R. Le Buhn, et al. (2006) Implementing continuing competency requirements for

health care practitioners. Citizens Advocacy Center, AARP Public Policy Institute. Downloaded from: http://www.aarp.org/ppi

• • Tian, J., N. Atkinson, et al. (2007). "A systematic review of evaluation in formal continuing medical

education." J of Continuing Education in the Health Professions 27(1): 16- 27.