Prof Dame Lesley Southgate: St Georges Medical School, University of London

Developing revalidation for UK physicians

Maintenance of competence in the age of technology

Prof Dame Lesley Southgate: St Georges Medical School, University of London

Prof Dame Lesley Southgate: St Georges Medical School, University of London

Prof Dame Lesley Southgate: St Georges Medical School, University of London

Purpose of this talk

To show that evidence for revalidation of physicians can be collected and judged within a systematic, transferable framework which could apply to all physicians at any stage of their career

To ask you, as I discuss some of the slides, to think over ways that storage and exchange of this evidence could be achieved without distorting or losing its meaning or creating false synergies

Prof Dame Lesley Southgate: St Georges Medical School, University of London

Revalidation has a primary purpose

to demonstrate that doctors on the register continue to meet the standards that apply to their own discipline

Implies a basic standard that nearly all doctors should reach

Prof Dame Lesley Southgate: St Georges Medical School, University of London

It can also be seen to have a number of secondary purposes

To promote continuous professional development among practitioners

To encourage improvement in the quality of care, team working, communications and appropriate behaviour of practitionersImplies standards which allow progress towards

excellence to be documented

Prof Dame Lesley Southgate: St Georges Medical School, University of London

It can also be seen to have a number of secondary purposes

To identify doctors, as far as is practicable, for whom there are significant concerns about their fitness to practise

To reassure, as far as is possible, the public and individual patients that their doctors are up to date and fit to practiseImplies accuracy around the ‘cut point’ for the

basic standard

Prof Dame Lesley Southgate: St Georges Medical School, University of London

Competence versus Performance

Competence is necessary, but not sufficient, for acceptable clinical performance and this concept is especially important when assessing the performance of doctors in difficulty where it is essential to determine if poor performance stems from a compromised knowledge and skill base or not- the difference between can’t perform, or won’t which may be related to a failure of professionalism or poor health or social circumstances

Prof Dame Lesley Southgate: St Georges Medical School, University of London

GMC Good Medical Practice

provides the broad domains underpinning a potential assessment framework

All Royal Medical Royal Colleges have elaborated the guidance to define the characteristics of the excellent (and unacceptable) physician.

GMP underpins every aspect of UK medical practice from undergraduate through residency to continuing professional development

It underpins licensure and certification

Prof Dame Lesley Southgate: St Georges Medical School, University of London

Good Medical Practice

Can be mapped to Canmeds and the ACGME competencies

We have a framework/classification for how we record information about a physician through the continuum of practice

Prof Dame Lesley Southgate: St Georges Medical School, University of London

Domains of performancefrom Good Medical Practice

Knowledge, skills and performanceMaintain your professional performanceApply knowledge and experience to practiceKeep clear, accurate and legible records

Safety and QualityPut into effect systems to protect patients and improve

care Respond to risks to safetyProtect patients from any risk posed by your health

Prof Dame Lesley Southgate: St Georges Medical School, University of London

Domains of performancefrom Good Medical Practice

Communication, partnership and team work Communicate effectivelyWork constructively with colleagues and delegate

effectivelyEstablish and maintain partnerships with patients

Maintaining trustShow respect for patients and colleaguesTreat patients and colleagues fairly and without

discriminationAct with honesty and integrity

Prof Dame Lesley Southgate: St Georges Medical School, University of London

The domains of performance

together determine the territory within which high level statements of values and principles

(criteria/standards) are set out which then need elaboration in order to map to

actual clinical practice to set levels of achievement (levels/standards

in the other sense) identify the best methods by which evidence of

that achievement can be gathered and judged for a defined purpose

Prof Dame Lesley Southgate: St Georges Medical School, University of London

Headings for a blueprint for recertification

Knowledge, skillsperformance

Safety, quality

CommunicationPartnershipteamwork

Maintaining trust

Curriculum Headings

evidence

Core practice

evidence

Actual practice

evidence evidence

Prof Dame Lesley Southgate: St Georges Medical School, University of London

Criteria and standards

From these, meaningful sub-divisions or criteria can be defined (high level statements that encompass values and principles), leaving the term “standard” as the word to be used to define the level of quality or excellence in relation to each criterion

Prof Dame Lesley Southgate: St Georges Medical School, University of London

And Evidence

The selection of tools to gather evidence will, however, also depend on the characteristics of the different methods, published research in the field of assessment, the likelihood of legal challenge to recommendations perceived by individuals as perverse, and the contribution each tool can make to the provision of evidence that samples the territory described within the domains, in the context of the doctor’s actual practice.

Prof Dame Lesley Southgate: St Georges Medical School, University of London

The utility index

The design of any assessment system will also need to take into account its utility. The utility, or usefulness, of an assessment has been defined as a product of its reliability, validity, feasibility, acceptability and educational impact.

Prof Dame Lesley Southgate: St Georges Medical School, University of London

A programme of workplace assessment should be

Specified (domains, criteria, standards)Developmental Based on the collection of evidence using

an appropriate variety of methodsTriangulatedQuality assured

These principles should be applied to any emerging framework for recertification.

Prof Dame Lesley Southgate: St Georges Medical School, University of London

The importance of context and actual practice in performance assessment Setting defensible standards for performance

assessment is difficult Assessment methods are required that provide

evidence about how doctors respond to real clinical encounters and situations, rather than simulations. 

As each encounter occurs in unique circumstances, assessors may have less data on which to base a decision about the level of performance, whether it be recorded numerically or as a qualitative judgment.

Most performance assessment methods are relatively new, particularly when used for making decisions that might restrict clinical practice. 

Prof Dame Lesley Southgate: St Georges Medical School, University of London

Sources of evidence for recertification of (for example) family physicians will be considered in three sections. Clinical governance data routinely

available (in theory!) from an employer and collected as a result of the doctor’s clinical practice.

Workplace assessment of the individual practitioner

Tests of knowledge and skills

Prof Dame Lesley Southgate: St Georges Medical School, University of London

Attribution

The concept of using performance data from sources other than the assessment of individual doctors is not new. controversial debates about confidentiality and consent, individual

freedoms and clinical autonomy, no single organization or employer is likely to possess sufficient

data to set standards for performance-based measures that reflect the scope of an individual’s clinical practice. 

need a framework for the development of standard-setting procedures that use patient-outcomes-orientated measures of clinical performance, based on a collaborative approach between several health care stakeholders

Prof Dame Lesley Southgate: St Georges Medical School, University of London

Clinical governance data

Early indications have suggested that routinely gathered data across nine areas related to practice might generate some useful insights into practice level performance

Areas 1-4. May be useful for individual assessment: but there are problems with attribution. All to be accompanied by development of

standardised documentation, a commentary and discussion within the appraisal meeting.

Prof Dame Lesley Southgate: St Georges Medical School, University of London

PACT

Prescribing Analysis and Cost (PACT) Data. Data must be attributable to individuals and prescribing in the acute situation, and within defined therapeutic areas is a promising approach. Recommended for piloting

Prof Dame Lesley Southgate: St Georges Medical School, University of London

Referral Information

….. there is conflicting evidence about the usefulness of these data and referral decisions may be better judged by methods proposed later. Not recommended

Prof Dame Lesley Southgate: St Georges Medical School, University of London

Complaints

with a reflective piece to accompany quantitative data in the portfolio of evidence. The definition of a complaint, and it’s separation from informal concerns raised in the course of practice, will be essential. Recommended for piloting

Prof Dame Lesley Southgate: St Georges Medical School, University of London

Significant Events

with specification for types of cases. Standard documentation and a commentary including own contribution to be part of the portfolio of evidence available for appraisal. Recommended for piloting

Prof Dame Lesley Southgate: St Georges Medical School, University of London

Quality and Outcomes Framework (QOF) Data.Area 5 Principally about the

practice/team, but may be useful as a basis for reflection/discussion with appraiser (documentation to be developed)

These data will be difficult to use in their present form. Not recommended

Prof Dame Lesley Southgate: St Georges Medical School, University of London

Practice achievement with the exception reporting tallies

The best candidate for recertification to be used as the basis for a written commentary in the portfolio of evidence available for discussion at appraisal

This is a major area for piloting, and must not be ignored because of political and academic difficulties

Prof Dame Lesley Southgate: St Georges Medical School, University of London

Clinical governance data

we recommend use of the methods proposed above for piloting with volunteer PCTs. To include development of standardised reporting

documentation, guidelines for how to present a commentarytraining for appraisers external review of judgements and

recommendations in order to begin the definition of standards.

Prof Dame Lesley Southgate: St Georges Medical School, University of London

Workplace assessment of the individual practitioner

The range of methods proposed below is included because together they will provide evidence in the domains for recertification/revalidation, to be considered, triangulated and judged with data generated from clinical governance activities.

Prof Dame Lesley Southgate: St Georges Medical School, University of London

Medical record review

We recommend piloting in conjunction with volunteer PCTs and practices.

The work to include survey of present approaches to examining

records during PCT visits to practicesdevelopment of standardised documentationtraining of new, and identification of

experienced reviewers and standard setting and judgments

Prof Dame Lesley Southgate: St Georges Medical School, University of London

Case based discussion (CbD)

We recommend this tool for recertification as part of accredited CPD and with one formal event as part of the 5 year appraisal cycle. Piloting as part of a prototype will be necessary.

Prof Dame Lesley Southgate: St Georges Medical School, University of London

Multi-source feedback

The choice of tools will be based on their evaluation against the utility index. MSF cannot stand alone to provide evidence for recertification. It should be triangulated with other evidence in order to stand legal challenge. Data protection, freedom of information and attention to the provision of feedback are crucial, as is central data management.

Prof Dame Lesley Southgate: St Georges Medical School, University of London

Patient satisfaction questionnaires (PSQ

The review by the Picker Institute Europe 17 has identified five patient feedback questionnaires that have been used within the UK and might be considered (after suitable modification) for use within a framework of recertification.

Prof Dame Lesley Southgate: St Georges Medical School, University of London

Formal Competence testing methods

The possibility of using the certifying examination/methods of assessment on a voluntary basis to satisfy some of the evidential requirements as part of CPD or as evidence to include in their recertification portfolio. This will not be a requirement and will be available only on request. Further discussion will be necessary to determine the weight that might be placed on these tests for doctors who pass them, and what other evidence they may substitute.

Prof Dame Lesley Southgate: St Georges Medical School, University of London

Managed CPD:1

Will be based on GP centred reflection and learning. Linked to “Essential General Practice,” this will be a systematic approach to CPD requiring GPs to demonstrate assimilation of new and evolving knowledge through an e-learning format.

Prof Dame Lesley Southgate: St Georges Medical School, University of London

Managed CPD:2

Some of the tools for WBA will form part of CPD within a programme where the assessor and the doctor are peers, both of whom are participating in an accredited programme to be recorded in the portfolio. As confidence grows in the process formal occasions may be requested to provide evidence of the doctor’s knowledge base.

We recommend that this approach be piloted soon, as we are certain that a minimum data set that depends heavily on MSF and untried clinical governance data will be a high risk strategy

Prof Dame Lesley Southgate: St Georges Medical School, University of London

Synthesis

Prof Dame Lesley Southgate: St Georges Medical School, University of London

Post script: who will judge the evidence?

The training needs of appraisers are not addressed in this presentation. They are considerable, and in particular they will be acute in the arena of feedback. The potential to do harm is greatest in this area, either to individuals or teams. But the appraiser is not the judge and jury for recertification, or indeed revalidation. While they will make judgements, they will be one of many, made both within the evidence assembled for the portfolio, and by other appraisers within the 5 year cycle

Prof Dame Lesley Southgate: St Georges Medical School, University of London