Audit for Registrars

Dr. Ramesh MehayCourse Organiser

Bradford VTSNOTE : Key points = core points to note for any

sytematic approach to audit

Definition

• Clinical audit is the systematic and critical analysis of the quality of clinical care.

• This includes the procedures used for diagnosis and treatment, the associated use of resources and the effect of care on the outcome and quality of life for the patient.

• Clinical Governance = improving standards

• Crombie et al. defined• Audit as the process of reviewing the

delivery of health care to identify deficiencies so that they may be remedied.

• Marinker (1990)• the attempt to improve quality of medical

care by measuring the performance in relation to desired standards and by improving on this performance

Definition – less formal

• Taking note of what we do

• Learning from it

• Changing it if necessary

• With the aim of improving care

Why do It?• Development of professional education and self regulation• Improvement of quality of patient care• Increasing accountability• Improvement of motivation and teamwork• Aiding in the assessment of needs• As a stimulus to research• Clinical audit aims to lead to an improvement in the quality of

service providing:-• improved care of patients • enhanced professionalism of staff • efficient use of resources • aid to continuing education • aid to administration • accountability to those outside the profession

Fundamental Principles

• All about improving patient care

• Should be seen as part of day to day practice

• Developing a critical eye on what we are doing

• Trying to improve things all the time

The Audit Cycle

What Should Be Happening

What Is Happening?

What changes are needed

The Audit Cycle

What Audit Is Not

• Not about:• Performance Appraisal of Staff• Disciplinary Actions• Needs Assessment• Research (which is usually about establishing

new knowledge)• Computers and Statistics• Competition between doctors• “Never judge good and bad professionals based

on audit” – it is about improving care

Audit vs Research

Research Audit

Defines Good Practice Assess extent to which good practice is being obeyed/improvements

May involve allocating pts to random treatment groups

Never involves this

May involve placebo Rx Never involves placebo Rx

Disturbs the pt beyond that required for normal clinical management

Never does this

May involvea completely new treatment

Never involves a completely new treatment

One off process OngoingOther notes

Both audit and Research are concerned with clinical practice effectiveness

Audit can contribute to research – issues that need further exploration

When to Use What

Method When to use it Why

Research Good practice is not defined and comparisons are needed

To define good practice

Data Collection or structured observation

Practice patterns unknown

To catalogue prevailing practice without making judgements

Audit Good practice is defined but we want to know how much we are sticking to it

To improve current performance

Does Audit Lead to Change

• Hearnshaw et al, BJGP 1998

• Of 1257 audits

• Around 80% on clinical care

• Around 65% led to change

Making Audit Easier – Avoid the Blocks

• BEFORE YOU START• Time – big audits can eat up time in an already busy schedule, so :• Keep it simple and small • Look at one or two criteria• Engage the whole team – otherwise it will be difficult! Is the team

ready? (Enthusiasm, wanting to improve)• WHEN YOU START• Delegate & Share the workload – involve others• Make life easier – use computers to do the laborious stuff (patient

searches)• Use protocols / standards already laid by others (why re-invent the

wheel?)• Be careful of data collection – choose a topic which does not entail

too much data collection to the extent it becomes exhaustive with subsequent loss of enthusiasm

Some Ideas• You can do an Audit of• Structure ie facilities being provided• Eg waiting times, availability of staff, record keeping (all patient

records should have a summary card), equipment• Process ie what was done to the patient eg referrals, prescribing,

investigations• Aspirin post MI, BP measurements 5 yearly in those aged 20-65• Outcome ie result for the patient• Eg patient satisfaction, patients with high BP aged between 20-35

should have a diastolic below 90mmHg within the first year of treatment

• high risk practices (significant event audits) eg pneumococcal vaccines in splenectomised patients, are significant events being acted upon?

• The outcome is the ideal indicator for care but the most difficult to measure.

Choosing a Topic• Condition has an important impact on health or of great local

concern KEY POINT ie serious consequences otherwise• Condition affects a large number of people• Good reasons for believing current performance can be improved or

improvements are needed KEY POINT• Convincing evidence about appropriate care is available• Data collection – choose a topic which does not entail too much

data collection to the extent it becomes exhaustive with subsequent loss of enthusiasm (? A pilot??)

• CHOOSE SOMETHING THAT REALLY INTERESTS YOU• NO POINT AUDITING SOMETHING YOU THINK THE PRACTICE

IS DOING REALLY WELL• Then discuss with others – are they interested too?

Choosing a Topic

• Remember, topic should be important :

• Chronic Disease Management eg referrals or use of lab services (INR’s in warfarin)

• Preventative Care eg childhood imms, Cervical Cytology

• Prescribing eg aspirin post MI, PPI’s (cost issue)

Examples• Ways of spotting audit topics                examples• Important clinical events                         admissions for asthma• “Significant events”                                 patient died of MI – no

record of smoking history or BP• Patients' complaints                                too long to get an

appointment• Observation                                             no system for ensuring bag

drugs up to date• Observations of staff                               patient on Warfarin not had

INR for 6 months• NICE subjects                                          post-MI patients on aspirin

Criteria

• = yardsticks• “An audit criterion is a specific statement of what

should be happening.” • A statement which • A) defines a measurable item of health care

which• B) can be used to assess quality• KEYPOINT

Criteria should be explicit. You must demonstrate evidence for justifying them (literature search, Evidence Based!).

Criteria – KEY POINTS

• Ensure that the criterion is measurable – • ·            “asthmatics should have had yearly PFs” is

difficult to measure (how many years will you go back?);• ·            “asthmatics should have had a PF recorded in

the past year” is more practical.• Don’t try to audit too many criteria at once – one or two

will keep you busy enough. • Try filling in the gaps of the following phrase to set your

audit criterion: • “All patients with xxxxx should have had a xxxxx in the

last xxxxx.”

Criteria

• "All eligible women aged 25-65 should have had a cervical smear in the last 5 years."

• “All asthmatics should have had a Peak Flow recorded in the past year.”

• “All drugs in our doctors’ bags should be in-date.”

Standards

• “An audit standard is a minimum level of acceptable performance for that criterion.”

• Make sure the standard is directly related to the criterion, also :-

• Should include a suitable timeframe

Standards

• → Examples: • "At least 80% of eligible women aged 25-65 should have

had a cervical smear in the last 5 years."• “At least 60% of asthmatics should have had a Peak

Flow recorded in the past year.”• “100% of drugs in our doctors’ bags should be in-date.”•  • The standard should reflect the clinical and medico-legal

importance of the criterion. • in the example above, 80% of women should have had a

cervical smear,• But of those who've had an abnormal smear, 100%

should have had action taken.

Standards

• How to set standards• Look at national guidelines• Literature (journals), textbooks• Local guidelines• Discussion with consultants/GPSI’s• Discussion with trainer/partners• KEY POINT : Standards set should be realistic

and attainable. Justifiable reasons for the standard set should be made explicitly clear.

Standards

• Some criteria are so important that they need 100% standard.

• However, 100% standards are unusual – patients or circumstances usually conspire against perfection and the standard needs to reflect that.

• Your literature search should give you an idea of what standards others have managed to reach.

• Your standard needs to follow on directly from your criterion – for example,

• “Patients on thyroxine should have had TFTs done in the last year; this should have happened in at least 90% of patients”.

TYING IT ALL TOGETHERExamples of Standards & Criteria

Criteria Standards

All children under 2 years should be immunised against tetanus and polio

90% of registered patients under the age of 2 years should have been immunised against polio and tetanus

All notes of those patients with an allergy to penicillin should be marked

95% of patients with an allergy to penicillin should be clearly marked

All patients in the surgery should wait no longer than 30 minutes before a consultation

70% of patients in the surgery should wait no longer than 30 minutes before a consultation

Preparation & Planning

• Must show evidence of teamwork – otherwise you will fail

Data Collection (1)

• You can collect information from:

• computer registers

• review of contents of medical records

• questionnaires – patients, staff or GPs

• data collection sheets

Data Collection

• Be careful of data collection – choose a topic which does not entail too much data collection to the extent it becomes exhaustive with subsequent loss of enthusiasm

• ? Sampling – random or systemic• Only collect essential information• Use computers, ?data collection forms• Use other staff & delegate – don’t do all the work

yourself• Set a deadline

Presenting the Results

• Collect Results• Analyse Results• Summarise Results• Present Results to the team• Simple arithmetic calculations• Use percentages• Results of 2nd data collection presented in

the same way as the 1st

Discussion – Data Collection 1

Comparing Results to Standards

Criterion Standard Observed Result

All patients should be seen within 15 minutes of their appointment timeMinimum

70% 45%

70%45%All diabetics to have had HbA1C in last 3 monthsMinimum

95% 90%

90%Drug allergies to be marked as “active problem” on computerMinimum

100% 95%

Discussion – Data Collection (1)

• KEY POINT (Discussion of Data Collection 1) : You need to explain why you think the practice didn't meet the standard that was set.

Discussion – why standards not met

• Think: What reasons are there for practices not meeting audit standards?

•  • For example : reasons have included:• Practice reasons: • ·         Results having been put down as free text on computer,

rather than coded;• ·         Opportunistic rather than formal recall system in use;• Doctor reasons: • ·         Not all GPs were aware of the practice policy;• ·         Not all partners agreed with the policy;• Patient reasons: • ·         Patients refusing to have tests done;• ·         Patients on holiday when tests due.

Implementing Changes

• The most challenging stage

• Audit can tell you whether changes are needed, but it can’t tell you what methods to use

Implementing Change

• The changes to be implemented should be a team discussion and decision (?a practice meeting)

• What to do at the Practice Meeting:

• Emphasise what has been achieved.

• What are we proud of?

• What are we not so proud of?

• How can we correct any deficiencies?

Implementing Change

• Changes must be practical!• How are you actually going to make the changes?• Simply saying “We’ve got to do better” won’t result in

change • You need to think through in detail • ·         what needs to be done• ·         who’s going to do it• ·         when• ·         and how.• If you get very low results, you may consider resetting

the standards to a more realistic level (but justify it)

Implementing Change• KEYPOINT• Just telling people to do things better won't result in

change. You need to write up in some detail how the changes will take place.

• FAIL Example:    "The GPs agreed to do a serum rhubarb on any patient that they see who is on Viagra" - fail - this wouldn't be likely to pass, as there is no system to help them remember.

• PASS Example :  • ”(a) The GPs were given a prompt card that they could stick on their

computer screen as a reminder to do a serum rhubarb on any patient that they see who is on Viagra;

• (b) the secretary will search every three months for patients who are overdue for their serum rhubarb, and flag it as an active problem on the computer system" - pass - as it should result in change.

Closing the Loop

• Ie repeating the cycle 

• Re-evaluate care to ensure that any remedial action has been effective.

• Audit is a continuous cycle – if you didn’t meet the standard and you’ve planned changes, you’ll need to repeat the audit to make sure the changes have happened.

Conclusions from the Audit

•   Summary of main issues learned•    KEYPOINTS: • Comment on any improvements that have resulted. • How well did your proposals for change work?• If you again didn't reach the standard that you set, why

not? • If you did, should you be aiming higher next time, or look

at something else e.g. whether abnormal serum rhubarbs have actually been acted on?

• Where should the practice go from here

Useful Resources

• MAAGs – medical audit advisory groups • Clinical Governance Advisory Groups• National/Local Guidelines• RCGP database of simple tested audits for

day-to-day use• Literature, Books• The WWW• Consultants, GPSI’s, Trainers, Partners

How To Fail• No justification for choice of audit• No justification for criteria/standard settings • Not having explicit criteria/standards• Setting unreasonable standards• A general lack of evidence based literature or using material that is not peer

referenced• Not explicitly displaying teamwork in the “method” – must give specific

examples• Numerical errors re: data collection • Presentation of data collection eg no graphs, no percentages (ie the reader

has to do the hardwork him/herself)• Not giving much thought to “changes to be evaluated” and not being specific

enough. Not delegating specific changes to specific people/persons. • Poor conclusions and what the process has taught you• No inclusion for possible sources of bias• References not properly quoted

IF YOU DON’T WANT TO FAIL

• Go through the following online tutorial

• http://www.mharris.eurobell.co.uk

• Look at the Marking Schedule – (yes, they provide you with an answer sheet!)

• www.mharris.eurobell.co.uk/marking.htm

• You must pass on all 8 criteria.

SHO’s doing Audit for Summative Assessment

• If you are doing the audit while an SHO, you need to choose a topic that looks at the GP-hospital interface. Referrals or discharge letters are possible areas for audit. Again, you need to demonstrate that you've found a problem that needs to be investigated.

•             I suggest that you discuss your proposed audit with your GP Scheme Organiser before you go ahead - your hospital colleagues may not know what's needed for Summative Assessment.

Checking GPR Understanding• DISCUSS THE FOLLOWING STATEMENTS• An example of the Audit of process is audit of referrals to hospitals.• Audit usually consumes an extensive amount of resources (of time, money

etc.).• Rare conditions should be audited.• The higher the standard the practitioner starts with, the stronger is the

resulting audit.• Maintaining clearly written notes of at least 20% of patients who are

sensitive to penicillin is an acceptable standard in general• practice.• The higher the amount of data the practitioner collects, the easier is the

decision making process in audit.• The most challenging stage in Audit is implementing change.• In data collection all in the target population must be included.• The agreed standards can be reset at realistic percentages after the first

round of data collection.

Clinical Audit Association Ltd

• Clinical Audit Association LtdCleethorpes CentreJackson PlaceWilton RoadHunbertonLincolnshire DN36 4AS

• Tel: 01472 210 682http://www.the-caa-ltd.demon.co.uk

Clinical Governance Research and Development Unit

• Dept of General Practice and Primary Health CareUniversity of LeicesterLeicester General HospitalGwendolen RdLeicester LE5 4PW

• Tel: 0116 258 4873Fax: 0116 258 4982email: cgrdu@le.ac.uk

• http://www.le.ac.uk/cgrdu

Cochrane Database of Systematic Review

• 020 7383 6185c/oBritish Medical AssociationBMA HouseTavistock SquareLondon WC1H 9JP

NICE

• 11 StrandLondonWC2N 5HR

Tel: 020 7766 9191Fax: 020 7766 9123

http://www.nice.org.uk

RCGP Effective Clinical Practice Unit

• School of Health and Related ResearchRegent Court30 Regent StreetSheffield S1 4DA

• Tel: 0114 222 5454Fax: 0114 272 4095Email: scharrlib@sheffield.ac.uk

• http://www.shef.ac.uk/~scharr/

RCGP NE Scotland Faculty

• The Primary Care Resource CentreForesterhill RoadAberdeen AB25 2ZP

• Tel: 01224 558 042Fax: 01224 558 047

• Email: rcgp@pcrc.grampian.scot.nhs.uk

• http://www.rcgp.org.uk/rcgp/faculties/nescot/index.asp

UK Cochrane Centre

• Dr Iain Chalmers, DirectorNHS Research and Development ProgrammeSummertown PavilionMiddle WayOxford OX2 7LG

Tel: 01865 516300

Cochrane Collaboration in the field of primary care

• For information concerning work by theCochrane Collaboration in the field of primary care, contact:Dr Lorne BeckerProfessor and ChairDept of Family MedicineSUNY Upstate Medical University475 Irving AvenueSyracuse, NY 13210USATel: +1 315 464 7010Fax: +1 315 464 6982E-mail: beckerla@hscsyr.edu http://www.update-software.com/ccweb/default.html

NHS Centre for Reviews and Dissemination

• University of YorkYork, YO1 5DD

• Tel: 01904 433 634Fax: 01904 433 661Email: revdis@york.ac.uk

• http://www.york.ac.uk/inst/crd

Sources

• This power point has been derived from :

• http://www.mharris.eurobell.co.uk

• http://kims.org.kw/bulletin/Issues/issue2/Audit.pdf