National Approach to Practice - NES...assessing clinical practice in nursing students [Spanish]."...

Page 1 of 106

National Approach to Practice

Assessment for Nurses

and Midwives

Exploring Issues in the use of

Grading in Practice:

Literature Review

Final Report

(Volume 2 Appendices)

September 2009

Contact person: Prof. Morag A. Gray, Associate Dean (Academic Development), Faculty of

Health, Life & Social Sciences, Merchiston Campus, Colinton Road, Edinburgh EH10 5DT

Tel: 0131 455 2465 [email protected]

mailto:[email protected]

Page 2 of 106

National Approach to Practice Assessment for

Nurses and Midwives

Exploring issues in the use of Grading in Practice:

Literature Review

Final Report

(Volume 2 Appendices)

Prof. Morag A. Gray

Dr. Jayne Donaldson

September 2009

Page 3 of 106

Contents

Page

Final Report: Volume 2

Executive Summary 4

Appendix 1 List of search criteria and databases used in electronic searching

6

Appendix 2a Reference list from the primary search 7

Appendix 2b Reference list from hand search 17

Appendix 3 Overview of excluded studies 21

Appendix 4 Overview of included studies 53

Page 4 of 106

Executive Summary

This literature review focuses on exploring issues of grading in practice, including

reliability, validity and the implications for mentor preparation and support.

Recommendations are made to inform the National Approach Working Group in

formulating guidance to Higher Education Institutions around the grading of practice.

Literature review

Literature was collected following a systematic search and spanned across fourteen

professional groups. In total a 119 articles were reviewed following application of

exclusion criteria. Of these, the majority were quantitative in nature (n=66); followed

by descriptive accounts or opinion based (n=28); literature reviews (n=19) and

qualitative (n=6).

Challenges in grading practice

Grading practice reflects the conclusion of a decision making process which

indicates how well a student is progressing in respect to a standard or criteria and

flags up areas where students can improve.

There are a number of documented challenges when grading practice. Some of the

challenges such as time available, consistency and accountability of assessors are

not specific to only grading practice. Those which are specific to grading relate to

validity and reliability issues of the tools used and grade inflation.

Grading Tools

The review of the literature presents a number of grading tools or systems, and we

conclude that evaluation of these tools is under-developed in terms of their

effectiveness, usefulness, reliability and validity. Grade inflation is a well-

documented problem and there are some suggested methods (although not fully

evaluated) from the literature to control this. The most promising of those appears to

be the use of rubrics and focused training of assessors.

Rubrics

Carefully constructed rubrics can ameliorate grade inflation. Rubrics are made up of

three key components: clearly defined performance criteria or elements; detailed

descriptions of what a performance looks like at each level (or grade) of proficiency;

and a rating scale which most commonly uses a three or four points. Rubrics can be

used both formatively and summatively.

Page 5 of 106

Training of assessors / mentors

If nursing and midwifery were to use grading within practice, there are a number of

training requirements suggested by the literature for mentors: developing an

understanding and interpretation of educational terminology; how to accurately

assess learning; how to assign grades; how to use the tools consistently and

effectively; how to write evidence to support the graded assessment given; how to

deal with borderline students; and how to deliver effective feedback. The time and

methods of training and updating mentors needs to be considered.

Conclusions and recommendations

From the literature reviewed, it is not possible to make any generalisations about

grading of practice. The findings from this literature review come from a wide source

of professional literature that often include limitations of being set in one

geographical area, in one profession, and calculated on relatively small sample sizes.

Most quantitative studies were survey design, and /or presented using descriptive

statistics only. Therefore the generalisability of findings from these quantitative

studies should be necessarily cautious. Only six studies demonstrated the use of a

qualitative theoretical basis for the study. A number of limitations were noted and

could be prone to bias. The literature review articles were mostly descriptive

accounts, of for example a grading tool, or an opinion on grading, or on a developed

tool. It should be borne in mind that opinion can be subject to writer bias. The level of

evidence on grading of clinical practice across the literature tends to sway towards

the lower end of the evidence band i.e. the usefulness, reliability, validity and

effectiveness of grading of practice has still to be proven.

If the grading of practice is adopted, we recommend that consideration is given to

the following:

The development, testing and use of rubrics;

The use of rubrics for formative as well as summative assessment;

The use of a multi-method approach to assessment;

Comprehensive training and updating sessions for assessors;

Ongoing evaluation and monitoring of the grading process used.

Page 6 of 106

Appendix 1: List of search criteria and databases used in electronic searching

Search terms used:

Mentorship / mentor*

Preceptorship / preceptor*1

Competency Assessment

Clinical Competence – Evaluation

Grading of practice / practice grading / grading tool*

Practice assessment / assessment tool*

Rubric*

Education, Clinical – Evaluation

Educational Measurement

Student Performance Appraisal

Student Satisfaction -- Evaluation

Students, Nursing – Evaluation Students, Midwifery – Evaluation

Students, AHP – Evaluation

Students, Medicine - Evaluation

Databases used:

CINAHL

British Nursing Index

Medline

ASSIA

British Education Index

AMED

EMBASE

Australian Education Index

ERIC

Index to Theses

CERUK (Current Educational Research in the UK)

Directory of Open Access Journals

SCIRUS

Google Scholar

1 *= truncated search term seeking all variations of a word

Page 7 of 106

Appendix 2a: Reference list from the primary search

Abbey, H. (2008). "Assessing clinical competence in osteopathic education: analysis of outcomes of different assessment strategies at the British School of Osteopathy." International Journal of Osteopathic Medicine 11(4): 125-131.

Albi GarcÃa, J., M. D. CaamaÃ±o Vaz, et al. (2003). "Development of a tool for assessing clinical practice in nursing students [Spanish]." Enfermeria Clinica 13(3): 146-153.

Allen, P., K. Lauchner, et al. (2008). "Evaluating continuing competency: a challenge for nursing." Journal of Continuing Education in Nursing 39(2): 81-85.

Andre, K. (2000). "Grading student clinical practice performance: the Australian perspective." Nurse Education Today 20(8): 672-679.

Bartlett, H., V. Simonite, et al. (2000). "A comparison of the nursing competence of graduates and diplomates from UK nursing programmes." Journal of Clinical Nursing 9(3): 369-381.

Baulcomb, S. and R. Watson (2003). "The internal consistency and intra-rater reliability of an assessment practice grid for community nurses." Clinical Effectiveness in Nursing 7(3/4): 168-170.

Ben, D. M. F., D. Snadden, et al. (2004). "Linking appraisal of PRHO professional competence of junior doctors to their education." Medical Teacher 26(1): 63-70.

Bogo, M., C. Regehr, et al. (2006). "Beyond Competencies: Field Instructors' Descriptions of Student Performance." Journal of Social Work Education, 42(3): 579-593.

Boley, P. and K. Whitney (2003). "Grade disputes: considerations for nursing faculty." The Journal Of Nursing Education 42(5): 198-203.

Brown, N. (2000). "What are the criteria that mentors use to make judgements on the clinical performance of student mental health nurses? An exploratory study of the formal written communication at the end of clinical nursing practice modules." Journal of Psychiatric & Mental Health Nursing 7(5): 407-416.

Buckingham, S. (2000). "Clinical competency: the right assessment tools?" Journal of Child Health Care 4(1): 19-22.

Cacamese, S. M., M. Elnicki, et al. (2007). "Grade inflation and the internal medicine subinternship: a national survey of clerkship directors." Teaching & Learning in Medicine 19(4): 343-346.

Calman, L., R. Watson, et al. (2002). "Assessing practice of student nurses: methods, preparation of assessors and student views." Journal of Advanced Nursing 38(5): 516-523.

Cassidy, S. (2009). "Interpretation of competence in student assessment." Nursing Standard 23(18): 39-46.

Chambers, D. W. (1999). "Faculty ratings as part of a competency-based evaluation clinic grading system." Evaluation and the Health Professions 22(1): 86-106.

Chambers, M. A. (1998). "Some issues in the assessment of clinical practice: a review of the literature." Journal of Clinical Nursing 7(3): 201-208.

Ciolfi, M. (2009). "Clinical Competencies Assessment Rubric System (C-CARS)." Journal of Chiropractic Education 23(1): 62-62.

Page 8 of 106

Clouder, L. (2005). "An Evaluation of the Validity of Assessment Strategies Used to Grade Practice Learning in Undergraduate Physiotherapy Students: Final Report to the Health Science and Practice Subject Centre of the Higher Education Academy." London: HEA. Available from: http://www.health.heacademy.ac.uk/projects/miniprojects/clouder.pdf [accessed 15 July 2009]

Coote, S., L. Alpine, et al. (2007). "The development and evaluation of a Common Assessment Form for physiotherapy practice education in Ireland." Physiotherapy Ireland 28(2): 6-10.

Cowan, D. T., I. J. Norman, et al. (2005). "Nurse competency. A project to establish a skills competency matrix for EU nurses." British Journal of Nursing (BJN) 14(11): 613-617.

Cowan, D. T., J. Wilson-Barnett, et al. (2007). "A European survey of general nurses' self assessment of competence." Nurse Education Today 27(5): 452-458.

Cowan, D. T., J. Wilson-Barnett, et al. (2008). "Measuring nursing competence: development of a self-assessment tool for general nurses across Europe." International Journal of Nursing Studies 45(6): 902-913.

Cowin, L. S., C. Hengstberger-Sims, et al. (2008). "Competency measurements: testing convergent validity for two measures." Journal of Advanced Nursing 64(3): 272-277.

Cox, K. (2000). "Examining and recording clinical performance: a critique and some recommendations." Education for Health 13(1): 45-52.

Crookes, P. (2007). Development of an undergraduate nursing competencies assessment tool, for use across Australian Universities.. Sydney: Australian Learning and Teaching Council. Available from: http://www.altc.edu.au/project-

development-undergraduate-uow-2007 [accessed 15 July 2009] Cross, V., C. Hicks, et al. (2001). "Comparing the Importance of Clinical Competence

Criteria across Specialties: Impact on undergraduate assessment." Physiotherapy 87(7): 351-367.

Cruess, R., J. H. McIlroy, et al. (2006). "The Professionalism Mini-evaluation Exercise: a preliminary investigation." Academic Medicine: Journal Of The Association Of American Medical Colleges 81(10 Suppl): S74-8.

Dalton, M., J. Keating, et al. (2009). "Development of the Clinical Assessment of Physiotherapy Skills (CAPS): A standardised and valid approach to assessment of clinical competence in physiotherapy. Sydney: Australian Learning and Teaching Council. Available from: http://www.altc.edu.au/resource-development-clinical-assessment-griffith-2009 [accessed 15 July 2009]

Darra, S. (2006). "Assessing midwifery practice form the mentor's perspective." British Journal of Midwifery 14(8): 458-461.

Darra, S., B. Hunter, et al. (2003). "Education. Developing a midwifery skills framework." British Journal of Midwifery 11(1): 43-47.

Davies, H., J. Archer, et al. (2008). "Specialty-specific multi-source feedback: assuring validity, informing training." Medical Education 42(10): 1014-1020.

Davies, H., J. Archer, et al. (2008). "Specialty-specific multi-source feedback: assuring validity, informing training." Medical Education 42(10): 1014-1020.

Dillard, N., S. Sideras, et al. (2009). "A collaborative project to apply and evaluate the clinical judgment model through simulation." Nursing Education Perspectives 30(2): 99-104.

http://www.health.heacademy.ac.uk/projects/miniprojects/clouder.pdf

http://www.altc.edu.au/project-development-undergraduate-uow-2007

http://www.altc.edu.au/project-development-undergraduate-uow-2007

http://www.altc.edu.au/resource-development-clinical-assessment-griffith-2009

Page 9 of 106

Dudek, N. L., M. B. Marks, et al. (2008). "Assessing the quality of supervisors completed clinical evaluation reports." Medical Education 42(8): 816-822.

Duffy, K. and J. Hardicre (2007). "Supporting failing students in practice 1: assessment." Nursing Times 103(47): 28-29.

Duffy, K. and J. Hardicre (2007). "Supporting failing students in practice, part 2: management." Nursing Times 103(48): 28-29.

ElBadrawy, H. and M. Korayem (2007). "The flexible requirement system for grading of clinical performance of undergraduate dental students." European Journal Of Dental Education: Official Journal Of The Association For Dental Education In Europe 11(4): 208-215.

Erik, W. D., O. Karlijn, et al. (2006). "Validity of portfolio assessment: which qualities determine ratings?" Medical Education 40(9): 862-866.

Everett, S. (2007). Grading of practice portfolios by Practice Teachers at the University of Northampton. London: HLSP. Available from: http://www.hlsp.org/files/page/180617/Grading_of_practice_portfolios.pdf [accessed 15 July 2009]

Feeley, T. H., A. T. Manyon, et al. (2003). "Toward validation of an assessment tool designed to measure medical students' integration of scientific knowledge and clinical communication skills." Evaluation & The Health Professions 26(2): 222-233.

Finnerty, G., L. Graham, et al. (2006). "Empowering midwife mentors with adequate training and support." British Journal of Midwifery 14(4): 187-190.

Fisher, M. and M. Parolin (2000). "The reliability of measuring nursing clinical performance using a competency based assessment tool: a pilot study." Collegian 7(3): 21-27.

Fitzgerald, L. M., A. Delitto, et al. (2007). "Validation of the Clinical Internship Evaluation Tool." Physical Therapy 87(7): 844-860.

Flanagan, J., S. Baldwin, et al. (2000). "Work-Based Learning as a means of developing and assessing nursing competence." Journal of Clinical Nursing 9(3): 360-368.

Fletcher, P. (2008). "Clinical competence examination -- improvement of validity and reliability." International Journal of Osteopathic Medicine 11(4): 137-141.

Fordham, A. J. (2005). "Using a competency based approach in nursing education." Nursing Standard 19(31): 41-48.

Frenk, J. and A. Donabedian (2000). "Evaluating physician competence." Bulletin of the World Health Organization 78(6): 857-60.

Gardner, G., J. Carryer, et al. (2006). "Nurse Practitioner competency standards: Findings from collaborative Australian and New Zealand research." International Journal of Nursing Studies 43(5): 601-610.

Gill, F., G. Leslie, et al. (2006). "Evaluation of a clinical performance assessment tool (CPAT) within a critical care context." Australian Critical Care 19(3): 105-113.

Girot, E. (2000). "Assessment of graduates and diplomates in practice in the UK: are we measuring the same level of competence?" Journal of Clinical Nursing 9(3): 330-337.

Glass, N. and L. Ward (2008). "Chapter 8. Advancing clinical nursing education in mental health: student self-assessment of clinical competencies." Annual Review of Nursing Education 6: 151-172.

http://www.hlsp.org/files/page/180617/Grading_of_practice_portfolios.pdf

Page 10 of 106

Glavin, R. J. and N. J. Maran (2002). "Editorial I. Development and use of scoring systems for assessment of clinical competence." British Journal of Anaesthesia 88(3): 329-330.

Glover, P., E. Ingham, et al. (1997). "The development of an evaluation tool for grading clinical competence." Contemporary Nurse: A Journal for the Australian Nursing Profession 6(3/4): 110-116.

Gopee, N. (2000). "Self-assessment and the concept of the life long learning nurse." British Journal of Nursing 9(11): 724-729.

Gopee, N. (2008). "Assessing student nurses' clinical skills: the ethical competence of mentors." Int J Therapy & Rehabilitation 15(9): 401-407.

Gordon, T. and D. Mitchell (2004). "A competency model for the assessment and delivery of spiritual care." Palliative Medicine 18(7): 646-651.

Grant, J. M., Mairead; Chambers, Katrina; Marshall, Joanne. (2007). "Workplace based assessments in medicine." Milton Keynes: Open University, Practice-based Professional Learning Centre. Available from: http://www.open.ac.uk/cetl-workspace/cetlcontent/documents/475bc732b6bad.pdf [accessed 15 July 2009]

Gray, J. and C. Bradshaw (2007). "Assessing general practitioners who may be underperforming: local assessment methods in two English health districts." Quality in Primary Care 15(3): 157-164.

Gupta, P. and P. Varkey (2009). "Developing a tool for assessing competency in root cause analysis." Joint Commission Journal on Quality & Patient Safety 35(1): 36-42.

Hale, C., T. Long, et al. (2006). "An evaluation of educational preparation for cancer and palliative care nursing for children and adolescents: issues in the assessment of practice arising from this study." Nurse Education Today 26(7): 528-537.

Hand, H. (2006). "Assessment of learning in clinical practice." Nursing Standard 21(4): 48.

Hanley, E. and A. Higgins (2005). "Assessment of clinical practice in intensive care: a review of the literature." Intensive & Critical Care Nursing 21(5): 268-275.

Hanley, E. and A. Higgins (2005). "Assessment of practice in intensive care: students' perceptions of a clinical competence assessment tool." Intensive & Critical Care Nursing 21(5): 276-283.

Haroon-Iqbal, H. and A. Jinks (2002). "Peer review of clinical education assessment: results of the Clinical Mentors Survey " Singapore Nursing Journal 29(2): 8-12.

Hemmer, P. A., K. K. Papp, et al. (2008). "Evaluation, grading, and use of the RIME vocabulary on internal medicine clerkships: results of a national survey and comparison to other clinical clerkships." Teaching & Learning in Medicine 20(2): 118-126.

Hewitt-Taylor, J. (2002). "Students' views on the assessment processes used in paediatric intensive care nursing courses." Intensive & Critical Care Nursing 18(1): 56-63.

Hill, L. H., J. C. Delafuente, et al. (2006). "Development of a competency-based assessment process for advanced pharmacy practice experiences." Am J Pharm Educ 70(1): 1.

http://www.open.ac.uk/cetl-workspace/cetlcontent/documents/475bc732b6bad.pdf

Page 11 of 106

Holaday, S. D. and K. M. Buckley (2008). "Chapter 7. A standardized clinical evaluation tool-kit: improving nursing education and practice." Annual Review of Nursing Education 6: 123-149.

Holmes, V. (1997). "Grading journals in clinical practice: a delicate issue." Journal of Nursing Education 36(10): 489-492.

Howley, L. D. (2004). "Performance assessment in medical education: Where we've been and where we're going." Evaluation and the Health Professions 27(3): 285-303.

Hyatt, S. A., L. Brown, et al. (2008). "Supporting mentors as assessors of clinical practice." Nursing Standard 22(25): 35-41.

Iramaneerat, C. and R. Yudkowsky (2007). "Rater errors in a clinical skills assessment of medical students." Evaluation & the Health Professions 30(3): 266-283.

Isaacson, J. J. and A. S. Stacy (2009). "Rubrics for clinical evaluation: objectifying the subjective experience." Nurse Education in Practice 9(2): 134-140.

Johnson, M. (2007). "Is passing just enough? Some issues to consider in grading competence- based assessments." Research Matters(3): 27-30.

Johnson, M. (2008). "Grading in competence-based qualifications - is it desirable and how might it affect validity?" Journal of Further and Higher Education 32(2): 175-184.

Knight, P. P., Anna. (2007). Assessment of „wicked‟ competences: report to the Practice-based Professional Learning Centre. Milton Keynes: Open University, Practice-based Professional Learning Centre. Available from: http://kn.open.ac.uk/public/getfile.cfm?documentfileid=10623 [accessed 15 July 2009]

Kogan, J. R., L. M. Bellini, et al. (2003). "Feasibility, reliability, and validity of the mini-clinical evaluation exercise (mCEX) in a medicine core clerkship." Academic Medicine: Journal Of The Association Of American Medical Colleges 78(10 Suppl): S33-5.

Lanphear, J. H. (1999). "In support of grading systems." Education for Health 12(1): 79-83.

Lasater, K. (2007). "Clinical judgment development: using simulation to create an assessment rubric." Journal of Nursing Education 46(11): 496-503.

Lauder, W., R. Watson, et al. (2008). "An evaluation of fitness for practice curricula: self-efficacy, support and self-reported competence in preregistration student nurses and midwives." Journal of Clinical Nursing 17(14): 1858-1867.

Leach, M. J. (2007). "Revisiting the evaluation of clinical practice." International Journal of Nursing Practice 13(2): 70-74.

Levett-Jones, T. (2008, Oct 2008). "'Real-world' assessment for nursing students." Sydney: Australian Learning and Teaching Council. Available from http://www.altc.edu.au/october2008-real-world-assessment [accessed 15 July 2009]

Lewis, L. K., K. Stiller, et al. (2008). "A clinical assessment tool used for physiotherapy students -- is it reliable?" Physiotherapy Theory & Practice 24(2): 121-134.

Lockyer, J. (2003). "Multisource feedback in the assessment of physician competencies." Journal of Continuing Education in the Health Professions 23(1): 4-12.

http://kn.open.ac.uk/public/getfile.cfm?documentfileid=10623

http://www.altc.edu.au/october2008-real-world-assessment

Page 12 of 106

London, S. (2008). "The assessment of clinical practice in osteopathic education: is there a need to define a gold standard?" International Journal of Osteopathic Medicine 11(4): 132-136.

Long, T. and J. Asbury (2000). "A tripartite approach to assessing practice." Journal of Child Health Care 4(1): 35-38.

Luhanga, F., O. J. Yonge, et al. (2008). ""Failure to assign failing grades": issues with grading the unsafe student." International Journal of Nursing Education Scholarship 5(1): 1-13.

Lunney, M., E. Gigliotti, et al. (2007). "Tool development for evaluation of clinical nurse specialist competencies in graduate students: a pilot study." Clinical Nurse Specialist: The Journal for Advanced Nursing Practice 21(3): 145-151.

Lyons, N. (2002). "Clinical competence: a review of methods used to assess competence and proposals for a realistic future strategy." Education for Primary Care 13(3): 326-335.

Mallik, M. and B. McGowan (2007). "Issues in practice based learning in nursing in the United Kingdom and the Republic of Ireland: results from a multi professional scoping exercise." Nurse Education Today 27(1): 52-59.

Maxted, M., J. Grant, et al. (2004). Assessing Healthcare Students in Practice Placements: An Overview of the Literature. Edinburgh, Association for the Study of Medical Education (ASME).

McAdams, C. C. and K. A. Montgomery (2003). "Narrowing the possibilities: using quality improvement tools to decrease competence assessment overload." Journal for Nurses in Staff Development 19(1): 40-46.

McCarthy, B. and S. Murphy (2008). "Assessing undergraduate nursing students in clinical practice: do preceptors use assessment strategies?" Nurse Education Today 28(3): 301-313.

McCready, T. (2007). "Portfolios and the assessment of competence in nursing: a literature review." International Journal of Nursing Studies 44(1): 143-151.

McGrath, P., J. Anastasi, et al. (2006). "Collaborative voices: ongoing reflections on nursing competencies." Contemporary Nurse: A Journal for the Australian Nursing Profession 22(1): 46-58.

McKinley, D. W., J. R. Boulet, et al. (2005). "A work-centered approach for setting passing scores on performance-based assessments." Evaluation & The Health Professions 28(3): 349-369.

McKinley, R. K., J. Strand, et al. (2008). "Checklists for assessment and certification of clinical procedural skills omit essential competencies: a systematic review." Medical Education 42(4): 338-349.

McLean, C., E. Monger, et al. (2005). "Assessment of practice using the National Health Service Knowledge and Skills Framework." Nursing in Critical Care 10(3): 136-142.

Molenaar, W. M. (2004). "Written case reports as assessment of the elective student clerkship: consistency of central grading and comparison with ratings of clinical performance." Medical Teacher 26(4): 301-304.

Munro, N., K. Hawthorne, et al. (2006). "Developing a new Clinical Skills Assessment (CSA) for licensing UK general practitioners: the why, the how and the when." Education for Primary Care 17(4): 301-310.

Murray, E., L. Gruppen, et al. (2000). "The accountability of clinical education: its definition and assessment." Medical Education 34(10): 871-879.

Page 13 of 106

Murray, S., T. Higgs, et al. (1999). "Assessment of competence in radiography education." Assessment and Evaluation in Higher Education 24(3): 315-326.

Neary, M. (2000). "Responsive assessment of clinical competence: part 1." Nursing Standard 15(9): 34-36.

Neary, M. (2000). "Responsive assessment of clinical competence: part 2." Nursing Standard 15(10): 35-40.

Neary, M. (2001). "Assessing the clinical competence of students in nurse education." Welsh Journal of Education 10(1): 46-58.

Neary, M. (2001). "Responsive assessment: assessing student nurses' clinical competence." Nurse Education Today 21(1): 3-17.

Nicholson, P., S. Gillis, et al. (2009). "The use of scoring rubrics to determine clinical performance in the operating suite." Nurse Education Today 29(1): 73-82.

Nielsen, A., S. Stragnell, et al. (2007). "Educational innovations. Guide for reflection using the clinical judgment model." Journal of Nursing Education 46(11): 513-516.

Norcini, J. J. (2007). Workplace-based assessment in clinical training. Edinburgh, Association for the Study of Medical Education.

Norman, I., R. Watson, et al. (2002). "The validity and reliability of methods to assess the competence to practise of pre-registration nursing and midwifery students." International Journal of Nursing Studies 39(2): 133-145.

Nursing and Midwifery Council. (2007). "Grading of clinical practice for pre-registration midwifery education programmes."(NMC Circular 25]). London: NMC Available from http://www.nmc-

uk.org/aDisplayDocument.aspx?documentID=3198 [accessed 15 July 2009] Nursing and Midwifery Council. (2008). "Standards to support learning and

assessment in practice: NMC standards for mentors, practice teachers and teachers." 2nd ed. . London: NMC. Available from: http://www.nmc-uk.org/aDisplayDocument.aspx?documentID=4368 [accessed 15 July 2009]

O'Connor, S., J. Pearce, et al. (2001). "An evaluation of the clinical performance of newly qualified nurses: a competency based assessment." Nurse Education Today 21(7): 559-568.

Ohlen , J. and G. Erixon (2005). "An assessment tool for clinical nursing education [Swedish]." Nordic Journal of Nursing Research & Clinical Studies / VÃ¥rd i Norden 25(3): 61-67.

Oldmeadow, L. (1996). "Developing clinical competence: a mastery pathway." Australian Journal of Physiotherapy 42(1): 37.

Page, A. and P. Knight (2007). "Assessing wicked competences." Educational Developments 8(3): 6-8.

Panzarella, K. J. (2003). Assessing clinical competency in the health sciences,. Ph.D. thesis. Buffalo, NY: State University of New York

Panzarella, K. J. and A. T. Manyon (2007). "A Model for Integrated Assessment of Clinical Competence." Journal of Allied Health 36(3): 157.

Panzarella, K. J. and A. T. Manyon (2008). "Using the Integrated Standardized Patient Examination to assess clinical competence in physical therapist students." Journal of Physical Therapy Education 22(3): 24-32.

Pasquina, P. F., S. Kelly, et al. (2003). "Assessing clinical competence in physical medicine & rehabilitation residency programs." American Journal of Physical Medicine & Rehabilitation 82(6): 473-478.

http://www.nmc-uk.org/aDisplayDocument.aspx?documentID=3198




Page 14 of 106

Pellatt, G. C. (2006). "Nursing mentors. The role of mentors in supporting pre-registration nursing students." British Journal of Nursing 15(6): 336-340.

Pender, F. T. and A. E. de Looy (2004). "Monitoring the development of clinical skills during training in a clinical placement." Journal of Human Nutrition & Dietetics 17(1): 25-34.

Pfeil, M. (2003). "Assessing the clinical skills performance of nursing students." Journal of Child Health Care 7(3): 191-206.

Phillips, M. and K. Bharj (1996). "Developing a tool for the assessment of clinical learning." British Journal of Midwifery 4(9): 471-475.

Pirie, E. S. and M. A. Gray (2007). "Exploring the assessors' and nurses' experience of formal assessment of clinical competency in the administration of blood components." Nurse Education in Practice 7(4): 215-227.

Prescott, C. L., C. P. M. v. d. Vleuten, et al. (2008). "Evidence for validity within workplace assessment: the Longitudinal Evaluation of Performance (LEP)." Medical Education 42(5): 488-495.

Price, B. (2007). "Practice-based assessment: strategies for mentors." Nursing Standard 21(36): 49.

Pulito, A. R., M. B. Donnelly, et al. (2007). "Factors in faculty evaluation of medical students performance." Medical Education 41(7): 667-675.

Redfern, S. (2002). "Assessing competence to practise in nursing: a review of the literature." Research Papers in Education 17(1): 51-77.

Roberts, C., D. I. Newble, et al. (2006). "Assuring the quality of high-stakes undergraduate assessments of clinical competence." Medical Teacher 28(6): 535-543.

Robinson, S., K. Boursicot, et al. (2007). "The East Anglian specialist registrar assessment tool." Emergency Medicine Journal 24(3): 180-184.

Robotham, A. (2001). The grading of health visitor fieldwork practice.. Ph.D. thesis Wolverhampton.: University of Wolverhampton

Rolland, S., R. Hobson, et al. (2007). "Clinical competency exercises: some student perceptions." European Journal Of Dental Education: Official Journal Of The Association For Dental Education In Europe 11(3): 184-191.

Rout, A. and P. Roberts (2008). "Peer review in nursing and midwifery: a literature review." Journal of Clinical Nursing 17(4): 427-442.

Rushforth, H. E. (2007). "Objective structured clinical examination (OSCE): review of literature and implications for nursing education." Nurse Education Today 27(5): 481-490.

Sadler, D. R. (2005). "Interpretations of criteria-based assessment and grading in higher education." Assessment and Evaluation in Higher Education 30(3): 175-194.

Salvatori, P. (1996). "Clinical competence: A review of the health care literature with a focus on occupational therapy." Canadian Journal of Occupational Therapy 63(4): 260-271.

Schmahmann, J. D., M. Neal, et al. (2008). "Evaluation of the assessment and grading of medical students on a neurology clerkship." Neurology 70(9): 706-712.

Seldomridge, L. A. and C. M. Walsh (2006). "Evaluating student performance in undergraduate preceptorships." Journal of Nursing Education 45(5): 169-176.

Sharp, S. (2006). "The grading of placement in initial teacher education in Scotland." Scottish Educational Review 38(2): 145-157.

Page 15 of 106

Silber, C. G., T. J. Nasca, et al. (2004). "Do global rating forms enable program directors to assess the ACGME competencies?" Academic Medicine: Journal Of The Association Of American Medical Colleges 79(6): 549-556.

Skingley, A., J. Arnott, et al. (2007). "Supporting practice teachers to identify failing students." British Journal of Community Nursing 12(1): 28-32.

Sluijsmans, D. M. A., G. J. J. M. Straetmans, et al. (2008). "Integrating authentic assessment with competence-based learning in vocational education: the Protocol Portfolio Scoring." Journal of Vocational Education and Training 60(2): 159-172.

Smith, J. (2007). "Assessing and grading students' clinical practice: midwives' lived experience." Evidence Based Midwifery 5(4): 112-118.

Speers, J. (2008). "Service user involvement in the assessment of a practice competency in mental health nursing -- stakeholders' views and recommendations." Nurse Education in Practice 8(2): 112-119.

Srinivasan, M., K. E. Hauer, et al. (2007). "Does feedback matter? Practice-based learning for medical students after a multi-institutional clinical performance examination." Medical Education 41(9): 857-865.

Taleghani, M., E. S. Solomon, et al. (2006). "Grading dental students in a "nongraded" clinical assessment program." Journal Of Dental Education 70(5): 500-510.

Tolhurst, G. and A. Bonner (2000). "Development of clinical assessment criteria for postgraduate nursing students." Collegian 7(2): 20-25.

University Campus, Suffolk. (2008). "Grading of Students Practice Performance”. Ipswich: UCS. Available from: http://www.ucs.ac.uk/study/facultiesAndCentres/healthWellbeingAndScience/HealthP

artners/Grading%20Practice.aspx [accessed 15 July 2009] Vess, P. J. (2007). Competence assessment methods in advanced practice nursing:

a synthesis of the literature. DNP thesis. Columbia, S.C.: University of South Carolina.

Violato, C., J. Lockyer, et al. (2003). "Multisource feedback: a method of assessing surgical practice." BMJ: British Medical Journal 326(7388): 546-548.

Walsh, C. M. and L. A. Seldomridge (2005). "Clinical grades: upward bound." Journal of Nursing Education 44(4): 162-168.

Walsh, C. M., L. A. Seldomridge, et al. (2008). "Developing a practical evaluation tool for preceptor use." Nurse Educator 33(3): 113-117.

Ward, H. and A. Willis (2006). "Assessing advanced clinical practice skills." Primary Health Care 16(3): 22-24.

Wass, V., C. Van Der Vleuten, et al. (2001). "Assessment of clinical competence." Lancet 357(9260): 945-949.

Watson, R., L. Calman, et al. (2002). "Assessing clinical competence in student nurses." Journal of Clinical Nursing 11(4): 554-555.

Watson, R., A. Stimpson, et al. (2002). "Clinical competence assessment in nursing: a systematic review of the literature." Journal of Advanced Nursing 39(5): 421-431.

Weaver, C. S., A. J. Humbert, et al. (2007). "A more explicit grading scale decreases grade inflation in a clinical clerkship." Academic Emergency Medicine 14(3): 283-286.

Webb, C. and P. Shakespeare (2008). "Judgements about mentoring relationships in nurse education." Nurse Education Today 28(5): 563-571.

http://www.ucs.ac.uk/study/facultiesAndCentres/healthWellbeingAndScience/HealthPartners/Grading%20Practice.aspx

http://www.ucs.ac.uk/study/facultiesAndCentres/healthWellbeingAndScience/HealthPartners/Grading%20Practice.aspx

Page 16 of 106

Wilkinson, J. R., J. G. M. Crossley, et al. (2008). "Implementing workplace-based assessment across the medical specialties in the United Kingdom." Medical Education 42(4): 364-373.

Williams, S. R., Alison. (2009). "Evaluation of National Pilot of Longitudinal Evaluation of Performance in Psychology: executive summary." Edinburgh: NHS Education for Scotland. Available from: http://www.nes.scot.nhs.uk/psychology/projects/documents/ExecutiveSummaryFINA

L.pdf [accessed 15 July 2009] Wimmers, P. F., S. L. Kanter, et al. (2008). "Is clinical competence perceived

differently for student daily performance on the wards versus clerkship grading?" Advances in Health Sciences Education 13(5): 693-707.

Yarbrough, S. S. (2008). "Assuring accountability in nursing education: analysis the psychometric rigor of clinical competence examination." Southern Online Journal of Nursing Research 8(2): 2p.

Yorke, M. (2005). "Issues in the assessment of practice-based professional learning: a report prepared for the Practice-based Professional Learning CETL at the Open University." Milton Keynes: Open University, Practice-based Professional Learning Centre. Available from: http://www.open.ac.uk/cetl-

workspace/cetlcontent/documents/464428ed4aa20.pdf [accessed 15 July 2009] Zaksek, T., D. Rebec, et al. (2008). "Assessment of clinical practice of nursing and

midwifery students [Slovene]." Obzornik Zdravstvene Nege 42(1): 35-39.

http://www.nes.scot.nhs.uk/psychology/projects/documents/ExecutiveSummaryFINAL.pdf

http://www.nes.scot.nhs.uk/psychology/projects/documents/ExecutiveSummaryFINAL.pdf

http://www.open.ac.uk/cetl-workspace/cetlcontent/documents/464428ed4aa20.pdf

http://www.open.ac.uk/cetl-workspace/cetlcontent/documents/464428ed4aa20.pdf

Page 17 of 106

Appendix 2b: Reference list from hand search

Alexander, H. A. (1996) "Physiotherapy student clinical education: the influence of subjective judgements on observational assessment." Assessment & Evaluation in Higher Education 21(4): 357-367.

Battistone, M. J. P., B., Milnes, C., Battistone, M.L., Sande, M.A., Hemmer, P.A., Shomaker, T.S. (2001) "Global Descriptive Evaluations Are More Responsive than Global Numeric Ratings in Detecting Students' Progress during the Inpatient Portion of an Internal Medicine Clerkship." Academic Medicine 76(10 Suppl): S105-107.

Benett, Y. (1993) “The validity and reliability of assessment and self-assessments of work-based learning.” Assessment & Evaluation in Higher Education 18(2)83-95

Biggs, J. (1992) "A qualitative approach to grading students." HERDSA News 14(3): 3-6.

Bloodgood, R. A., Short, J.G., Jackson, J.M., Martindale, J.R. (2009) "A Change to Pass/Fail Grading in the First Two Years at One Medical School Results in Improved Psychological Well-being." Academic Medicine 84(5): 655-662.

Borman, W. C. (1975) "Effects of Instructions to Avoid Halo Error on Reliability and Validity of Performance Evaluation Ratings." Journal of Applied Psychology 60(5): 556-560.

Burchell, H., Higgs, T., Murray, S. (1999) "Assessment of Competence in Radiography Education." Assessment & Evaluation in Higher Education 24(3): 315-326.

Carr, S.J. ((2004) “Assessing clinical competency in medical senior house officers: How and why should we do it?” Postgraduate Medical Journal 80(940): 63-66

Colletti, L. M. (2000) "Difficulty with Negative Feedback: Face-to-Face Evaluation of Junior Medical Student Clinical Performance Results in Grade Inflation." Journal of Surgical Research 90(1): 82-87.

Collins, E., Davis, M.W., Godwin, M., Hughes, H., Lloyd, V., Morgan, K., Northway, R., Pick, D., Rowlands, N., Thomas, J. (2004) “Fitness for Practice – All Wales Initiative Assessment Strategy.” Caerphilly, Healthcare Inspectorate Wales.

Dahlgren, L.O., Fejes, A., Abrandt-Dahlgren, M., Trowald, N. (2009) “Grading systems, features of assessment and students approaches to learning.” Teaching in Higher Education 14(2) 185-194

Davies, D.A. (2006) “Accuracy of Physician Self-assessment Compared With Observed Measures of Competence: A Systematic Review.” JAMA 296(9): 1094-1139

Page 18 of 106

Domingues, R. C. L., Amaral, E., Zeferino, A.M.B. (2009) “Global overall rating for assessing clinical competence: what does it really show?” Medical Education 43: 883-886

Downing, S. M., Haladyna, T.M. (2004) "Validity threats: overcoming interference with proposed interpretations of assessment data." Medical Education 38(3): 327-333.

Dudek, N. L., Marks, M. B., Regehr, G. (2005) "Failure to Fail: The Perspectives of Clinical Supervisors." Academic Medicine 80(10 suppl): S84-87.

Duffy, F.D., Holmboe, E.S. (2006) “Self-assessment in lifelong learning and improving performance in practice: Physician Know Thyself” Journal of Amercian Medical Association 296(9): 1139-1145

Fraser, D.M. (2000a) “Action research to improve the pre-registration midwifery curriculum Part. 2: Case study evaluation in seven sites in England”. Midwifery 16 (4): 277-286

Fraser, D.M. (2000b) “Action research to improve the pre-registration midwifery curriculum Part 3: can fitness for practice be guaranteed? The challenges of designing and implementing an effective assessment in practice scheme.” Midwifery 16: 287-294

Gonnella, J.S., Erdmann, J.B., Hojat, M. (2004) “An empirical study of the predictive validity of number grades in medical school using 3 decades of longitudinal data: implications for a grading system”. Medical Education 38: 425-434

Hager, P., Butler, P. (1996) “Two models of educational assessment.” Assessment & Evaluation in Higher Education 21(4): 367-379

Hardcastle, M. (1999) “Assessment of mental health nursing competence using level III academic marking criteria: the Eastbourne assessment of practice scale.” Nurse Education Today 19: 89-92

Karayurt, O., Mert, H., Beser, A. (2009) “A study on development of a scale to assess nursing students‟ performance in clinical settings.” Journal of Clinical Nursing 18(8): 1123-1130

Kings College London (2009) “Nursing competence: what are we assessing and how should it be measured?” Policy plus evidence, issues and opinions in healthcare 18: 1-2 http://www.kcl.ac.uk/content/1/c6/05/68/69/PolicyIssue18.pdf

Kruger, J., Dunning, D. (1999) “Unskilled and Unaware of it: How Difficulties in Recognising One‟s Own Incompetence Lead to Inflated Self-Assessments.” Journal of Personality and Social Psychology 77(6): 1121-1134

Kulick, G., Wright, R. (2008) “The Impact of Grading on the Curve: A Simulation Analysis.” International Journal for the Scholarship of Teaching and Learning 2(2): 1-17

http://www.kcl.ac.uk/content/1/c6/05/68/69/PolicyIssue18.pdf

Page 19 of 106

Lee, L. (2006) “An analysis of assessment” Journal of Community Nursing 20(7): 4-8

McManus, I. C., Thompson, M., Mollon, J. (2006) "Assessment of examiner leniency and stringency ('hawk-dove effect') in the MRCP (UK) clinical examination (PACES) using multi-facet Rasch modelling." BMC Medical Education 6(42): 1-22.

Montgomery, K. (2000) “Classroom Rubrics: Systematizing What Teachers Do Naturally”. The Clearing House 73(6): 324-328

Moskal, B. M., Leydens, J.A. (2000) "Scoring Rubric Development: Validity and Relaibility." Practical Assessment, Research & Evaluation 7(10): 1-11.

Nagle, B. (1998) “A Proposal for Dealing with Grade Inflation: The Relative Performance Index.” Journal of Education for Business 74(1): 40-43

Parker, P. (2009) “What should we assess in practice?” Journal of Nursing Management 17: 559-569

Pasquina, P.F., Kelly, S., Hawkins, R.E. (2003) “Assessing clinical competence in physical medicine and rehabilitation programs.” American Journal of Physical Medical Rehabilitation 82: 473-478

Ravelli, C., Wolfson, P. (1999) "What is the 'Ideal' Grading System for the Junior Surgery Clerkship?" American Journal of Surgery 177: 140-144.

Robb, Y., Fleming, V., Dietert, C. (2002) “Measurement of clinical performance of nurses: a literature review.” Nurse Education Today 22: 293-300

Rohe, D. E., Barrier, P.A., Clark, M.M., Cook, D.A., Vickers, K.S., Decker, P.A. (2006) "The Benefits of Pass-Fail Grading on Stress, Mood, and Group Cohesion in Medical Students." Mayo Clinic Proceedings 81(11): 1443-1448

Schwartz, R.W., Donnelly, M.B., Sloan, D.A., Johnson, S.B., Strodel, W.E. (1995) “The Relationship Between Faculty Ward Evaluations, OSCE, and ABSITE as Measures of Surgical Intern Performance.” The American Journal of Surgery 169: 414-417

Somers-Smith, M.J., Race, A.J. (1997) “Assessment of clinical skills in midwifery: some ethical and practical problems.” Nurse Education Today 17: 449-453

Sonner, B.S. (2000) “A is for „Adjunct‟: Examining Grade Inflation in Higher Education.” Journal of Education for Business September/October: 5-8

Smith, M.A., Burton, W.B., Mackay, M. (2009) “Development, impact and measurement of enhanced physical diagnosis skills.” Advances in Health Sciences Education: Theory and Practice 14(4) 547-556

Speer, A.J., Solomon, D.J., Fincher, R.M.E. (2000) “Grade Inflation in Internal Medicine Clerkships: Results of a National Survey.” Teaching and Learning in Medicine 12(3) 112-116

Page 20 of 106

Taleghani, M., Solomon, E.S., Wathen, W.F. (2006) “Grading dental students in a „nongraded‟ clinical assessment program.” Journal of Dental Education 70(5): 500-510

Tolhurst, G., Bonner, A. (2000) “Development of clinical assessment criteria for postgraduate nursing students.” Collegian 7(2): 20-25

Truemper, C. M. (2004) "Using Scoring Rubrics to Facilitate Assessment and Evaluation of Graduate-Level Nursing Students." Journal of Nursing Education 43(12): 562-564Van Leeuwen et al. (2009)

Weaver, C. S., Humbert, A. J., Besinger, B.R., Graber, J.A., Brizendine, E.J. (2007). "A more explicit grading scale decreases grade inflation in a clinical clerkship." Academic Emergency Medicine 14(3): 283-286.

Williams, M., Bateman, A. (2003) Graded assessment in vocational education and training. Leabrook, Australia, Australian National Training Authority: 1-66.

Williams, S. Rowlands, A. (2009) “Evaluation of National pilot of Longitudinal Evaluation of Performance in Psychology”. Edinburgh, NHS Education for Scotland

Willingham, W., Pollack, J.M., Lewis, C. (2002) “Grades and Test Scores:

Accounting for Observed Differences.” Journal of Educational

Measurement 39(1) 1-37

Yudkowsky, R., Otaki, J., Lowenstein, T., Riddle, J., Nishigori, H., Bordage, G. (2009)

“A hypothesis-driven physical examination learning and assessment

procedure for medical students: initial validity evidence.” Medical Education

43(8) 729-40

Page 21 of 106

Appendix 3: Overview of excluded studies

Author Aim Type of Study Sample size Data collection methods

Data analysis

Main findings

Rationale for exclusion

Limitations Notes

1. Allen (2008)

To discuss professional nurse competency.

Delphi like study.

25 „expert‟ nurse leaders.

2-day Competency Conference using recorded large and small group discussions.

Qualitative analysis.

Three themes identified: components of competency evaluation; barriers and challenges to competency evaluation and recommendations for competency evaluation.

Focus more on the on-going professional competency of qualified nurses.

2. Bartlett et al. (2000)

Compare competencies of Project 2000 diplomates and BA (Hons) Adult Nursing graduates from 2 UK nursing Programmes.

Longitudinal descriptive study (1 year). University setting.

Graduates n=52; Diplomates n=28 Details of those not participating Grads n=29 and Diplomates n=13 are not given.

Self-completion questionnaires data collected on graduation; at 6/12 and 12/12 post graduation.

Multivariate comparisons

Focus of study is on comparing competence over time between Grads and Diplomates post-registration.

Does not address grading of practice.

Page 22 of 106

Author Aim Type of Study

Sample size Data collection methods

Data analysis

Main findings Rationale for exclusion

Limitations Notes

3. Ben-David et al. (2004)

Gather information on current problems as well as identify feasible innovative approaches to PRHO‟s appraisal / assessment information from multiple perspectives

Qualitative. Grounded theory.

Random but purposive sample PRHOs, supervisors, nurses, SHOs.

Focus groups 6 groups with PRHOs 4 groups with educational supervisors 2 groups with specialist registrars 2 groups with nurses 2 groups with SHOs On average each focus group contained 8 members.

Constant comparative method.

1. General themes concerning education and appraisal/assessment of PRHOs – need for standardised tools to record soft and hard behaviours at a very early stage of training to detect deficiencies and allow tailored intervention; 2. Dimensions of competencies and their associated critical behaviours incorporated into a 360 degree diagnostic appraisal tool; 3. Dimensions & associated progressive behaviours (benchmarks) that serve as assessment criteria for educational sessions in a form of portfolio; Aspects of feasibility, reliability and validity of methods employed and cost must be studies further prior to any national implementation.

Does not address aims of the literature review on grading of practice.

PRHOs = Pre-registration House Officers

Page 23 of 106



Data analysis


Limitations Notes

4. Bogo et al. (2006)

Elicit the complex constructs underlying experienced field instructors‟ perspectives about competence.

Exploratory qualitative study.

Purposive sample of 18 experienced field instructors / students focus of article.

Qualitative interviews.

Nvivo – thematic analysis.

Constellation of personal qualities possessed by students perceived as affecting their approaches to learning, their interactions with others, their relationship with field instructors and their ability to develop relationships with clients. Field instructors‟ overall opinions of a student tended to override their opinions regarding student‟s specific skills. Strong tendency for field instructors to place a very high value on students‟ ability to conceptualise their practice.

Descriptive account of what influences assessment however does not focus on grading of practice.

Social work educators / students focus of article

Page 24 of 106



Data analysis Main findings Rationale for exclusion

Limitations Notes

5. Boley & Whitney (2003)

Discussion paper focusing solely on grade litigation appeals in America.

Opinion. NA NA NA Overview of law suits (USA) and recommendations to prevent law suits.

Does not address grading of practice.

6. Brown (2001)

Explore the nature and frequency of items reported in the formal written feedback from mentors at the end of a clinical practice module.

Descriptive triangulated study design.

238 end of placement reports from 3 groups of mental health students: From 83 1

st

year; 75 2nd

year and 100 3

rd year

reports a random sample of 50 from each year were selected.

End of placement records – documentary analysis.

Qualitative and quantitative methods used to analyse data. Content analysis and descriptive statistics.

Personal characteristics of students exert a great influence on judgements about clinical performance (76% of mentors, n=115). Search for the ideal clinical assessment tool has evaded educationalists.

Focuses more on competence versus behavioural learning outcomes.

7. Buckingham (2000)

Literature review debating what and how to measure competencies

NA NA NA NA Various tools described.

Not focussed on grading tools.

Focus on Children‟s nursing.

8. Cassidy (2009)

Review to identify different interpretations of competence and competence assessment.

NA NA NA NA Need for research that examines the veracity of student assessment in a contemporary nursing context. Focus on holistic assessment approach.


Page 25 of 106


Data collection methods

Data analysis


Limitations Notes

9. Cowan et al. (2005)

A review of documentation and literature on nursing competence.

Literature review.

NA Documentary review. Literature: Nursing competence self-assessment questionnaire tool (x2).

NA Self-assessment of clinical competence should form an important element in the overall assessment strategy.


10. Cowan et al. (2008)

Development and psychometric testing of a nurse competence self-assessment tool.

Quantitative Convenience sample 588 post-registration nurses across 5 countries.

EHTAN EU nurse competence self-assessment tool.

Inferential statistics

Psychometric testing as expressed by Cronbach‟s alpha values and by principal component factor analysis suggests that the questionnaire has an acceptable degree of reliability, construct validity and further supports the content validity.


11. Cox

(2000) Opinion article advocating the use of cases as a form of assessment of competence on student doctors.

NA NA NA NA Each intern develops a progressive recorded profile of core cases in which performance has been assessed as competent. If the intern finishes that attachment without a positive assessment on one or more core cases, the responsibility rests with him or her to seek such cases and to improve performance until assessed as competent and signed up.

Not focussed on grading tools

Page 26 of 106


Data analysis


Limitations Notes

12. Crookes (2007)

Very brief description of project to develop an undergraduate nursing competencies assessment tool, for use across Australian Universities.

NA NA NA NA No details as project on-going.

Emailed Prof. Crookes to see if there are any interim findings.

Very early stages – nothing to report

13. Cross et al. (2001)

Investigation of the level of agreement across and within five clinical specialities in physiotherapy on the relative importance of 89 activities associated with clinical competence.

Delphi study – used 3 rounds – 3

rd round

focus of this article.

60 Clinical educators from 2 different UG physio programmes involved in this 3

rd round (but

weren‟t involved in previous 2 rounds).

Random listing of 89 observable behaviours to be rated according to their perceived importance in effective practice.

SPSS inferential statistics.

Conclusion that these factors provided a basis for discussion about clinicians‟ and academics‟ contributions to assessment and a starting point for the development of a clinical assessment instrument that could optimise the validity and reliability of clinical assessment decisions. Model developed – accessible domains and lines of communication n assessment of professional competence in physiotherapy undergraduates.

Focus on the rating of aspects of competency that should be measured. Does not focus on grading of competence/clinical practice.

Focus on physio students

Page 27 of 106



Data analysis


Limitations Notes

14. Darra (2006) Development of a framework in Wales (UK) which is embedded in the perspective of the practising midwife, but which addresses the practice-theory gap by guiding learners and their mentors to what needs to be learnt and to what level.

Descriptive NA NA NA Midwifery skills framework useful day to day tool to help bridge the practice-theory gap.


Focus on midwifery framework tool and role of mentors.

15. Darra (2003) Development of a framework in Wales (UK) of 'composite skills' to teach and assess midwifery skills.

Descriptive NA NA NA Developed framework and give example.

Grading is achieved or not achieved - there is no evidence of grading scale in use.

Page 28 of 106



Data analysis


Limitations Notes

16. Donabedian (2000)

Article that takes a brief look at some methods for assessing physician competence: assessment of performance in test situations and assessment of performance in actual practice.

Opinion NA NA NA Physicians have a rubric of activities and there is no one best method to assess competence.


Focus on physician competence.

17. Duffy & Hardicre (2007a)

Two-part series of articles examining the issue of nursing students who fail in clinical practice. Part 1 explores reasons for failure, assessment and the emotional challenges mentors face when supporting failing students.

Descriptive NA NA NA Describes good practice for mentors when dealing with an underperforming student (nursing).

Focus on failure to fail - not on grading.

Page 29 of 106

Author

Aim Type of Study


Data analysis


Limitation Notes

18. Duffy & Hardicre (2007b)

Two-part series of articles examining the issue of nursing students who fail in clinical practice. Part 2 discusses the management of failing students.

Descriptive NA NA NA Describes good practice for mentors when dealing with an underperforming student (nursing)

Focus on failure to fail - not on grading

19. Driessen et al (2006)

Which criteria affect raters‟ judgments of students‟ reflective skills? Which criteria carry the most weight?

Descriptive Stratified sample of 40 Year 1 medical students‟ portfolios

Portfolio Quality Analysis Rating Inventory


Portfolio ratings associated with quality of reflection and not with aspects of presentation or writing style.


Focus on medical students and portfolios.

20. Everett (2007)

Very short paper to describe a series of workshops between specialist tutors and practice teacher to develop grading of assessment of practice by portfolio

Descriptive NA NA NA Following workshop preparation for practice teachers, written feedback on portfolio work was better and assessment more robust


Page 30 of 106



Data analysis


Limitation Notes

21. Finnerty et al (2006)

Investigation of the preparation and support for the mentor role, student experience and assessment processes.

Multi-method case study approach across 5 sites in England.

No actual numbers given.

Each case study site – individual interviews with Lead Midwives, Midwifery Lecturers and Link Tutors and focus group interviews with students and qualified midwives. Diaries of student & mentor experiences; observation and exit interviews.

Findings demonstrate the need for increased funding to enhance the clinical curriculum. This includes formal protection of time for mentors to provide quality learning experiences. Investment in the mentor/student dyad is essential as successful mentoring can literally be a „gift‟ to student midwives.


Focus on midwifery students & mentors.

22. Flanagan et al (2000)

Descriptive account of use of work-based learning within learning and assessment of practice

Descriptive NA NA NA Work-based learning literature has the potential to bridge the gap between theory and practice.


Mostly focused on process and outcomes of WBL.

23. Fletcher (2008)

Discussion advocating the use of continuous assessment rather than a snap-shot long case examination

Descriptive NA NA NA Literature favours a move away from a non-standardised final clinical competence assessment to a continuous clinical competence assessment.


Focus on osteopathy training.

Page 31 of 106



Data analysis


Limitation Notes

24. Fordham (2005)

Article examines the development of competency based approach to assessment in practice within pre-registration nursing.

Descriptive NA NA NA Competency-based approach to assessment can help to maintain professional standards. Practice assessment alone may not be accurate assessment of the student's understanding and skills - other ways, such as portfolios should be used.


25. Gardner et al. (2006)

Research nurse practitioner practice to inform development of generic standards.

Multi-methods approach – 2 settings – Australia and New Zealand

Purposive sample of nurse practitioners (no numbers given)

Interviews Documentary analysis

Statistics and content analysis probably – not specified.

Generic standards for nurse practitioners will support a standardised approach across 2 countries.

Nothing specific to grading practice.

Focus on qualified nurse practitioners and the development of agreed set of competencies.

26. Girot (2000) Revisits the debate on the meaning of competence in relation to fitness for practice.

NA NA NA NA Partnership approach between student, practitioner/assessor and academic could usefully share responsibilities and ultimately empower the individual for their life-long learning.


Page 32 of 106

Author Aim Type of Study Sample size


Data analysis


Limitation Notes

27. Glass & Ward (2008)

Project aimed to focus on students‟ reflection and reflexivity in practice.

Descriptive 34 mental health nursing students.

Self-assessed competencies.

Descriptive statistics

Self-assessment allows student voices to be heard especially in practice context.


28. Gopee (2000)

Explores the concept of self-assessment in the context of life-long learning for nurses.

Descriptive NA NA NA Nurses should be taught to self-assess themselves given its importance to life-long learning and professional development


29. Gopee (2008)

Discusses the current difficulties with assessment, including barriers to ethical assessment and the issue of failing to fail students in the absence of evidence of competence in specific clinical skills.

Descriptive NA NA NA Suggests that education for ethical practice should be more emphatically incorporated in the educational programme for mentors and similar roles.


30. Gordon & Mitchell (2004)

Article seeks to set spiritual care in a practice context by offering a model for spiritual assessment and care based on the individual competence of all healthcare professionals to deliver spiritual and religious care.

Descriptive with pilot study results.

NA Seminar groups with reflective practice sessions.

Not defined

Conclude that competencies area viable and crucial first step in „earthing‟ spiritual care in practice.


Page 33 of 106



Data analysis


Limitation Notes

31. Gupta (2009)

Focuses on root cause analysis related to medical errors.

Quantitative. Not specified. Rating scoring sheet.

Inferential statistics.

RCA may be a tool for assessment of medical competence but requires further testing on larger sample sizes.


32. Hale et al. (2006)

Map and analyse the educational preparation for cancer and palliative care nursing for children and adolescents.

Qualitative. 26 nurses (students & assessors.

Interviews – face to face & telephone.

Thematic analysis NVivo.

Variety in what was assessed, in particular the lack of assessment on clinical skills, the problem of student selected content of assessment, the different starting points of students and the lack of reliability and validity in the assessment process. Key features of a good assessment process identified.


33. Hanley & Higgins (2005a)

Explores the literature on assessment of clinical practice with specific emphasis on the assessment of competence within the Intensive Care environment.

Literature Review

NA NA NA Lack of clarity around the term competence has led to a number of scales being developed.


Page 34 of 106



Data analysis


Limitation Notes

34. Hewitt-Taylor (2002)

Capture post-registration students views on the assessment processes used and the extent to which they perceive these measure their skills and knowledge.

Qualitative article focuses on case study from a larger study.

4 students. 2 specialist teachers. 15 teachers – all associated with the ENB 415 course.

Observation of selected lessons and course processes. Semi structured interviews. Student diaries. Personal field notes.

Constant comparative analysis.

How student learning and practical skill acquisition can be measured in a way that will enhance rather than detract from learning is a challenge for nurse educators. This merits ongoing investigation, to enable learning and to contribute to a pathway of lifelong learning.


35. Holmes (1997)

Discussion paper (mainly opinion) on the use of journals in clinical practice

NA NA NA NA NA Not focussed on grading tools.

36. Hyatt et al. (2008)

Presents findings from an audit to elicit if the training and support given to mentors were effective and how this strengthened the credibility of the assessment process of students in practice.

Audit Audit of all mentors (n=45) working in medical placements.

Questionnaire. Descriptive statistics.

Audit reinforced the need to increase PEFs providing advice and support to mentors in the assessment process.


Visual aids and „how to‟ guides about the assessment criteria and accurate documentation regarding achievement of learning outcomes were developed.

Page 35 of 106



Data analysis


Limitation Notes

37. Iramaneerat & Yudkowsky (2007)

Demonstrate how to use an MFRM analysis to detect the 4 common types of rater errors in an OSCE.

Quantitative 183 students – each had 6 encounters. 17 standardised patients (actors).

Rating scores completed by SPs.


Literature on common rating errors. Confirmed that rating the performance of medical students in a clinical skills examination is a complicated cognitive process that is prone to various rater errors similar to those found in other performance assessments.

Contains rating – undertaken by SPs.

Focus on medical students and OSCEs.

38. Jinks (2002) To determine how educational assessors rate their current assessment proficiency, future education and training needs.

Survey 343 educational assessors (mentors). Overall response rate 47% (n=156).

46-item questionnaire – main areas addressed: orientation & induction; assessment process, clinical practice, learning process and mentors.

Descriptive statistics.

Whole process of developing, updating and delivering orientation programmes requires review as does the production and use of portfolios in the assessment process.

No mention of grading tools.

Page 36 of 106



Data analysis


Limitations Notes

39. Lauder et al. (2008)

Explore differences between access routes, cohorts and HEIs in levels of self-efficacy, student support and self-reported competence in a nationally rep. sample of student nurses & midwives

Cross-sectional survey.

Stratified random sample of student nurses and midwives (n=777).

Data collected via questionnaire included demographic information, generalised perceived self-efficacy, student support and self-reported competency.

SPSS Inferential statistics.

Relatively positive picture of pre-reg fitness for practice curricula. Support from mentors and HEIs may need to be improved.

nothing about grading / assessment or practice

40. Leach (2007)

NA NA NA NA NA NA Not focussed on grading tools.

41. Long & Asbury (2000)

Discussion article re tripartite approach to assessing practice and the use of multiple sources of evidence.

Opinion NA NA NA NA Use of more than one source of evidence to assess students.


Page 37 of 106



Data analysis


Limitations Notes

42. Luhanga et al. (2008)

To examine the process of precepting an unsafe student.

Grounded Theory.

22 nurse preceptors involved in a final-year clinical practicum.

Mainly 1:1 interviews with some documentary sampling.

Constant Comparative Analysis.

Themes: Reasons for presenting as an unsafe student; Reasons for failure to fail borderline or unsafe students Role of the preceptor as a gatekeeper to the profession.

Focuses on failure to fail and not grading.

Majors on failing to fail – some interesting points which could perhaps be tied into recommendations, training course etc.

43. Lyons (2002) Review methods to assess continuing competence of GPs

Opinion NA NA NA Need to develop appraisal system which covers all areas of practice.


44. Mallik & McGowan (2007)

To complete a scoping exercise on the nature of practice education in 5 selected healthcare professions: Dietetics; Nursing; OT; Physio and radiography.

3 year Mixed Method study

Questionnaires via HoS to most relevant person to complete in HEI. 193 questionnaires distributed. Overall response rate: 40.9% (n=79) Dietetics – 46.2% (n=6); Nursing 23.8% (n=19); OT 67.7% (n=21); Physio 56.8% (n=21); Radiography 46.2% (n=12).

Survey. Focus Groups. Secondary sources. Case studies.

Descriptive stats and thematic analysis

Lack of knowledge of the relative impact of the differing mentor preparation programmes highlighted by problems mentors had in dealing with difficult students and in understanding the assessment process and documentation.


Focuses on Dietetics; Nursing; OT; Physio and Radiography. Details of sample etc. from summary document on website.

Page 38 of 106



Data analysis


Limitations Notes

45. McAdams & Montgomery (2003)

Focuses on a prioritisation matrix technique to reduce burden of assessing competence.

Opinion. NA NA NA NA Not focussed on grading tools.

46. McCready (2007)

Explores the literature on the portfolio as a tool for the assessment of competence in nurse education

Literature review

NA NA NA Highlights importance of clear guidelines for portfolio construction & assessment. Importance of tri-partite support during portfolio development and guidelines for qualitative assessment.


McKinley et al. (2005)

Describe and evaluate a work-centred approach for determining the passing score for a performance based assessment of clinical skills.

Quantitative 2 panels (n=17) Each group reviewed 12 cases.

Standardised Patient Examination.

Descriptive & inferential statistics.

Expert judges can make reliable judgements about standard setting for performance -based examinations.


Page 39 of 106



Data analysis


Limitation Notes

47. McKinley et al. (2008)

Systematic Review - Develop generic criteria for the global assessment of clinical procedural competence and to quantify the extent to which existing checklists allow for holistic assessment of procedural competencies.

Systematic Review.

67 papers retrieved from 2994 papers identified.

NA Descriptive Many checklists reviewed assessed a limited range of competencies. Performance in areas such as team-work and compassion are not explicitly measured/ rated.


48 McLean et al. (2005)

Evaluation of the Scholes & Endacott‟s (2002) competencies

Quantitative 7 students (100%response rate) 26 mentors Response rate 42% (n=11).

Questionnaires – 1 for students and 1 for mentors.


Suggestion that students and / or mentors were cherry-picking the competencies to be achieved whilst ignoring other areas of practice. Exemplars used to indicate level of performance required or knowledge that may be typically associated with a particular competence – increases reliability.


Focus on Critical Care Nursing.

Page 40 of 106


Data analysis


Limitation Notes

48. Munro et al (2006)

Overview of the content of the Clinical Skills Assessment (CSA) for Membership to the Royal College of General Practitioners.

Descriptive. NA NA NA Tests doctors' abilities to apply learned understanding of disease processes and person-centred care.


49. Neary (2000b)

Highlight concerns that students and practitioners might have about the process of assessment of their nursing practice and to suggest an alternative model.

Quantitative Survey: 300 student nurses (CFP). 155 nurse practitioners. Interviews: 70 students 80 assessors.

Questionnaire. Interviews (type not noted).

Looks like descriptive stats and content analysis.

Suggests a new model of assessment called responsive assessment. Responsive assessment changes the assessor role by requiring 2 actions: description and judgement. Responsive assessment records performance within as situational context.


50. Neary (2001a)

Exploration of clinical competence of pre-registration nursing students and the effectiveness of mentorship in 3 colleges of nurse education in the UK

Qualitative 155 nurses practitioners and 300 nursing students

Not defined Not defined

Confusion over the role of mentor, assessor and supervisor


Page 41 of 106



Data analysis


Limitation Notes

51. Neary (2001b)

Argues for a new approach to continuous assessment of pre-registration nursing students in clinical practice in UK.

Mixed methods.

155 nurses practitioners and 300 nursing students. 70 students and 80 assessors.

Questionnaire. Interview.

Not defined.

Suggests that assessment should take account of the 'merit' and 'worth' which combines description and judgement.


52. Nielsen et al. (2007)

Produced guide for reflection using Tanner‟s (2006) Clinical Judgement Model

Opinion NA NA NA NA Not focussed on grading tools.

53. O‟Connor et al. (2001)

Compare the expectations of senior nurses regarding the level of competence of newly qualified nurses with that of the actual level of competency as assessed by the preceptors after 8 weeks in post.

Prospective comparative design.

137 senior nurses (F & G Grades). 37 newly qualified nurses paired with their preceptor (n=37).


Senior nurses have clear subjective expectations of the competence level of newly qualified nurses. These expectations are consistency lower than the actual level of competency of newly qualified nurses.


Page 42 of 106



Data analysis


Limitation Notes

54. Pasquina et al. (2003)

Survey to gather data regarding the variety of assessment tools currently being used in physical medicine and rehabilitation residency programmes in USA and to solicit programme directors‟ impressions regarding the relative value of these measurement tools.

Survey. 81 physical medicine and rehabilitation programme directors. Response rate 75%.

Questionnaire. Descriptive Physical medicine and rehabilitation programme directors apply a variety of assessment tools in evaluating resident clinical competence. High value placed on direct observation of residents by Faculty. Programme directors not comfortable about making decisions re clinical competence alone but rather rely greatly on direct observation of residents by Faculty.


55. Pellatt (2006)

Discussion of the development band current practice of the mentor role in the support and assessment of students undertaking pre-registration programmes.

Descriptive. NA NA NA Not focussed on grading tools.

56. Pfeil (2003) Development of new skills assessment criteria.

Mixed methods.

145 1st year

students. 16 lecturers. 40 practitioners.

Questionnaire. Focus groups. Questionnaire. Individual interviews.

Criteria designed and piloted with substantial practitioner input - highly usable owing to their clarity and comprehensiveness.


Focus on children‟s nurses.

Page 43 of 106



Data analysis


Limitation Notes

57. Phillips & Bharj (1996)

Discussion tracing the development and implementation of a suitable tool to assess clinical competence in a shortened midwifery programme.


58. Price (2007) Preliminary investigation of what practice-based assessment involves in consultation with university and clinically based colleagues and to examine how assessment might be arranged or enhanced.

Literature / opinion based -

NA NA NA Case made for assessing students‟ performance during episodes of care and abandoning the more protected, less objective and difficult to arrange continuous assessment.


59. Roberts et al. (2006)

Describes the development of the integrated final-year assessment of clinical competence at University of Sheffield.

Descriptive. NA NA NA Development of tools requires resource investment to create a tool that truly reflects the student‟s competence and ability.

Nothing about grading

Page 44 of 106



Data analysis


Limitation Notes

60. Rolland et al. (2007)

Investigation into students‟ perceptions of two different competency assessment processes (formative and summative).

Survey. 56 final year students (response rate 89%).


We must encourage students to undertake their competency assessments once they have accrued sufficient clinical experience and reassure them that timing has little effect on the grade achieved.

Not specifically looking at the grading system.

Focus on dental students.

61. Rout & Roberts (2007)

Exploration of peer review of clinical assessment of pre-registration nurses.

Systematic literature review

NA NA NA Whilst peer-review is commonplace in nursing and midwifery practice, there is a lack of robust literature about its use.

Does not discuss grading

Focused on peer- review only.

62. Rushforth (2007)

Narrative review of some of the key issues affecting the use of OSCE within the assessment of nursing students.

Opinion NA NA NA Caution must be applied in relying on OSCE as a sole means of practitioner assessment and used carefully it can make a helpful and meaningful contribution to health professional education.

Grading but focussed within OSCE rather than clinical practice.

Makes reference to marking tools.

63. Skingley et al. (2007)

Article focusing on failure to fail. Main focus is on Community Specialist Practice students .

Opinion NA NA NA Compared reasons for failure from Duffy‟s pre-registration work to that of Community Practice Teachers. Many similarities. Also looks at challenges and potential solutions. Role of facilitator is pivotal so HEIs need to ensure that practice teachers are well supported and are able to identify student weakness in a timely manner.


Main focus is on Community Specialist Practice students Makes reference to social work too.

Page 45 of 106



Data analysis


Limitation Notes

64. Speers (2008)

Investigate the views of stakeholders (service users, lecturers; mentors; ex-students and student nurses) about the potential involvement of service users in the assessment of student mental health nurses‟ competence in forming therapeutic relationships.

Qualitative. 5 service users; 7 students; 4 ex-students; 6 mentors; 2 lecturers.

Focus groups Individual interviews.

Thematic analysis

Service users interviewed had a largely positive attitude towards this potential development. Advises that service user can give formative feedback in a variety of forms, but pass/fail decision should lie with the mentor.


65. Srinivasan et al. (2007)

Examine the impact of receiving feedback via performance benchmarks on medical students‟ self-assessment after a clinical performance exam (CPX).

Quantitative 280 students

Standardised patients and 8 station CPX Phase 1 – students assessed their own performance after CPX. Phase 2 – one month later, students watched their video-taped CPX and reassessed.


Practice based learning exercises that incorporate feedback to medical students hold promise to improve self-assessment skills.


Focus on medical students.

Page 46 of 106



Data analysis


Limitation Notes

66. University Campus Suffolk (2008)

Focuses on Steinaker & Bell experiential taxonomy.


67. Violato et al. (2003)

Development and evaluation of a multi-source feedback system for surgeons designed to assess a broad range of competencies.

Quantitative Random sample of 252 surgeons to test the instrument.

Instrument and survey


Multi-source feedback systems can be used to assess key competencies

No reference to grading.

Focus on surgeons‟ competencies.

68. Ward & Wills (2006)

Discuss how the OSCE was creatively adapted into an OSCA and used to assess the clinical component of a MSc Nurse Practitioner programme.

Not defined Not defined Not defined Not defined

OSCA can be used at Masters level to assess student‟s competence/

focuses on OSCAs and OSCEs and no mention of grading

69. Williams & Rowlands (2009)

Evaluate the acceptability, feasibility, reliability and validity of using the Longitudinal Evaluation of Performance in Psychology (LEPP) tool.

Mixed methods. Quantitative analysis of LEPP forms 2 staged qualitative approach.

No details in executive summary.



Supervisors agreed that LEPP criteria enabled them to assess trainee performance in relation to 4 competency domains. Trainees and supervisors found that the LEPP overall was not an effective tool.

Nothing re grading of practice.

hard copy of full report obtained from NES.

Page 47 of 106



Data analysis


Limitation Notes

70. Wimmers et al. (2008)

Explore which competencies are considered important for daily performance of student on the wards and to what extent clinical teachers consider the same competencies important for clerkship grading

Survey 218 questionnaires returned from 17 different hospital settings and 10 specialities

Questionnaire listing 21 student characteristics with 2 response columns – one for importance related to daily performance on wards and the other related to importance re grading of clerkship.

Descriptive statistics and factor analysis

Components of clinical competence considered important for adequate performance are not necessarily in alignment with what is required grading.

Despite title of article nothing about grading

Page 48 of 106

Hand search articles – exclusion


Sample size


Data analysis


Limitation Notes

1. Bargagliotti et al (1999)

Overview of safe practice in nursing with transition to competency based education.

Opinion. NA NA NA Importance of involving stakeholders in understanding competency based education

No mention of grading in practice.

2. Benett (1993)

Investigate to what extent are work-based learning assessments valid and reliable.

Opinion. NA NA NA Under certain stated conditions, it is possible to determine whether assessments (and self-assessments) of work-based learning are valid, reliable and comparable.


3. Carr (2004) Overview of how and why competency based assessment should take place for senior house officers.

Opinion. NA NA NA There are disadvantage and advantage to a competency based system.


4. Dahlgren et al. (2009)

Identify the relationships between grading systems, assessment characteristics and students‟ learning.

Survey 402 distributed.

Questionnaire with 14 fixed response and 2 open ended questions.

Multi-step grading scales may have detrimental repercussions on the nature of the assessment tasks and thereby indirectly on the students‟ approaches to learning.


No indication of response rate. No mention of pilot.

5. Duffy & Holmboe (2006)

Explores links between self-assessment, life-long learning and performance in practice.

Opinion/ Literature review.

NA NA NA There are limitations to self-assessment, but could help to highlight development areas in relation to practice performance


Page 49 of 106


Sample size


Data analysis


Limitation Notes

6. Fraser (2000a)

Identify factors which facilitates and inhibit effectiveness of midwifery programmes.

Qualitative. 99 students, 59 assessors, 29 teachers, 41 new midwives, 33 mentors, 37 midwives , 32 supervisors of midwives

Interviews. Not defined.

Confident, competent students required, who should be prepared to take on whole role of the midwife.


7. Fraser (2000b)

Design a robust assessment scheme to more effectively identify midwives‟ fitness to practice.

Action research.

Not defined.

Not defined. Not defined.

Mentors need to better understand student competence.


8. Hager & Butler (1996)

Basic assumptions, features and appropriate uses of the scientific measurement model and the judgmental model are compared and contrasted by referring them to a 3-level conceptual model of education, training and assessment for workplace performance.

Descriptive literature review.

NA NA NA Argued that there are 2 distinctive models available for educational assessment and that only the judgemental model is suited to recent and increasingly widespread innovations and initiatives such as PBL, education for capability and portfolio-based performance assessment.


Page 50 of 106


Sample size


Data analysis


Limitation Notes

9. King‟s College London

Discusses challenges, education and management of developing nursing competence.

Opinion. NA NA NA Need behavioural and holistic approaches to assessment of competence, and a multi-method approach to assessment is the gold standard.

Not relevant to grade inflation or grading tools.

10. Kulick & Wright (2008)

Discusses grading on the curve.

Opinion/Literature review

NA NA NA Pros and cons of grading on the curve


11. Kruger & Dunning (1999)

Investigate why people hold overly favourable views of their abilities in many social and intellectual domains.

Quantitative Study 1: 65 psychology undergraduates Study 2: 45 psychology undergraduates Study 3: 84 psychology undergraduates Study 4: 140 psychology undergraduates

Study 1 – questionnaire Study 2 – reasoning test Study 3 – grammar test Study 4 –logical reasoning test

Paired t tests P value

Test scores put participants in the 12

th

percentile but they estimated themselves to be in the 62

nd. Several

analyses linked this miscalibration to deficits in metacognitive skill, or the capacity to distinguish accuracy from error. Paradoxically improving the skills of participants and their metacognitive competence helped them recognise the limitations of their abilities.


12. Lee (2006) How are nurses assessed in clinical practice.

Opinion NA NA NA Assessment methods continue to develop.


13. Lundgren et al. (2008)

Grading on theoretical thesis.

Not relevant to practice.

Page 51 of 106



Data analysis


Limitation Notes

14. Nagle (1998) Proposal to introduce a Relative Performance Index to eliminate grade inflation.

Descriptive Based on example of factious data.

NA Descriptive statistics.

Grade inflation is a sensitive topic in higher education. In some cases it has rendered the traditional measure of student performance almost meaningless. The introduced Relative Performance Index does not eliminate grade inflation.

Focuses on theoretical assessments

Focus on Business School students.

15. Parker (2009)

Exploration of current tool and consider other tools which might be used.

Qualitative evaluation approach

63 practitioners doing a mentorship preparation course (Focus Groups) 6 assessors, 7 students and 10 lecturers (individual interviews) 10 students and 6 assessor (questionnaire)

Documentary analysis Focus groups (n=4) Individual interviews Questionnaire

Inductive analysis

The current tool was not consistently used and some of the criteria being used to assess practice were not in the tool used at the time.


Small sample size.

Useful example of scale descriptors.

16. Pasquina et al. (2003)

Identify range and evaluate methods of assessing competence employed in physical medicine and rehabilitation.

Survey. 81 programme directors (75% response rate).

Questionnaire.


One third felt that they did not adequately assess practice-based learning. One quarter did not use other professions to rate competence.

Not focussed on grading of practice.

Page 52 of 106



Data analysis


Limitation Notes

17. Robb et al. (2002)

Critical review of the research exploring the measurement of nurses‟ clinical performance.

Literature review.

13 articles. Cochrane Research Database (CRD) guidelines used for searching literature.

NA A variety of approaches have been used to identify aspects of the role of the nurse and to assess the performance of nurses both during their education programmes and at the point of qualification.

Focuses mainly on the criteria to be assessed rather than on grading scales.

18. Sonner (2000) Examination of grade inflation in relation to theory.

NA NA NA NA There were differences between markers


19. Smith et al. (2009)

Tested inter-rater reliability in marking student „write-ups‟ of patient encounter.

Quantitative. 52 write-ups. Documentary evidence independently marked and then compared.

Correlation co-efficient.

Better inter-rater reliability than before.

Does not use grading tool.

20. Shakespeare (2009)

Explores issues related to practice.

Opinion NA NA NA General discussion Does not use grading tool

21. Somers-Smith & Race (1997(

Discusses assessment of clinical skills in midwifery.

Opinion NA NA NA Discusses issues such as regular contact with student, reliability and validity of assessment


22. Willingham et al. (2002)

Explains why grades and scores differ in school children.

Opinion Not relevant

Page 53 of 106



Data analysis


Limitation Note

s

23. Yudkowsky et al. (2009)

To obtain initial content validity evidence, performance and reliability estimates, and impact data for a tested physical examination

Quantitative 66 third year medical students undertaking 160 physical examinations

Documentary evidence

Statistical Physical examination was reliable and valid

Does not use grading tool

Page 54 of 106

Appendix 4: Overview of included studies



Data analysis

Main findings

Rationale for inclusion

Limitations Notes

1 Abbey (2008)

To analyse outcomes of two different methods of assessing clinical competence and evaluate their validity and congruence with professional standards.

Quantitative: Retrospective statistical analysis of 2 student cohorts.

1998 – 2000 n=240 2004-2006 n=228

Retrospective student grades from Final Clinical Competence Assessment (FCCA) [snap-shot] and Clinical Tutor Reports (CTR) [continuous].

Non-parametric statistical tests with probability set at p<0.01.

More research needed. Description of correlation between 2 ways of measuring assessment – one was graded.

Description of correlation between 2 ways of measuring assessment – one was graded.

Osteopathy students focus of research.

2 Andre (2000)

Aim to extend the grading of practice debate with particular application to undergraduate nurse education.

Literature review.

N/A N/A N/A Graded, competency based assessment is generally achievable, though the educational advantages and quality of instruments requires further research. No clear argument that precludes the use of graded assessment in the evaluation of clinical practice.

Literature on grading of practice.

Search terms and databases not described – therefore dubious systematic nature of review.

Page 55 of 106


Data analysis

Main findings Rationale for inclusion

Limitations Notes

3 Baulcomb & Watson (2003)

To evaluate several aspects of a measurement instrument the APG which included internal consistence and inter-rater reliability .

Quantitative: Opportunistic study – very preliminary piece of work.

No number supplied. Convenience sample of all practitioners undertaking a programme of study within one university/BSc community nursing (post-reg).

Assessment of practice grid - APG measures 13 items such as motivation, initiative, independence and self directed.

SPSS – interferential statistics t-test and Bland and Altman plot.

Inter-rater reliability judged significant - individuals make the same measurement at 2 different time points. Marks were similar for the student over two trimesters - perhaps suggests reliability or that the instrument is not sensitive enough to detect changes. Authors note that in very high grade there is less reliability to the previous or next grade. Lower and middle grades are more reliable. I think there should be an improvement in grade over the next semester but authors argue that this could be due to 'plateau in performance' as they are already experienced practitioners. Authors suggest further development and testing.

Interesting points for inter-rater reliability.

Very small sample and not majorly tested.

Focus on post-registration BSc Community Nursing programme

Page 56 of 106


Data analysis


Limitations Notes

4 Cacamese et al. (2007)

Determine the existence, extent and possible causes of sub-internship grade inflation.

Survey - 16 questions.

278 sample size, 141 returned survey – 51% response rate. Sent to national (USA) sample of internal medicine clerkship.

Questionnaire – Likert scale and yes/no.

Inferential statistics using t tests, analysis of variance and Krusal-Wallis tests.

Grades described as honours/high pass/ pass/ low pass/fail or A/B/C/D/E or pass/fail and 11 others. Grade inflation defined as a percentage of excellent scores than student performances warrant. 80% receive honours level grade while directors feel that only 1/3rd should achieve this grade. Grade inflation exists. 18% of assessors admitted passing a student that they felt should have failed. Difficulty delivering negative feedback is the top explanation for grade inflation.

Authors conclude that grades are not particularly helpful in discriminating student performance.

Response rate 51%, some questions were unanswered. Retrospective recall required.

Focus on student doctors. Failure to fail.

Page 57 of 106


Data analysis


Limitations Notes

5 Calman et al. 2002

To describe the methods of measuring progress in achieving competence of preregistration Scottish nursing & midwifery students and to describe the philosophy and approaches to competence assessment in HEI & FE Colleges.

Phased using quantitative and qualitative methods - 13 programmes across 7 HEIs.

13 directors of programmes (survey) 12 group interviews with students (6 nursing & 6 midwifery), comprising 72 students (36 of each group).

Postal questionnaire Documentary analysis Interviews with key stakeholders.

Consensus views of students were summarised by the interviewer under the topic headings within the interview guide. No other information provided on analysis of data.

Four key findings - competence assessment methods, preparation of practice assessors; consequences of failure to meet expected level of outcome and students‟ views. At the time of the study 2 institutions awarded merits/distinction for academic excellence only. One institution had used a grading system for clinical practice but this was discontinued "because of a tendency for high grades to be awarded ...there was concern too that the system was open to subjective bias of the assessor." [p. 520]. Also suggests lack of consistency to assessment.

Students' views suggest that there is little confidence in methods of clinical competence assessment. No formal reliability and validity testing.

Sampling could be biased.

Makes reference to the introduction of a national clinical competence tool (p523).

Page 58 of 106



Data analysis


Limitations Notes

6 Chambers (1999)

Address one dental school‟s efforts to address issues of competency-based evaluation.

Quantitative. 77 faculty members.

Quarterly rating system developed and used to replace daily grading. In year 1 703 ratings were completed on 126 students - average 5.3 ratings per student. In year 2 1967 ratings were completed on 142 students - more than 15 ratings per student, on average.


Grade inflation brought under control. Progression halted for incompetent students. More extensive info about remedial needs of students. Excellent face validity and rater consistency in some cases.

Grade inflation seemed better following implementation of this tool.

Tested in one dental school in USA.


7 Chambers (1998)

Literature review on some issues of assessment of practice.

Literature Review

NA NA NA The validity and reliability of a plethora of tools designed to measure it is problematic and the truly objective practice assessment tool has yet to be developed.

Graded measures are described (but not critically analysed).

No search criteria suggested.

Adds in assessment of reflection into debate.

Page 59 of 106


Sample size


Data analysis Main findings Rationale for inclusion

Limitations Notes

8 Clouder & Toms (2005)

Explore the validity of assessment strategies used to assess practice-based learning within an UG physiotherapy programme in the UK.

Qualitative. Purposive sample 18 visiting tutors; 18 students; unclear as to the number of clinicians involved.

Individual semi-structured interviews.

Analysed independently by 2 researchers.

Assessment strategies used possess face, content and criterion-referenced validity. The necessity for assessment to go beyond „showing‟ and „doing‟, to explore the thinking underpinning practice is affirmed.

Used grading from exceptional to unsatisfactory.

One of the assessment strategies used – clinical reasoning viva (CRV) is unique to the programme

Focus on UG Physio students

9 Ciolfi (2009)

Unable to source reference in time

10 Coote et al. (2007)

Development of a Common Assessment Form (CAF) for grade assessing physio students from a number of HEIs on practice Education placements in Republic of Ireland.

Quantitative Piloted CAF (common assessment tool) on a small number of sites then used.

54 practice educators.

Pre-pilot Questionnaire to 54 practice educators Testing validity - 2 experienced practice educators graded students at the end of placement – 71 data sets returned for analysis.

Testing inter-rater reliability practice tutor and a practice educator rated the same student on the CAF at the end of the placement – 43 data sets returned for analysis.

SPSS. Pearson correlation coefficient . C Intraclass Correlation Coefficients. Bland and Altman method.

The validity study suggested good face, content and construct validity and the correlation coefficients and means of differences between score suggest high reliability. The possibility of systematically higher scoring by the practice educators warrants further investigation.

Common Assessment Tool. Discusses validity and reliability of CAF.

No response rates stated.

Identified main features required in an assessment tool.

Page 60 of 106



Data analysis


Limitations

Notes

11 Cowin et al. (2008)

Investigate whether the Australian National Competence Standards for Registered Nurses demonstrates correlations with the Finnish Nurse Competency Scale.

Non-experimental cross-sectional survey design.

Convenience sample of 116 new graduate nurses.

Questionnaire to new graduate nurses in one state in Australia.

SPSS Inferential statistics.

Correlational analysis of self-assessed levels of competence revealed a relationship between the Australian National Competence Standards and the Nurse Competency Scale. Both use a Likert 0-100, low to high, grading scale.

Used grading Likert scale.

Was not trialled in practice - was completed as paper exercise. Only 1 state in Australia.

Makes reference to importance of self-assessment of competency. Focus on newly qualified nurses.

12 Cruess et al. (2006)

Unable to source reference in time.

Page 61 of 106



Data analysis


Limitations Notes

13 Dalton et al. (2009)

The development and embedding of a performance based assessment instrument within Physiotherapy curricula nationally.

Quantitative: Action Research across 9 Universities in Australia.

Pilot – 295 3rd

& 4

th year physio

students tested instrument followed by 8 Focus groups – purposive sample of 79 students. Field tests –Clinical educator training pack developed. Field Test 1 – n=747 (students & clinical educators) 2 Focus groups (n=20). Field Test 2 - 538 students forms were collected by 450 clinical educators.

Use of Assessment of Physiotherapy Practice (APP) tool. Clinical Educator demographic form. Feedback – clinical educator survey. To allow triangulation of data, several methods were employed to gather opinions and feedback from stakeholders: • „think-aloud‟ interviews (n=9); • focus groups; • survey; and • training and information sessions.

SPSS Rasch analysis.

Production of a single instrument with known validity and reliability available to replace the twenty-five distinct assessment forms formerly in use. In addition, the instrument provides unprecedented opportunity for national discussion regarding measurable standards of practice. Resources that provide ongoing support for clinical educators and students using the APP have been developed, but more can still be done. Further research is suggested to advance support material, develop an on-line version of the APP and investigate the long term impact of assessment standardisation. APP was user friendly, comprehensive and time efficient.

Used a 5 point rating scale 0-4 for each competency.

Focus on physio students.

Page 62 of 106



Data analysis


Limitations Notes

14 Davies et al. 2008

Determine whether a generic MST instrument could be used across all specialities or whether speciality-specific modification will be needed.

Quantitative. 92 histopathology trainees Year 1. Assessor response rate 92% no raw figure given.

Modified multi-source feedback (MSF) instrument and OPSE (Objective Structured Practical Examination).


Multi-source feedback instruments used to inform decisions about a doctor‟s progress in training or fitness to practice must be appropriately validated. Continued development, evaluation and vigorous debate are required.

Used 6 point rating scale.

Fairly small sample n= 92. Authors note ethnic group bias and consultant level assessors as potential biases.

Focus on year 1 speciality training in histopathology.

15 Dillard et al. (2009)

Describe faculty level development of the Lasater Clinical Judgement Rubric (LCJR); faculty application of LCJR in evaluating students‟ clinical judgement skills during a simulation scenario.

Quantitative and qualitative components used.

Faculty evaluation of workshop (n=16). Student self-evaluation (n=68). Guided reflections (n=25 students).

Faculty workshop. student self-evaluation. Guided reflections .

Descriptive stats and content analysis.

A standardised method for evaluating student learning in simulation is lacking. More work needs to be done to improve the integration of clinical judgement and the use of a conceptual framework and rubric.

Could this thinking be transferred to assessment of real practice?

Focus on simulation and clinical judgement.

Focus on simulation and clinical judgement.

Page 63 of 106



Data analysis


Limitations Notes

16 Dudek et al. (2008)

Attempt to formalise the dimensions and qualities that distinguish effective versus less useful styles of form completion.

Quantitative: Modified Delphi technique using 3 phases.

Focus Group (n=10). Scale development (n=10). Reliability & validity testing (n=72). Final data set – 990 evaluations from 55 participants.

Focus group. Brainstorming. Use of Completed Clinical Evaluation Report Rating (CCERR).


This study identifies features that constitute a high-quality completed evaluation report and describes an objective measure of supervisors‟ ability to complete a quality evaluation report that may be used for formative and research purposes.

Authors‟ state need for numerical rating scales to include descriptive comments that are specific, detailed and use concrete examples.

Small sample sizes

Focus on medical training programmes

17 ElBadrawy & Korayem (2007)

Description of the development of the 'Flexible Requirement System' for grading of dental students - on a 5-point scale.

Descriptive. One university in Canada.

NA NA Claims to be a student-friendly means of assessing clinical practice.

Used a 5-point rating scale.

One site used, descriptive account only.


Page 64 of 106



Data analysis


Limitations Notes

18 Feeley et al. (2003)

To validate an assessment tool of medical students‟ ability to integrate clinical skills and scientific knowledge within the patient encounter within OSCE.

Quantitative. 140 1st year

medical students with a convenience sample of 48 selected. 48 clinical examinations evaluated by 2 raters to test inter-rater reliability. In total 10 faculty participated.

Integrated standardised patient examination – video-taped.


Significant amount of variance in student scores was due to faculty raters. It is recommended that future studies undertake a training workshop for raters. …

Used 4-point rating scale.

One medical school in NY, USA. Simulated OSCE using standarised patient - not tested in real practice.


19 Fisher & Parolin (2000)

To examine the use of competency based assessment tool applied to the assessment of novice registered nurses‟ clinical practice within their actual working environment via direct examination of inter-rater reliability between 2 observers.

Quantitative: Pilot Correlation descriptive design.

Convenience sample of 30 new graduate nurses in Australia.

Each new graduate nurse was observed by 2 observers for a mean period of 127.5 minutes.

Inferential statistics Paired Sample t test, Cronbach‟s coefficient alpha; Cohen‟s Kappa coefficients; Person‟s correlation coefficients.

Very little rating agreement was found for individual items. There were no significant differences in the total competency scores between the two observers. Competency assessment tool is valid and internally consistent. However, inter-rater reliability for individual competency items was found to be poor.

Uses 5 point rating scale (Bondy).

Small pilot study One institution in Australia

Some interesting comments in literature review.

Page 65 of 106



Data analysis


Limitations Notes

20 Fitzgerald et al. (2007)

Provide evidence for the validity of the Clinical Internship Evaluation Tool (CIET).

Quantitative. 228 students. 26 Clinical Instructors. 7 Faculty members.

Student evaluations. Survey of clinical instructors. Item review by Faculty.


CIET is a useful tool for evaluating clinical performance of student physical therapists.


One school in Pittsburgh, USA.


21 Gill et al. (2006)

Examine the applicability, validity and reliability of the Clinical Performance Assessment Tool (CPAT) for post-graduate critical care nurses.

Quantitative:3 phase descriptive correlational study.

Phase 1 – 6 experienced clinical nurses. Phase 2 – 8 students and 8 clinical facilitators. Phase 3 survey 9 assessors, 13 students.

CPAT. Questionnaire. Interviews – students & their assessor.

Descriptive statistics and thematic analysis using NUDIST.

CPAT facilitated assessors to assist students to develop their clinical performance. However substantial refinement was required to make it useful as a clinical assessment tool. Use of 5 point assessment rating scale - too complex and unrealistic for assessors and students to use. Grade inflation admitted.


Small sample sizes, further evaluation not apparent from literature in subsequent years.

Focus on critical care nursing – PG students.

Page 66 of 106



Data analysis


Limitations Notes

22 Glavan & Maran (2002)

Development of use of scoring systems for assessment of clinical competence.

Descriptive. NA NA NA New tools need to be developed and sufficiently tested to ensure construct validity and inter-rater reliability. Subjects need to have confidence in the tool itself and those carrying out the assessment.

Good points to note when developing a scoring tool for simulator and real practice.

Opinion and descriptive account of literature.

23 Glover (1997)

To evaluate and grade 3

rd

year undergraduate nursing students‟ clinical performance during practicum.

Exploratory Pilot study using both quantitative and qualitative methods.

Total cohort of 160 3

rd year

student nurses. 97 preceptors completed the students‟ assessment and became a convenience sample. 36 out of 160 students completed the self-assessment tool.

Assessment tool- which had both quantitative and qualitative components.

Descriptive statistics. Content analysis.

Preceptors rated student performance at a higher than the expected level. Preceptors rated student performance slightly higher than students‟ self-assessment. Preceptors found it difficult to support their ranking of student performance using ANCI cues. Preceptors grades were ranked higher than theoretical grades.

Grading / rating tool used.

Difficult to tell how many grade points were used in the assessment tool. Domains converted to % but no detail as to how this one done.

Page 67 of 106


Sample size


Data analysis


Limitation Notes

24 Grant et al. (2007)

Overview of 4 different competency-based assessments including Multi-source feedback; Clinical Evaluation Exercise; Case-based Discussion.

Opinion. NA NA NA There is significant concern that these forms of assessment are only appropriate at lower stages of training where the complex clinical performance is less important than the underlying separate competencies. Despite this they are being implemented at all levels.

No data to support – opinion only

Focus on medical students

25 Gray & Bradshaw (2007

Provides an overview of the methods used in 2 English primary care organisations to locally assess GPs who may be underperforming

Descriptive NA Use a rating scale on colleagues of GPs as part of wider data collection methods.

Assessed on all aspects of data collected - no real data presented.

Triangulation of evidence promotes objectivity because assessors make recommendations based upon the weight of evidence rather than subjective impressions.

Triangulation of data felt necessary for reliability and validity purposes.

Small geographical area. No data presented.

Focus on GPs.

26 Hand (2006)

Explore assessment in clinical practice including the measures that can be taken to ensure that it is conducted fairly and the appropriateness of the methods undertaken.

Descriptive. NA NA NA Aspects of fair assessment considered. Observation, questioning, testimonies and reflective discussions included. Triangulation of assessment explained.

Good points for developing scoring tools.

Descriptive opinion only.

Page 68 of 106


Sample size


Data analysis


Limitations Notes

27 Hanley & Higgins (2005b)

Exploring student nurses‟ perceptions of a new clinical competence tool recently introduced into a postgraduate intensive care nursing course using Benner (1984) levels novice, advanced beginner and competent. Novice regarded as fail by the end of the placement.

Qualitative descriptive exploratory design.

11 students - 6 individual interview and 5 in a focussed group discussion.

Interviews with pre-registered intensive care (ITU) nurses - individual and focus group.

Qualitative analysis producing themes.

Language confusing, and lack of focus within ITU, inter- rater reliability problems. Recommended review of language used within the tool and improving inter-rater reliability.

Common problems identified

Small sample size, limited scale used

Focus on post-registration students in ICU

28 Haroon-Iqbal & Junks (2002)

Unable to source reference on time

29 Hemmer et al. (2008)

Describe current evaluation methods, use of the Reporter-Interpreter-Manager / Educator (RIME) framework and grade assignment by internal medicine clerkship directors.

Quantitative – survey.

Clerkship directors (n=84) Response rate 77%.

Survey Inferential statistics.

Internal medicine clerkship directors continue to emphasise descriptive evaluation of trainees – and also shifted to including a wide variety of examination and other methods, including observations with standardised and real patients. RIME gaining widespread acceptance. Triangulation of methods used to rate candidates. More grade inflation in criterion-referenced grading.

Interesting framework for descriptive evaluation combined with direct observation.

Response rate.


Page 69 of 106



Data analysis


Limitation Notes

30 Hill et al. (2006)

To develop and implement a competency-based assessment process for the experiential component of a pharmacy education curriculum.

Quantitative Survey.

Students (n=74: 76% response rate). Faculty (n=90: 30% response rate).

Survey. Inferential statistics

Faculty and student perceptions of the assessment process were generally positive. Moderately successful in reducing grading inflation.

Used 5-point grading tool, providing example.

Response rate.

Focus on pharmacy students.

31 Holaday & Buckley (2008)

Presentation of an innovative tool-kit used by 3 major university (USA) nursing schools to assess, evaluate and measure student performance and growth across clinical settings and at all levels of educational preparation in a nursing programme.

Descriptive NA NA NA Evaluation tool-kit has demonstrated that the measurement of performance of standardised competencies in the clinical setting not only is possible but that the education of the user is critical in ensuring the validity and reliability of the outcome performance data. States that limited exposure to encounters in the clinical setting and the amount of evidence the assessor requires to make inference about the quality of performance makes the performance assessment less reliable if this is not witnessed 6-10 times.

Used 5-point rating scale, providing example.

Descriptive account of development. No evaluative data available/ presented.

Page 70 of 106



Data analysis


Limitation Notes

32 Howley (2004)

Review of selected research literature related to performance assessment and suggests 3 areas for future research and development in the assessment of clinical competence.

Literature review.

NA NA NA Areas for future research and development in the assessment of clinical competence with standardised patients.

Not focussed on grading tools but does focus on reliability of SPs' assessing.

33 Isaacson & Stacy (2009)

Discuss faculty and student concerns with regard to clinical evaluation and to explore the use of rubrics as a tool to help objectify the clinical evaluation process.

Literature review/ opinion

NA NA NA Rubrics provide one possible solutions to the concerns of faculty and students related to effective clinical evaluation.

Rubrics could be worth further exploration.

No research data.

Very useful re grade inflation and details re rubrics.

34 Johnson (2007)

Discusses some of the issues that surround the grading of competence based assessments including grading and motivation; effects of grading on (mis)classification; grading and accountability etc.

Literature review/ opinion

NA NA NA Potential benefits of grading needs to be balance against its potential drawbacks.

Good summary paper of the pros and cons.

No research data.

Page 71 of 106



Data analysis


Limitation Notes

35 Johnson (2008)

Literature review attempting to synthesise the positive and negative effects of grading and discuss the implications.

Literature review.

NA NA NA Questions about the desirability of grading competency-based assessments are related to issues of validity , with the question hinging on the simultaneous existence of two mutually supporting factors: „use value‟ and „validity‟.

Literature review on process of grading, discussing implications.

No details of search strategy or number of articles included / excluded.

36 Knight & Page (2007)

Identify wicked competencies from 6 subject areas: accounting; early years education; nursing; secondary school education; social work; youth work.

Quantitative and Qualitative: Mixed method.

83 informants (working in HEIs) completed survey. Sub-sample of 14 phone interviews.

Survey. Telephone interviews.

Statistical analysis. Thematic analysis aided by Nvivo7 software.

Survey data produced the unexpected finding that respondents saw few problems in assessing „wicked‟ competencies. It was remarked by some informants that attending to these problems would involve giving them priority in the curriculum, investing in staff training, and providing more resources to allow more time to be spent.

Some reliability issues are addressed.

Small sample within nursing.

Focus on wicked competencies – defined as competencies that cannot be neatly pre-specified, take time to develop and resist measurement-based approaches to assessment

Page 72 of 106



Data analysis


Limitation Notes

37 Kogan et al. (2003)

Determine the feasibility, reliability and validity of the mCEX when used to evaluate American medical students‟ clinical skills in a medical core clerkship.

Quantitative. Students (n=165) completed 9 mCEX during internship (89% completion rate).

mCEX instrument.

Mean mCEX scores were correlated with exam scores and course grades.

Data support the feasibility, reproducibility and validity of the mCEX in evaluating medical students‟ clinical skills.

7-point scale used.

Small sample size, single institution in USA.


38 Lanphear (1999)

Discusses the rationale for assessing and grading student performance.

Text / opinion.

NA NA NA Grading is not simple, but if developed well could be essential for maintaining standards and ensuring the public of doctors‟ competence – opinion.

Some proc and cons discussed.

Opinion. Focus on medical students.

39 Lasater (2007)

Pilot test a rubric in simulation lab to describe the development of clinical judgement.

Quantitative: Exploratory mixed method design. Cycle of theory-driven description-observation- revision – review – repeated every 3 weeks until rubric developed enough to pilot test.

Observations (n=53) of 39 3

rd year

student nurses in a simulation lab.

Qualitative observations in simulation lab. Quantitative scoring using the rubric. Focus group of 8 of the observed students.

Descriptive statistics. Themed for qualitative data presentation but analysis method not defined/described.

Rubric offers latitude for a variety of clinical contexts. Recognition that a student‟s individual background can affect the clinic judgement process. Rubric offers language that is understood by both faculty and students and sets standards that students can comprehend and work toward.

Rubrics may be worth further exploration.

Small sample size. One institution used.

Focuses on clinical judgement. Good background info on rubrics.

Page 73 of 106


Sample size


Data analysis


Limitations

Notes

40 Levett-Jones (2008)

Discuss the Structured Observation & Assessment of Practice (SOAP) tool that was developed in response to criticisms that university assessments were not always comprehensive or rigorous enough. Afterwards an hour is spent providing detailed feedback to students to identify areas of strength and weakness and any required remediation.

Opinion: SOAP involves a full day holistic assessment of students‟ clinical knowledge, skills, behaviours, attitudes and values, undertaken in a clinical context. Opinion from developer of the SOAP.

NA Students are observed and assessed by an experienced nurse as they attend to real patients at one of their clinical placement locations. The assessment is conducted via observation, open-ended questions and comparisons of performance against Australian N&M Council Competence Standards for the Registered Nurse.

One student comment and developer's opinion.

SOAP increased students' confidence in their own clinical competence. Feedback and remediation immediately available.

Opinion on grading performance in simulation.

No presentation of evaluative methodology

41 Lewis et al. (2008)

Evaluation of the inter- and intra-rater reliability of an assessment tool used for physiotherapy students during a clinical placement.

Quantitative: Prospective test, retest study design.

5 clinical educators. 1 academic.

Each rater independently marked 22 student written assessment that had been completed by students viewing a videotaped patient physiotherapy assessment. Repeated again 7 weeks later in randomised order.


A high level of intra-rater reliability was found for clinical educators. Inter-rater reliability was poor. Reliability of clinical assessment tools used in other areas of physiotherapy education should be formally measured rather than assumed.

Small sample size. One institution used.

5-point rating scale used.

Focus on physio students. Need for education / training in the use of tool.

Page 74 of 106


Sample size


Data analysis


Limitation Notes

42 Lockyer (2003)

Discussion article about multi-source feedback (MSF). Goal is to look at a person‟s work from a variety of perspectives, including those at the same level in the organisational chart, those above and those at lower levels – helps to frame a more complete picture of performance.

Descriptive NA NA NA MSF shows promise as a means of assessing physician competence across a broad range of competencies.

Uses 5-point rating scale

Descriptive account of development No evaluative data available/ presented.

43 London (2008)

Discussion re the need to define a gold standard for assessing practice.

Lit review/ opinion

NA NA NA Need to develop a reference standard for assessment of osteopathy competencies as a benchmark that is more explicit than exists at present.

Good points for developing 'gold standard'.

Descriptive account of development No evaluative data available /presented.

44 Lunney et al. (2007)

Psychometric testing of the Self Evaluation of CNS Competency Development tool.

Quantitative. 33 students 6 preceptors

Use of the Self Evaluation of CNS Competency Development tool by students and preceptors.


Results show partial support for criterion-related validity and construct validity. Tool reliability satisfactory.

Used self-rating on 3-point scale.

Focus on Clinical Nurse Specialists.

Page 75 of 106



Data analysis


Limitation Notes

45 Maxted et al. (2004)

Overview of the literature on assessing healthcare students in practice placements.

Literature Review.

Pool of 1000 references. 62 selected because they addressed work place assessment (WBA). Then further specific review related to healthcare professions undertaken. 43 addressed WBA.

2 researchers scrutinised abstracts. 7 abstracts rejected for irrelevance leaving 98 references.

NA. Considerable confusion between the terms competence and performance. No single method of assessment can adequately measure competence and/or performance. Need to develop existing and new methods of assessment with known validity, reliability and predictive power. Assessment of students in practice placements targets both the 3

rd and 4

th

level of Millar‟s pyramid. Considerable confusion between terms competence and performance. Still required technical studies of validity and reliability if they are used for high -stakes purposes.

Overview of literature relating to competence /performance measurement.

No search criteria, inclusion /exclusion criteria defined.

Focus on medical skills (UG & PG).

Page 76 of 106



Data analysis


Limitations Notes

46 McCarthy & Murphy (2008)

Explore to what extent preceptor nurses use the devised assessment instruments to clinically assess BSc students in one University in Republic of Ireland.

Quantitative descriptive Survey.

970 questionnaires distributed to all known preceptors. Response rate 48.5% (n=470).


Many preceptors are inexperienced, did not fully comprehend the assessment process and were not applying all of the recommended assessment strategies. Nurse educators must make assessment strategies more user-friendly and more transparent for preceptors.

Training recommendations useful.

Small sample One institution

47 McGrath et al (2006)

Reflective account of competence assessment processes

Opinion NA NA NA Competency standards in Australia not well understood and therefore not well used. Difficult to measure attributes such as attitude. Validity and reliability issues identified. More training for preceptors required.

Opinion on competence measurement and training.


Page 77 of 106



Data analysis


Limitation Notes

48 Molenaar et al. (2004)

Comparison of graded written case reports with rating of clinical performance.

Quantitative. 4 reviewers carried out rating of written case reports. Clinical supervisors (n=not defined) rated clinical performance. Analysis of grades of 710 case reports and 189 ratings.

Scoring list to grade case reports.


Ratings of clinical performance were skewed towards excellent. Rating of case reports were normally distributed. Therefore very low rating between the two. Suggest that case reports measure primarily cognitive aspects as compared with overall clinical performance. Clinical performance was not discussed amongst raters, but 'moderation' done for written case reports.

Comparison of different methods of rating competence.

Small sample. One institution used.


49 Murray et al. (2000)

Literature review to determine the extent currently available assessment approaches can measure potentially relevant medical education outcomes addressing practitioner performance

Literature review.

MEDLINE and search words defined.

NA NA Validity and reliability are likely to come from multiple sampling and triangulation of data.

Comparison of different methods of rating competence.

No inclusion/ exclusion criteria defined.

Focus on qualified medical practitioners and professional competence.

Page 78 of 106



Data analysis


Limitations Notes

50 Murray et al. (2000)

To determine to what extent currently available assessment approaches can measure potentially relevant medication education outcomes.

Literature review.

No details of the number of references included or excluded. Literature search was from USA, Canada, UK and Australia.

NA NA There are extensive tools for assessing clinical skills and knowledge. Challenge facing medical educators is to develop new tools, many of which rely on professional judgement, for assessing broader competencies and outcomes.

Discusses validity and reliability.

Not an exhaustive review as intended only as initial summary of helpful points.

Focus on medical students .

51 Neary (2000a)

Highlight concerns that students and practitioners might have about the process of assessment of their nursing practice and to suggest an alternative model

Quantitative Survey: 300 student nurses (CFP). 155 nurse practitioners. Interviews: 70 students 80 assessors .

Questionnaire Interviews (type not noted).

Not included

Students welcome flexibility; stressed by continuous assessment but seen as important as academic assessment; time should be allowed to discuss progress during and after practical experience. Students appreciated need for continuous assessment but would like change in way assessment booklets used. Students and assessors rarely knew the detail of the assessment criteria or how to interpret them. Suggests a new model of assessment called responsive assessment.

Important facts to include within training of assessors.

Small sample size but good that 3 institutions were used.

Mentions grading systems (p36).

Page 79 of 106



Data analysis


Limitation Notes

52 Nicholson et al. (2009)

Need to examine the validity and inter-rater reliability of a set of performance-based scoring rubrics.

Quantitative. 40 perioperative nurses.

3 video clips that captured varying performance or nurses performing as instrument nurses in the operating suite were recorded and used as prompts by expert raters who judged the performance using rubrics (analytical and holistic).


Holistic rubrics led to more consistent judgements than analytical rubrics, yet the latter provided more diagnostic information for intervention purposes.

4-point grading tool used.

Small sample size, one institution used.

Focus on operating theatre.

53 Norcini (2007)

Literature review of work-based assessment of clinical training for doctors.

Literature review.

NA NA NA PG assessment not as well developed as undergraduate assessment. 'Considerable research supporting' validity of single performance observation scales.

Single assessment validity discussed.

No search criteria, inclusion/ exclusion criteria defined.


Page 80 of 106



Data analysis


Limitations Notes

54 Norman et al. (2002)

Test selected competence tools for reliability and validity.

Quantitative: Multi-method approach.

257 nursing students and 43 midwifery students from 4 HEIs in Scotland.

Programme documentation Clinical assessment tools. Nursing competencies questionnaire (self-rated by students at two time points). Entrance qualifications of each student.

Inferential stats

Different methods address different abilities. No single method is appropriate for assessing clinical performance. A multi-method UK-wide strategy for clinical competence assessment for nursing and midwifery is needed. Very few students fail making the tool's ability to discriminate between competent and incompetence difficult to test. National clinical competence assessment scheme – 4 elements proposed: national (minimum standards) clinical competence assessment tool; self-assessment; assessment of clinical tasks and procedures under simulated conditions and assessment of students by patients.

Aspects important to consider in developing national tool.

Scotland only.

Page 81 of 106



Data analysis


Limitation Notes

55 NMC 2007 Standards to support learning and assessment in practice and grading of midwifery practice.

NMC Standards.

NA NA NA Statutory requirement for grading of midwifery practice. Not so for nursing.

Requirement No research data to support introduction of grading of midwifery practice.

56 NMC 2008 Standards to support learning and assessment in practice and grading of midwifery practice

NMC Standards.

NA NA NA Statutory requirement for grading of midwifery practice. Not so for nursing.

Requirement No research data to support introduction of grading of midwifery practice.

57 Oldmeadow (1996)

Reports the rationale reliability, validity and evaluation of a mastery pathway to guide practice and assessment of competence of student physiotherapists

Descriptive NA NA NA Necessary academic requirement – score must be assigned. They use 3 separate gradings with corresponding numerical scores.

4-point rating scale used.



58 Page & Knight (2007)

Not able to source reference in time

59 Panzarella (2003)


Page 82 of 106



Data analysis


Limitation Notes

60 Panzarella & Manyon (2008)

Analysis of new performance assessment tool specifically designed for assessing clinical competence in Doctoral Physical Therapy (DPT) students.

Quantitative 34 DPT students 12 expert evaluators.

Expert evaluators scored students‟ encounters with standardised patients.


Acceptable reliability was demonstrated via inter-rater agreements on items that used a dichotomous scale, whereas the items requiring the use of the 4-point rubric were somewhat less reliable. Future studies should examine the effect of rater training upon reliability.

Questions reliability of rubric scale.

Small sample size, and one institution used.

Focus on Doctoral Physical Therapy students.

Page 83 of 106



Data analysis


Limitations

Notes

61 Panzarella & Manyon (2007)

Testing reliability of a new model for assessment of clinical competence based on validation and pilot results with first year medical students in USA.

Quantitative Testing reliability: 20 faculty members Pilot testing ISPE 140 1

st year

medical students and 12 Standarised Patients (SPs) 10 Faculty members evaluated an average of 12 students each To estimate inter-rater reliability, 48 students were observed by 4 faculty members on video monitor.

Testing reliability: Faculty members given 25 minutes training on tool and rubric. They then watched 2 videotaped ISPE cases, contrasting student abilities. Raters scored students using ISPE. Focus groups after scoring was completed. Pilot testing ISPE SPs trained Students randomly allocated to one of 2 clinical cases and conducted problem-focused history with SP and answered 4 set questions from SP – allowed 15 minutes and encounter was video-taped.

Inferential statistics Within group inter-rater reliability coefficient. Kendall‟s W.

Developed Integrated Standarised Patient Examination (ISPE) – a performance assessment tool. Acceptable reliability via inter-rater agreement as well as inter-rater correlations on items that used a dichotomous scale, whereas the items requiring the use of the 4-point rubric were somewhat less reliable. The rubric seems important, however, to capture the developmental component of competency and to display growth as students progress through their educational programme.

Used a 4 point rubric and discusses reliability.

Pilot study Only on 1 cohort of 1

st year

medical students from one institution.


Page 84 of 106


Data analysis


Limitation Notes

62 Pender & de Looy (2004)

Define the key skills necessary for a competent dietetic student practitioner and to devise a reliable assessment tool to measure and track performance in these key skill areas throughout the period of clinical placement.

Quantitative 43 students. Experienced dieticians (n=27).

Students observed by experienced dieticians (n-27) as part of routine assessment during their training programme.


Confident that the assessment tool is valid and reliable and measures skill performance objectively.

Used visual analogue scale as grading tool.

Small samples size, one geographical area.

Focus on Dietetic students.

63 Pirie & Gray (2007)

Development & evaluation of a tool for assessing clinical competency for staff involved in transfusing blood.

Mixed methods Phenomenology & Survey

17 transfusion practitioners 4 assessors

Survey Individual interview

Colazzi‟s 7 step framework Descriptive statistics

Assessing clinical competency using a criterion-referenced tool gave practitioners the opportunity to relate theory to practice, promote best practice and encourage adherence to hospital transfusion policies.

Use of a 5-poiny rating scale.

Experiences presented, and not any quantitative data.

Focus on post-registration students.

Page 85 of 106



Data analysis


Limitations Notes

64 Prescott-Clements et al. (2008)

Investigate the validity of longitudinal evaluation of performance (LEP) used in assessment of postgraduate dental students.

Retrospective Quantitative.

10,000 LEPs 200 Trainers

LEP results from 2 consecutive student cohorts. Annual process evaluation questionnaire to trainers.


There is rich evidence supporting the validity, educational impact and feasibility of the LEP. Trainers were highly positive about this approach and considered it useful in identifying trainees' progress and helping to drive learning.

Used a 9-point rating scale.

Evaluation data only presented.

Focus on post-grad. dental students.

65 Pulito et al. (2007)

Determine if some performance characteristic are more highly associated with the overall faculty grade than others.

Quantitative. 211 surgery clerkship students. 3 Faculty preceptors. In total 585 student evaluations.

Faculty rated students on a 10 specific performance characteristics using a 5-point scale.


Rating on any single characteristics predicts a student‟s overall grade very accurately. Little to be gained by using multi-item forms in assessing medical students‟ clinical performance.

Used 5-point rating scale .

One geographical area, small sample size.


Page 86 of 106



Data analysis


Limitations Notes

66 Redfern et al. (2002)

Analyse methods of assessing competence to practise in nursing and to draw conclusions on their reliability and validity.

Literature Review.

No details regarding number of articles included / excluded

No mention of keywords or search terms used.

NA Findings of the review Questionnaire rating scales used to assess competence in nursing have not been rigorously tested for reliability and validity. The methodological limitations of observation can be overcome with use of criterion-referenced scales or simulations such as the OSCE or its variations, as has been found in research on competence assessment in medicine. The Bart‟s Nursing OSCE, designed for the end of the common foundation programme of the nursing diploma course, has promise because it provides a close simulation to real life and is grounded in an interpretive approach to skill acquisition. Reflection on practice using portfolios, learning contracts and self-assessment is valid if based on rigorous analysis of critical incidents rather than simple description. Reflective approaches have the advantage of being located in real-life settings and can integrate theory with practice but they need skilled collaboration between teachers, employers and practitioners. The conclusion is that a multi-method approach enhances validity and ensures comprehensive assessment of the complex repertoire of skills required of students in nursing.

Overview of variety of methods to assess competence

Only 1992-2002 included No details of literature searching methodologolgy

Page 87 of 106



Data analysis


Limitation Notes

67 Robinson et al. (2007)

Outline the process of how a system of formally assessing competence was achieved.

Quantitative. 11 trainers. 8 trainees.

Assessment tool data.

Descriptive stats and content analysis.

Identification of reasons that may have hindered widespread use of tool which was a problem within the study. Those using the tool explained that it had helped them train and assess students and had a better idea of the student‟s strengths and weaknesses.

Evaluative data on use of a grading tool.

Focus on specialist registrars.

68 Robothan (2001)


69 Sadler (2005)

Review of the most common grading policies that purport to be criterion-based.

Literature Review.

NA NA NA There is no common understanding of what criteria-based means or what it implies for practice. Additionally the concepts of criteria and standards are often confused. Despite using criteria, the fundamental judgements teachers make about the quality of student work remain subjective and substantially hidden from students‟ view. As they stand, none of the approaches identified is fully capable of delivering on the aspirations of criterion-based grading. Shifting the primary focus to standards and making criteria secondary could, however. Lead to substantial progress. Suggests standards-referenced grading.

Outlines 4 different grading models and arguments for each.

No search criteria identified.

Page 88 of 106


Sample size


Data analysis


Limitations Notes

70 Salvatori (1996)

Review of the health care literature on clinical competence with a focus on occupational therapy in Canada.

Literature review.

NA NA NA Asserts that evidence is clear that one comprehensive tool that provides a reliable, valid and cost-effective measure of clinical competence does not exist; therefore a combination of evaluation methods is needed for assessment purposes.

Reports validity and reliability issues in a number of assessment tools.

No details of search terms, inclusion / exclusion criteria; number of articles included.

Focus on OTs.

71. Schmahmann et al. (2008)

Develop a grading system that assesses multiple skills and reflects proficiency.

Quantitative longitudinal study.

409 medical students.

Bedside Examination Exercise (BEE) completed with existing techniques.

Interferential statistics.

Need for a composite score derived from different test instruments.

BEE helped to limit grade inflation by using a composite score between tests.

One geographical are, small sample size.


72. Seldomridge & Walsh (2006)

Exploration of the challenges in evaluating student performance in Preceptorship over a 3 year period.

Quantitative. Used retrospective 204 preceptor student grades.

Documentary evidence.


Discusses the challenges to preceptors, faculty and students and suggests some solutions. Favours Faculty as having primary responsibility for assigning grades for clinical performance as they are experienced educators.

95% of students scored A – challenges of grading explored.

Small sample size, one geographical area.

73. Sharp (2006) Consider whether grading of practice is possible.

Quantitative Compared Primary Education and Physical Education courses.

Primary Ed = 204 cases Physical Ed = 248 cases.

Data from assessments.


Advocates professional attainment based on a 3 point scale satisfactory/ unsatisfactory/distinction (same as medics).

Makes some good points esp. re medics & grading.

Focus on Initial Teacher Education

Page 89 of 106



Data analysis


Limitation Notes

74. Silber et al. (2004)

Ascertain whether an existing global rating scale could be modified to assess additional competencies.

Quantitative 1,295 residents

Global rating forms.


Multidimensional approach is intuitively appealing and reduces the chances that a marginal trainee will be able to compensate for serious deficiencies in one area with strengths in another.

Global rating scale use.

One geographical area.


75. Sluijsmans et al. (2008)

Portfolio scoring system including work-based assessment, simulated assessment, written work.

Description of its development.

NA NA NA Use of portfolio may provide a flexible approach to assessment of performance although the educational value has still to be investigated.

Use of multi-layered approach to practice assessment.

No evaluative data available.

76. Smith (2007) To explore midwives‟ experiences of assessing and grading clinical practice of student midwives.

Qualitative descriptive approach.

12 Midwife Mentors

Semi-structured interviews Demographic details via questionnaire.

Thematic analysis

4 main themes: clinical competence versus academic ability; ability of midwife mentor to award academic grade; grading process; social process of grading.

Advocates multi-method assessment approach to enhance validity – should combine practice assessment by mentor with that of a portfolio assessed by a lecturer.


Focus on student midwives.

77. Vess (2007) Not able to source reference in time

Page 90 of 106



Data analysis


Limitations Notes

78. Walsh et al. (2008)

To create a clinical assessment tool that was easily completed, would provide formative and summative performance feedback and could be used by multiple preceptors for any given student across a variety of hospital settings in USA.

2 year Pilot descriptive study.

3 different hospitals Preceptors, students and Faculty – but no indication of numbers involved.

Preceptors completed tool every clinical day and shared them with students who placed them in their clinical notebooks for Faculty to see. Faculty also made frequent observations of students to rate their levels of performance. A composite preceptor rating was calculated by summing the daily feedback scores. The same was done for Faculty observations. Both Preceptor and Faculty composite scores were entered into the final grade calculation.

P values Preceptor Evaluation Tool (P-CET) – outcome of pilot disappointing. Lacked useful rating scale and clear descriptions of what performance would look like at each level. Revised tool – to avoid grade inflation they will pilot a less „personal‟ delivery method for preceptor feedback, sending ratings directly to Faculty (email or post). Faculty would then provide students with their results.

Used 10-point rating scale. Revised tool used rubric with 4-point grading scale.

No indication of numbers involved in sample. No details at all re testing of the revised tool.

Emailed (24/8/09) Catherine Walsh re results of pilot for revised tool.

Page 91 of 106


Sample size


Data analysis


Limitation Notes

79. Walsh & Seldomridge (2008)

Examine the relationship between grades earned in theory and practice.

Quantitative. 10 paired courses were compared.

Documentary evidence of grades achieved.

Wilcoxon rank tests.

Grades for theory courses approached normal distribution and had a slight positive skew, while grades for clinical courses did not have a normal distribution and were negatively skewed.

Used grading system.

Number of student in each course not defined – aggregated data were used.

80. Walsh et al. (2009)

OSCE as a measurement of competence.

Literature review

NA NA NA Major gaps exist in the literature regarding the examination of psychometric properties of the OSCE, the suitability of OSCE design, structure and tools employed and costs associated with this methods of assessment.

Not relevant to grade inflation or grading tools. Advocates research to combine methods of competency assessment.

81. Wass et al. (2001)

Discussion about validity and reliability in assessing clinical competence for post-graduate medical students.

Opinion/ Literature review.

NA NA NA Assessment of performance in real practice has reliability and validity issues. Current methods rely too heavily of knowledge-base assessment rather that the assessment of „doing‟.

Need for combination of assessment types e.g. OSCE, practice, oral exams.

No search criteria or databases defined.


Page 92 of 106



Data analysis


Limitations Notes

82. Watson et al. (2002a)

Reliability and validity of clinical competence assessment instruments for pre-registration nursing and midwifery students in Scotland.

Quantitative. Nursing Competence Questionnaire used 2 times in branch programme 6 months apart – 300 students time 1 and 287 students at time 2.

Nursing Competence Questionnaire (NCQ).


Self-assessment should form an important element in the overall assessment of student nurses. The NCQ could form part of a multifaceted assessment of clinical competence.

Self-assessment using a grading tool.

Small sample, one geographical area.

83. Watson et al. (2002b)

What is the research evidence for the use of clinical competence assessment in nursing?

Systematic review.

Search terms identified. 245 papers – 212 deemed relevant. 61 papers selected for review.

Searched between 1980 and 2000.

NA Generally speaking there are very few papers in this field which have sufficient rigour to inform the debate about clinical competence, many papers are hard to classify on the basis of method and very few address the measurement of clinical competence rigorously.

Questions rigour of instruments used.

84. Weaver et al. (2007)

Test the hypothesis: a simple change of the shift grading cards, using explicit criteria, would decrease grade inflation and aid to redistribute the shift evaluations.

Quantitative: Before and after study on emergency medical clinical clerkship.

1612 before change evaluations. 1737 after change evaluations. Students on emergency department clerkship.

Completed shift evaluation cards.


A simple change in shift evaluation cards to include more explicit grading criteria resulted in a significant change in grade distribution and greatly decreased grade inflation.

Part-remedy to grade inflation.

One geographical area, no sample size given Description of data collection and analysis sketchy.

Page 93 of 106



Data analysis


Limitation Notes

85. Webb & Shakespeare (2008)

To deepen understanding of how mentors make judgements about students‟ clinical competence.

Qualitative. Convenience sample of 9 3

rd year

students. 10 experienced mentors. 5 inexperienced mentors.

Critical incident technique in interviews (group and individual – face to face and telephone).

Thematic analysis.

Three themes: good mentors; good students; mentoring relationships. Linchpin in making judgement is the mentoring relationship. Make some pertinent recommendations.

Useful background re mentor training and support only.

86. Wilkinson et al. (2008)

To evaluate the reliability and feasibility of assessing performance of medical specialist registrars using 3 methods.

Quantitative: Feasibility study.

Trainees – 128, 59, 230 for mini-CEX, DOPS (Directly observed procedural skills) and MSF (Multi-source feedback) assessments respectively.

Mini-clinical evaluation (mini-CEX; DOPS; MSF.


The methods are feasible to conduct and can make reliable distinctions between doctors‟ performance. However there were some doctors who scored satisfactory overall, but obtained unsatisfactory for some aspects of care.

Grading system used and reported.

Focuses on medical specialist registrars.

87. Yarborough (2008)

Assess psychometric rigor of a criterion-referenced examination constructed to test nursing student‟s clinical competence.

Quantitative Random sample of 1304 examinations selected over a year.

Not enough detail in abstract.

Not enough detail in abstract.

Reliability estimates indicate that battery of tested areas provides a stable measure of clinical competence. Comparison to external measures provides evidence to support conclusions that this examination is a valid predictor of graduate success.

Final decision once full text seen.

88. Yorke (2005) Literature review on practice-based professional learning.

Literature review/ opinion.

NA NA NA Discusses use of „level‟ – level of gradation of actual performance at a particular level. Concludes that concept of „level‟ is more problematic than some think.

Level discussion pertinent.

No search criteria or databases searched included in report.

Page 94 of 106

Hand search literature – Included papers


Sample size


Data analysis


Limitations Notes

1 Alexander (1996)

Investigate whether or not subjective judgements are involved in the assessment of physiotherapy students on clinical placement (in Scotland).

Quantitative: Preliminary study.

Unclear as to the sample size as the figures not included in text.

Clinical Placement Diaries over a 3 week period. Distributed to all clinical tutors, visiting lecturers and 2

nd & 3

rd year

physio students.

Unclear as figures missing from paper.

Preliminary results demonstrate that physiotherapy assessors made subjective judgements of their students, that these contributed to the determination of assessment grades and that in certain cases these judgements were erroneous.

Reference to grade inflation.

Low response rates of between 32% and 43% for the three different groups.

Focus on physio students.

2 Andrew (2009)

Discusses grading using rubrics – holistic and analytical.

Opinion. NA NA NA A hybrid of holistic and analytical can provide a traditional overall grade, while providing detailed analysis of the student‟s work.

Use of hybrid model of rubric suggested.

3 Battistone et al. (2001)

Test hypothesis: that the RIME vocabulary used in the setting of formal evaluation and feedback sessions would be more responsive to change in student performance than would a methods of global numeric ratings.

Quantitative 97 students Evaluators

Standarised clerkship evaluation form. Evaluators asked to describe students‟ performances using the RIME vocabulary.

Paired t-tests

Using a method of descriptive assessment, instructors were able to detect and describe significant changes in students‟ performances over each successive three-week interval of observation, in contrast to evaluations using the global numeric system. Researchers assert that their method can limit grade inflation.

Used 4 point scale Addresses grade inflation.

No indication of the number of Evaluators within the study. Study not randomised. Single group of students in one institution.

RIME = reporter to interpreter to manager / educator

Page 95 of 106


Data analysis


Limitations Notes

4 Biggs (1992) Description of a grading system that uses categories denoted by letter-grades, where the categories are hierarchical, reflecting successively higher cognitive levels.

Opinion/ descriptive

NA NA NA By treating grades as profiles, it is easily possible to combine them without making arbitrary and misleading quantitative assumptions.

Focus on grading

5 Bloodgood et al. (2009)

To measure the impact of a change in grading system in the first 2 years of medical school from graded to pass/fail on medical students‟ academic performance, attendance, residency match, satisfaction and psychological well-being (USA)

Quantitative: Comparative study.

Web-based survey to class using grading tool (n=141) and those using pass/fail (n=140). Response rate 75%.

Student course performances: Grading tool n=141; Pass/fail n=140.

Independent t tests. Chi square. Cohen effect. P values.

The move from grading to pass/fail was not associated with a decline in students‟ academic performance. Change to pass/fail grading system resulted in a statistically significant improvement in students‟ well-being.

Focus on grading.

One institution.

Focus on medical students,

Page 96 of 106



Data analysis


Limitations Notes

6 Borman (1975)

Investigate the effects of a short training session designed to reduce halo error in performance ratings.

QuantitativeOne group, pre-test-post-test design.

90 middle managers.

Vignettes and a 7 point rating scale.

One-tailed t test; Spearman-Brown.

Results suggest that a brief training session may be successful in reducing halo effect significantly. However inter-rate agreement results indicate that making greater distinction among different performance categories may lead to a reduction in inter-rater reliability as well.

Uses 7 point grading scale. Training session.

No control group.

Referred to by a number of other authors in recent times.

7 Burchell et al (1999)

Created a revised summative assessment tool in light of findings from data collected.

Quantitative: Action research.

8 representatives from clinical radiography practice and university within action research group, 75 lecturers/ practitioners.

75 (response rate 61%) answered questionnaire, 14 interviews.

Not defined. Revised assessment tool incorporated professional judgement and practical task skills.

Grading using 6-point scale – used number, scale and description of each point on the scale.

Response rate, one geographical area, data analysis not explicit.

8 Colletti (2000)

A new ward evaluation system (NS) was compared with the standard written ward evaluation system (OS) using a standard t-test.

Quantitative. 24 medical students on a surgery rotation.

Assessment tools: New ward evaluation system (NS) and standard written ward evaluation system (OS)

Student‟s t-test.

Faculty poor at giving direct, objective, face to face feedback, particularly when it involves negative feedback, with resultant grade inflation.

Uses 6 point grading scale.

Small sample. Only one institution.


9 Collins et al. (2004)

All Wales Policy of pre-registration.

All Wales Policy of pre-registration – descriptive account of development

NA NA NA Assessment of clinical practice uses pass / fail grades but assessment of professional attitudes uses a 7 point rating scale.

Uses 7 point rating scale.

Page 97 of 106


Sample size


Data analysis


Limitations Notes

10 Cooper (1981)

Review of the literature around rating scales in general.

Literature review.

NA NA NA Improving scales can be done by training workshops, increasing rater-rater familiarity, have content validity and discussion on how to make judgements.

Rating practices and development

Search criteria not defined.

11 Davis et al. (2006)

To determine how accurately physicians self-assess compared with external observations of their competence.

Systematic Review.

725 articles with 17 meeting inclusion criteria.

NA NA Physicians have a limited ability to accurately self-asses. The process currently used to undertake professional development and evaluate competence may need to focus more on external assessment.

Accuracy of self-assessment.

Focus on qualified physician.

12 Domingues et al. (2009)

Identify which aspects of clinical competence are assessed through global ratings.

Quantitative 106 students scored by 19 faculty members.

Global rating assessments


When faculty staff attribute a global single-item overall rating to a student‟s clinical competence, they tend to focus more on technical skills.

Used 10 item grading scale.


Page 98 of 106


Data analysis


Limitations Notes

13 Downing & Haladyna (2004)

Discuss 2 major threats to validity: construct under-representation and construct-irrelevant variance.

Discussion paper.

NA NA NA Construct under-representation threats relate primarily to under-sampling or biased sampling of the content domain or the selection or creation of assessment items or performance prompts that do not match the construct definition. Construct-irrelevant variance threats introduce systematic, rather than random, measurement error, and reduce the ability to interpret assessment outcomes in the proposed manner.

Discusses validity issues.

14 Dudek et al. (2005)

Explore factors identified by Canadian clinical supervisors that affect their willingness to report poor clinical performance.

Qualitative: Grounded Theory.

21 clinical supervisors

Semi-structured interviews

Constant Comparative Analysis NUD*IST

Clinical supervisors know how to use the assessment forms but lack a willingness to fail students because of 4 areas: lack of documentation; lack of knowledge of what to specifically document; anticipating an appeal process and lack of remediation options.

Focus on ways to improve training / education of clinical supervisors.

Used a modified form of constant comparative analysis.

Page 99 of 106


Data analysis


Limitations Notes

15 Gonnella et al. (2004)

Examine predictive validity of number grades in medical school.

Retrospective quantitative study.

6656 medical students.

Retrospective documentary analysis.


Scores were progressive over years of medical school and if grading system is appropriate, it should be predictive of future educational and professional outcomes.

Rating used 4-point rating scale.

One geographical area.

16 Hardcastle (1999)

To develop an practice assessment tool for CBT that combines observation of real life practice and viva style questioning with an assessment marking grid (Eastbourne Assessment of Practice Scale).

Descriptive. Not included. Not included. P values. Within the specific area of CBT, the assessment tool was found to have good inter-rater reliability. Face validity also considered good.

Includes a 5 point scale.

No details of the determination of inter-rater reliability of tool was included.

Focus on mental health students doing CBT.

17 Heneman (1988)

To test rater format and rater training on performance appraisal.

Quantitative 2x3 factorial design.

87 supervisors in a utility company.

All undertook same training programme, and videotaped performance scoring scale.


Traits were more accurately rated than behaviours.

Interesting points on training needs.

Page 100 of 106


Data analysis


Limitations Notes

18 Karayurt et al. (2009)

To develop a valid and reliable scale to asses nursing student performance in clinical settings in Turkey.

Quantitative: Instrument Development Study over 2 years.

52 3rd

year students. 45 4

th year

students. In total 350 performance evaluations of 97 students.

Students‟ performance was repeatedly evaluated by lecturers.

SPSS. Exploratory factor analysis. Orthogonal Varimax rotation method. Cronbach‟s alpha analysis.

The Nursing Student Clinical Performance Evaluation Scale had three factors and the reliability coefficient of both the scale in general and of its subscales was high. For this reason, the scale can be considered valid and reliable and can be used for the performance evaluation of nursing students.

Used 10 point rating scale.

The scale used is based on 3 factors: Nursing process; professionalism and ethical principles. Contains no competencies

Page 101 of 106


Data analysis


Limitations Notes

19 McManus et al. (2006)

Carry out a multi-facet Rasch modelling of the paired judgements made by examiners in the clinical examination (PACES) of the MRCP (UK), where identical candidates were assessed in identical situations allowing calculation of examiner stringency.

Quantitative: Incomplete paired comparison design.

Total number of candidates taking exam was 10,145, an average of 1,127 on each diet. Overall the analysis considered a total of 142030 marks.

Each candidate on each of the 5 stations were always seen by 2 examiners. Results from the first nine diets of PACES (2001 to 2004) were analysed. Where possible demography of examiners and candidates were gathered.

Bradley-Terry-Luce Model. Rasch modelling.

FACETS analysis is appropriate and adequate as a first approach to studying examiner effects on leniency and stringency. Found highly significant differences in examiner behaviour. The reasons for the differences is unclear. Might be an argument for pairing examiners on the basis of their stringency so that if a candidate sees one examiner known to have a high stringency then the other will have a relatively low stringency.

Makes reference to judgement in assessing clinical skills and grade inflation issues.

Authors report that multi-facet Rasch model has some limitations and caution the total accuracy of the results.

MRCP (UK) = Membership of the Royal Colleges of Physicians of the UK PACES = Practical Assessment of Clinical Examination Skills

20 Moskal & Leydens (2000)

Discussion paper addressing issues of validity and reliability and the role of rubrics.

Discussion paper

NA NA NA A scoring rubric is likely to result in invalid interpretations, for example, when the scoring criteria are focused on an element of the response that is not related to the purpose of the assessment.

Focus on validity and reliability.

Page 102 of 106



Data analysis


Limitations Notes

21 O‟Brien et al. (2008)

To evaluate a rubric-based method of assessing pharmacy students‟ case presentation.

Quantitative. 86 pharmacy students.

Documentary evidence from scores – three levels of competency were scored and a numerical composite score calculated.


Assessor scores consistently lower for knowledge than self-scoring, but consistently higher for behaviour than self-scoring. Scores for skills demonstrated no difference. Peer assessment scores were higher than assessor and self assessment scores.

Used rubric composite score.


22 Ravelli & Wolfson (1999)

To compare the grading systems and grading patterns for 3

rd year

medical students in surgery throughout USA and Canada and to attempt to determine an „ideal‟ grading system.

Survey 216 questionnaires sent out to Surgery Clerkship Directors. 166 questionnaires received – response rate = 77%.


An ideal grading system should consist of descriptors with 4 or 5 categories.

Focus on grading systems.

No mention of piloting. No indication of how many items in questionnaire.

Page 103 of 106



Data analysis


Limitations Notes

23 Rohe et al. (2006)

To objectively measure the effect of a pass-fail grading system on stress, mood, group cohesion and test anxiety in medical students (USA).

Prospective quantitative study.

2 classes of medical students – no numbers provided.

4 self-report instruments: Perceived Stress Scale; Profile of Mood States; Perceived Cohesion Scale and Test Attitude Inventory.

2-sample Wilcoxon rank sum test. 2-sample t test. P values .

Students graded with the pass-fail system had less perceived stress and greater group cohesion than their counterparts tested using the 5 interval grade.

Focus on grading systems.

No indications of how many students participated. No details re response rate.


24. Speer et al. (2000)

To determine if grade inflation exists.

National Survey in USA.

Total population of Internal Medicine Clerkship Directors (n=125). 83 completed survey – response rate = 66%.

Survey questionnaire.

P values. Grade inflation does exist in Internal Medicine Clerkships. 43% of Internal Medicine Clerkship Directors felt that some students pass who should have failed the clerkship. Grade inflation could be limiting our ability to identify student performances below the minimum pass level.

Addresses grade inflation issues.

Response rate, sample size.

25. Taleghani et al. (2006)

Paper discussing merger of graded and non-graded aspects of student assessment.

Descriptive examples of combination of grades to provide overall grade.

NA NA NA Multiple strategies required to provide a reliable grade point average.

Perspective on combining grades.

Very descriptive, one geographical area.

Focus on dentistry students.

Page 104 of 106


Sample size


Data analysis


Limitations Notes

26. Tolhurst and Bonner (2000)

Explores the development and implementation of specific criteria to determine the level of clinical performances of postgraduate nursing students during 1

st year

of Master of Nursing Course.

Descriptive Not defined.

Not defined. Not defined.

Concluded that the tool was easy to interpret, and suitable for postgraduate nursing.

Uses novice to proficient as grading tool.

No data presented, one geographical area.

Focus on PG nursing students.

27. Truemper (2004)

Discussion paper re use of rubrics in assessment of nurses.

Discussion paper

NA NA NA Scoring rubrics can be used for both written and oral assignments and can be individualised to fit the context of the subject matter. Rubric also facilitates grading consistency.

Reference to grading consistency.

Focused mainly on written assessments.

Page 105 of 106



Data analysis


Limitations Notes

28. Williams & Bateman (2003)

In the absence of a clear policy on graded assessment in vocational education and training (VET), a range of practice has evolved in the Australian national training system. The purpose of this study was to update the research to take account of the changes in the VET environment.

Mixed method study – survey and interviews.

120 stakeholders.

Survey of state training authorise in Australia. Focus groups, telephone and face-to-face semi-structured interviews.

Thematic analysis.

Proposed principles for graded assessment: criterion-referenced; applied once competence is determined; transparent; discretionary.

Reference to grading as well as validity and reliability.

Page 106 of 106

Grey literature: included items

Author Nature of literature Nature of grading Rationale for inclusion

Limitations Notes

1. Lee, E. (2009)

Email communication regarding the development of a set of grading criteria for

years 2 and 3 of pre-registration midwifery programme in Tayside. They

assess but do not grade year 1. The grading taxonomy is a broad adaption of

Bondy and the SCQF level descriptors and uses qualitative descriptors that

mentors are familiar with.

Use a 5-point grade: U = unsatisfactory S = satisfactory G = good VG = very good E = excellent

Uses rating scale

2. McFarlane, W. (2009)

Students must pass all competencies as set out in the Competency

Assessment Book for each placement. The competencies assessed in each

placement are grouped into four elements and all competencies in each

individual element must be achieved. The elements criteria statements reflect

the Scottish Credit Qualification Framework (SCQF) Levels 7, 8 and 9 that are

relevant to the student‟s stage of study in the programme.

The sign-off mentor uses the Practice Marking Tool to assess the student‟s overall performance in achieving the competencies within each element. The mentor decides on a mark from the range of marks available (100-80, 79-60, 59-40 and 39-0) that reflects the overall performance of the student using the competency elements criteria. The pass mark for each placement must be 40 or more.

Uses rating scale

3. # Meier, K. (2009)

Email communication regarding details of practice assessment of nursing

students at Addenbrookes. This is the focus of her PhD study – early stages.

Don‟t grade but use pass / fail in the first two years of programme and in third year use 4 point scale in year 3 (fail / pass/ good / excellent).

Uses rating scale

Date post:	09-Jul-2020
Category:	Documents
Upload:	others
View:	2 times
Download:	0 times

National Approach to Practice - NES...assessing clinical practice in nursing students [Spanish]."...

Documents