SCHOLARLY PROJECT:DEVELOPING A CLINICAL RUBRIC Erin Kibbey

Ferris State University

Objectives

Describe the project goals and process taken in developing a clinical rubric

Discuss relevant research and survey results influencing rubric development

Describe a plan for future implementation of clinical rubric

Project Description

Goals Create rubrics for measuring clinical competency of

critical care interns at Munson Implement rubrics for use by preceptors working with the

interns Objectives

Obtain references containing information about the use of rubrics and their development

Collaborate with team and staff educators for input Create an evaluation tool Present rubrics to preceptors Trial rubrics on units Obtain and compile feedback

Procedure/Methodology

“A carelessly planned

project takes three times longer to

complete than expected; a

carefully planned

project takes only twice as

long” ~ Golub's Law

Procedure/Methodology

Develop Rubric

Review Standard

s

Needs Assessm

ent

Research

Procedure/Methodology - Step One

Research Topics Clinical performance evaluation Evaluation strategies Using rubrics as a clinical evaluation

method Rubric development Clinical competency development Adult learning theory Cognitive learning theory

Clinical Evaluation Research

Issues Consistency Subjective Values

Norm-referenced: Compared to others

Criterion-referenced: Compared to pre-determined criteria Known in advance &

used as basis for evaluation

Observing performance & arriving at judgments

Compares data collected about performance to a set of standards

(Bonnel, 2008; Gaberson & Oermann, 2010)

Best Practices for Clinical Evaluation Extensive formative evaluation and periodic

summative evaluation Timely & continuous feedback with ways to

improve Based on preset outcomes, clinical objectives,

or competencies Without a guide to the process = No basis for

evaluation NEED a framework for observing performance and

arriving at judgments Preparation and tools to promote consistency in

evaluation (Bonnel, 2008; Gaberson & Oermann, 2010)

Rating Scales & Rubrics

Creation and use of rating scales or rubrics is supported in a wide range of academic subjects Economics, writing, speech, dentistry, and

chiropractic medicine (McGoldrick & Peterson, 2013; O’Donnell et al., 2011; Rezaei &

Lovorn, 2010; Saxton, Belanger, & Becker, 2012; Xiaohua & Canty, 2012)

Can be used for self-evaluation (Bonnel, 2008)

Despite the lack of consistent use of rubrics in clinical nursing education, rubrics have many benefits (Frentsos, 2013)

What is a Rubric?

Rubrics are “scaled tools with levels of achievement and clearly defined criteria placed in a grid. Rubrics establish clear rules for

evaluation and define the criteria for performance”

(O’Donnell, Oakley, Haney, O’Neill, & Taylor, 2011, p. 1163)

What Does the Research Say?S

evera

l B

en

efits

Support adult learning principles

Provide competency documentation required by regulatory agencies

Can improve quality of care Allow more discrimination in

judging behaviors Can increase knowledge gain(Bonnel, 2012; Frentsos, 2013; O’Donnell et al., 2011;

Walvoord & Anderson, 2010)

Procedure/Methodology – Step Two

The first step to rubric

creation is

reflection

(Stevens & Levi,

2005)

Needs assessment survey Target Population -

Critical care intern preceptors

32 Question Survey Sent to 55 Preceptors

Respondent Demographics

12.5%

37.5%22.5%

27.5%

What department do you currently work

and precept in?

ED A2 A3 ICU

DepartmentYears Precepting in Department

17.5%

25.0%27.5

%

15.0%

15.0%

Number of years serving as a precep-

tor in current de-partment at Munson?

less than 1 year

1-3 years

4-6 years

7-10 years

more than 10 years

Respondent Demographics

ED A2 A3 ICUED A2 A3 ICU

02468

101214161820

5

15

911

5

20

1416

Survey Respondents By Unit

# of Respondants Total # of Surveys Sent

Key Findings/Results 1

Strongly Agree

Agree Disagree Strongly Disagree

Unable to comment

0

5

10

15

20

25

4

10

21

2 1

I have a tool I consistently use to provide written feedback.

Nu

mb

er

of

Resp

on

ses

Key Findings/Results 2

Strongly Agree

Agree Disagree Strongly Disagree

Unable to comment

02468

101214161820

2

18

12

24

Interns have a tool they con-sistently use for self-assess-

ment of learning.

Nu

mb

er

of

Resp

on

ses

Key Findings/Results 3

Strongly Agree

Agree Disagree Strongly Disagree

Unable to comment

0

5

10

15

20

25

6

20

9

2 1

Evaluating progression is in-consistent among preceptors.

Nu

mb

er

of

Resp

on

ses

Key Findings/Results 4

Strongly Agree

Agree Disagree Strongly Disagree

Unable to comment

02468

1012141618

2

16 16

4

0

My unit uses a uniform method for measuring progression of compe-

tence.

Nu

mb

er

of

Resp

on

ses

Key Findings/Results 5

Strongly Agree

Agree Disagree Strongly Disagree

Unable to comment

0

5

10

15

20

25

0

12

21

41

My unit uses a tool that accu-rately communicates levels of

progress.

Nu

mb

er

of

Resp

on

ses

Key Findings/Results 6

Strongly Agree

Agree Disagree Strongly Disagree

Unable to comment

0

5

10

15

20

25

0

12

22

31

The criteria for evaluating the progression of competence is

clear.

Nu

mb

er

of

Resp

on

ses

Key Findings/Results 7

Strongly Agree

Agree Disagree Strongly Disagree

Unable to comment

0

2

4

6

8

10

12

14

16

5

13

15

3

1

I am familiar with the use of rubrics as an evaluation tool.

Nu

mb

er

of

Resp

on

ses

Key Findings/Results 8

Strongly Agree

Agree Disagree Strongly Disagree

Unable to comment

02468

1012141618

3

16

3

0

15

Rubrics would be beneficial for use in evaluating progression

of competence.

Nu

mb

er

of

Resp

on

ses

Key Finding/Results 9

Strongly Agree

Agree Disagree Strongly Disagree

Unable to comment

0

5

10

15

20

25

2

20

6

0

9

I am interested in using rubrics for evaluating progression of

competence.

Nu

mb

er

of

Resp

on

ses

Procedure/Methodology – Step 3 How is competence defined?

ANA (2010) Expected level of performance that integrates

knowledge, skills, abilities, and judgment Can be influenced by the nature of the situation

IOM (2002) Five Domains: Patient centered care,

interdisciplinary team, evidence-based practice, quality improvement

QSEN adds to these domains: safety & informatics (Cronenwett, et al., 2007)

Procedure/Methodology – Step 3 Standards of Practice

QSEN - Quality & Safety Education for Nurses ANA – American Nurses Association AACN – American AssociationOf Critical-Care Nurses IOM – Institute of Medicine NCSBN – National Council of State Board of Nursing Joint Commission

Benner’s Novice to Expert Model(ANA, 2010)

Rubric Development

3 Main partsScale, Dimensions, Description of dimensions

(O’Donnell et al., 2011)

Rubric Development

Reflection

Defining Learning Objectives

Grouping Similar Components

Applying dimensions & descriptions

(Kirkpatrick & DeWitt, 2012; O’Donnell et al., 2011; Stevens & Levi, 2005)

Important Design Considerations Ease of use Valid & Reliable Fair Flexible & Adaptable Evaluated

(Bargainnier, 2003; O’Donnell et al., 2011; Stevens & Levin, 2005)

What’s the Next Step?

Feedback & evaluation from you!

Metarubric Reliability and

validity

Rubric evaluati

on

Preceptor

Education

Trial Use

Feedback &

Revision

(Stevens & Levi, 2005; O’Donnell et al., 2011)

Questions and Discussion

References

American Nurses Association [ANA]. (2010). Nursing scope and standards of practice (2nd ed.). Silver Spring, MD: American Nurses Association.

Bargainnier, S. (2003). Fundamentals of rubrics. Retrieved from http://www.webpages.uidaho.edu/ele/scholars/practices/Evaluating_Projects/Resources/Using_Rubrics.pdf

Bonnel, W. (2012). Clinical performance evaluation. In D. Billings & J. Halstead (Eds.), Teaching in nursing: A guide for faculty (4th ed.). (pp. 485-502). St. Louis, MO: Elsevier Saunders.

Cronenwett, L., Sherwood, G., Barnsteiner J., Disch, J., Johnson, J., Mitchell, P., . . . Warren, J. (2007). Quality and safety education for nurses. Nursing Outlook, 55(3), 122-131.

Frentsos, J. M. (2013). Rubrics role in measuring nursing staff competencies. Journal for Nurses in Professional Development, 29(1), 19-23.

Gaberson, K. & Oermann, M. (2010). Clinical teaching strategies in nursing (3rd ed.). NY: Springer Publishing Company.

Hallgren, K. A. (2012). Computing inter-rater reliability for observational data: An overview and tutorial. Tutorials in Quantitative Methods for Psychology, 8(1), 23-34.

Institute of Medicine [IOM]. (2002). Who will keep the public health: Educating health professionals for the 21st century. Washington, DC: National Academy Press.

References continued

McGoldrick, K., & Peterson, B. (2013). Using rubrics in economics. International Review of Economics Education, 12, 33-47.

O’Donnell, J.A., Oakley, M., Haney, S., O’Neill, P.N., & Taylor, D.  (2011). Rubrics 101: A primer for rubric development in dental education.  Journal of Dental Education, 75(9), 1163-1175.

Rezaei, A. R., & Lovorn, M. (2010). Reliability and validity of rubrics for assessment through writing. Assessing Writing, 15, 18-39.

Saxton, E., Belanger, S., & Becker, W. (2012). The critical thinking analytic rubric (CTAR): Investigating intra-rater and inter-rater reliability of a scoring mechanism for critical thinking performance assessments. Assessing Writing, 17(4), 251-271.

Stevens, D. D., & Levi, A. J. (2005). Introduction to rubrics: An assessment tool to save grading time, convey effective feedback, and promote student learning . Sterling, VA: Stylus. Retrieved from https://resources.oncourse.iu.edu/access/content/user/fpawan/L540%20_%20CBI/steven-rubrics.pdf

Walvoord, B., & Anderson, V. A. (2010). Effective grading: A tool for learning and assessment. San Francisco, CA: Jossey-Bass.

Xiaohua, H., & Canty, A. (2012). Empowering student learning through rubric-referenced self-assessment. Journal of Chiropractic Education, 26(1), 24-31.