Competency Workshop

MedbiquitousMay 2011

Susan Albright

Workshop Objectives

By the end of this workshop, participants will be able to:

Describe a process for developing competency frameworks

Identify challenges and potential barriers to developing and implementing competency frameworks

Identify opportunities available with the implementation of competency frameworks in an electronic system

Agenda

Introductions Part 1 – Background on the Tufts

Experience Activity

Part 2 Assessment Discussion

Break Part 3 – Developing the system Discussion Summary

Why Competencies?

“Competency-based education is characterized by the development of clearly defined and explicitly published competencies, mapping the curriculum to achieve competencies, and assessment process matched to competencies.”

(Ron Harden, Medical Teacher 2002 #24 Vol 2

 

What is Competency Based Education CBE is an institutional process that moves education

from focusing on what academics believe graduates need to know (teacher-focused) to what students need to know and be able to do

CBE is focused on outcomes (competencies) that are linked to workforce needs as defined by the profession.

CBE necessitates complex assessment through portfolios or experiential learning assessement in field experience

Large skill sets are broken down into competencies which sequestial levels of mastery

CBE

Competencies may reinforce one another from basic to advanced as learning progresses

Continual refinement of competencies is essential as this a process not a product

Consistency in expectations across the curriculum

Consistency in gradinghttp://www.ceph.org/pdf/Competencies_TA.pdf

Council on Education for Public Health

Competency Frameworks

ACGME Scottish Doctors Tomorrow’s Doctors The Tuning Project CanMeds MSOP AAMC

ACGME Competency Domains

Patient care Medical knowledge Practice-based learning and

improvement Interpersonal and communication

skills Professionalism Systems-based practice

AAMC Learning objectives

The first of the four attributes is that physicians must be altruistic. There are seven learning objectives, including the objective that before graduation, the student can demonstrate compassionate treatment of patients and respect for their privacy and dignity.

The second attribute is that physicians must be knowledgeable; one of the six learning objectives is that the student can demonstrate knowledge of the normal structure and function of the body and of each of its major organ systems.

The third attribute is that physicians must be skillful; one of the eleven learning objectives is that the student have knowledge about relieving pain and ameliorating the suffering of patients.

The last attribute is that physicians must be dutiful; one of the six learning objectives is that the student have knowledge of the epidemiology of common maladies within a defined population, and the systematic approaches useful in reducing the incidence and prevalence of those maladies.

Planning Process at the Tufts Cummings Veterinary School

One year + planning process involving:•Educational Dean•Chair of Rotation Directors •All Rotation Directors•Support Workshops provided by the Tufts Center for Teaching and Learning

Part 1 The Tufts Cummings Veterinary School Experience Start with Competency Statements of

accrediting body – AVMA Localize to School Wide graduating

competencies for clinical rotations Workshops on CBE and competency

writing Build rotation/clerkship Competencies Produce Grading Rubrics

Vet School Experience Match each rotation back to the school

wide competencies All competencies come back to

curriculum committee for approval Informatics had not been identified by

any rotation – curriculum committee added these competencies

Curriculum committee added the preclinical competencies

Helpful Tips Provide a good model Provide support – workshops Do not insist on uniformity Make sure grading rubrics match

back to competencies Identify it as a work in progress Hard part was getting the attention

of the rotation directors

Helpful Tips

One person brings it all together Work with external bodies –

curriculum committee to provide overview

How to write Competencies

Understand the level (school-wide, course, content)

Action Words Verbs followed by nouns Measureable and/or observable Performance based

How to write….

Consider relationships up and down the framework

Do not use evaluative words in the competency (this will come later)

Who owns the competency? Who can create? Who can modify?

Feel free to use terms like ‘outcomes’, learning objectives

Part 2

Assessment and grading rubrics

How rubrics help: For Courses:

Rubrics enable faculty to clearly communicate expectations for student performance to students. They support more consistent and objective assessment of student work. They also facilitate detailed feedback to students. When areas of strength and weakness in performance are noted, students more easily comprehend the reasoning behind their professor’s assessment of their work.

A rubric is a scoring guide that seeks to evaluate a student's performance based on the sum of a full range of criteria rather than a single numerical

score. For Student Outcomes Assessment:

Rubrics facilitate quality conversations among faculty about student learning, shorten the time it takes to do SOA, and contribute to the validity and reliability of the assessment process.

- Stevens and Levi, Ch. 2www.uni.edu/chfasoa/IntroductiontoRubrics.ppt

Developing a Rubric cont. Scale

How well or poorly has the student done on the task?

Sophisticated, competent, partly competent, not yet competent. Exemplary, proficient, marginal, unacceptable. Advanced, Intermediate, Novice. Distinguished, proficient, intermediate, novice. Accomplished, developing, beginning. (Huba and Freed, 2000)

Note: Consider whether a scale needs more than three levels. Some research indicates that information about student learning obtained from a three-level scale is comparable to that obtained from a five-level scale. More levels typically means more time spent on assessment.

Developing a Rubric cont.

Dimensions

Dimensions break down a task into components and identify the importance of these components.

Dimensions are descriptive, not evaluative (e.g., “organization” not “good organization”).

Dimensions help students see that the work they are doing is multidimensional and draws on multiple abilities.

Developing a Rubric cont.

Description of the Dimensions

Rubrics should contain at least a description of the highest level of performance.

With experience and in response to the complexity of the performance, descriptions of all levels can be written.

Students need not fit cleanly into a single category. On oral presentation skills, a student might speak in a clear voice but lack eye contact.

Descriptions for each level help students see that the work they are doing does have varying levels of achievement. There is a difference between poor and excellent work.

Examples of Grading Rubrics

handout

Competency Assessment- TUSK mock-ups

Planning Process

After the competencies and grading rubrics TUSK had a series of meetings with :•Educational Dean•Chair of Rotation Directors •All Rotation Directors

•Iterative process of moving current process to Online version integrated with TUSK• Create Powerpoint ”story board” which developer used as functional spec

Process to get to competency assessment in TUSK

TUSK meetings with vet school leaders Understand business processes Translate these to the online world Develop powerpoint story board to achieve

agreement of functional spec Iterate over 7 months Check in with all rotation directors

Test and train Pilot test in two clerkships Release to all clerkships

View the Process through Every Lens Course Administrators

Create form Link to competencies Link Faculty to students

Faculty Assessors Able to select/deselect students See picture of student Easy access to form

Course Director See each assessor’s view as well as summary Able to add comments Able to override scores

Student View the results Compilation of all clerkship results

Registrar Access final grade (allow resubmit)

Competency AssessmentFeatures

Prepare TUSK to accept competencies Link to School-wide/course competencies Configurable form for each course Multiple assessors Access to Registrar Show scores or only feedback Multiple rubrics per form Override score/grade Student retains access to scores over time

Overview

I. Admin Viewa) Populate Competency Datab) Populate Competency

Assessment Form & Datac) Other Assessment Form Data

II. Faculty Viewa) Access to the Assessmentb) Selecting Additional Studentsc) Assessment Form

III. Director Viewa) Assessment Results for All

Studentsb) Completion Report

III. Director View (con’t)c) Individual Student Resultsd) Completed Faculty/Staff

Forme) Summary Form

IV. Student Viewa) Accessing Assessment

Results

V. Registrar ViewIV. View final grades

I. Administrator View

Creating and Populating

Competency Assessment Data

POPULATE COMPETENCY DATA

Done at the School Administrator level

Step 1: Create Competency Category

Modify Competency Category

Add Sub-Competency Levels (if applicable)

Sort Competencies

Sort Child Competencies

POPULATE COMPETENCY ASSESSMENT FORM DATA

Done at the Course Admin level

Step 2: Create Assessment Form(s)

Note: The “View Reports” button is only visible to the course director “Balance Weights” still needs to be further discussed/clarified

Populate Assessment Form Metadata

Grading is numeric. There will be an option in the grade book to give an equivalent letter or pass/fail grade

Scoring Range defaults to ‘No’

Score Display:• Score & Feedback• Feedback Only

Scoring Value:• Max value = 25

Frequency:• Max No. = 10

Step 3: Populate Fields and Data

Question type: Scaling Scaling w/Sub-question Single Select Single Select w/Sub-question

Multi-level questions (options with sub-questions)

Link competencies to questions Free-text feedback option

Add Field: Scaling

Make text boxes bigger

Linking Competencies

Scaling w/ Sub-Questions

Add Field: Single Select

Single Select w/ Sub-level Questions

OTHER COMPETENCY ASSESSMENT FORM DATA

Done at the Course Admin level

Considerations

Display student images? Display elective information? Display summary comments Multiple assessors? Who selects assessors?

Considerations?

Use Numeric scores? Display scores to students or just

comments? Display scores to assessors? 360 degress review? What else?

Link Faculty/Student

II. Faculty View

Getting to the form

Accessing the Assessment(s)

Accessing the FormEvaluator: J. Berg

Assessment Form – Scaling

Assessment Form – Single Select

III. Director View

Viewing assessment results

Assessment Results – All Students

Assessment Results – Individual Student

Note: To view a form completed by a faculty/staff click their name

Note: When ready, click here to fill out a final assessment form for the student

Completed Form by Faculty/Staff

Summary Form

Hover over score to see actual individual

scores

This is at the bottom of the form

Click to view student’s patent logs in a pop-up window

Student’s Patient Log Summary

This opens up as a pop-up window

Completion Report

Send Email Reminders

IV. Student View

Access via the Grade Book

Accessing Assessment Form

Note: Link to view final assessment form completed by the course director

Accessing Assessment Form

J. Berg happens to also be the course director

V. Registrar View

View Final Grades

Grades by Course

Step 1: Select the course

Grades by Course

Step 2: Select the time period

Part 3

Building the Framework

First steps

A set of clear definitions Use cases A survey of Canadian, US and UK

health professional education organizations (24 respondents)

See definitions_and_usecases.pdf for details (medbiquitous.org)

Comparison of published frameworks

Accreditation Council on Graduate Medical Education (ACGME) Competencies8

Acute Care Nurse Practitioner Competencies American Association of Critical-Care Nurses (AACN) General

Patient Care Competencies American Society of Health-System Pharmacists CanMEDS 20059

CanMEDS Specialty Competencies Good Medical Practice (UK) Good Medical Practice (USA) The Scottish Doctor7 Tomorrow's Doctors Tuning (MEDINE) Women’s Healthcare Competencies

DEVELOPMENT OF A CONCEPTUAL MODEL

Outcomes for Clinical Skills

A competency object . . .

Relates to other competency objects . . .

Which can relate to external resources . . . Learning

ObjectAssess-ment

Performance data

Outcomes for Clinical Skills

A competency object . . .

Relates to other competency objects . . .

Which can relate to external resources . . . Learning

ObjectAssess-ment

Performance data

Current Specifications

Specs and Schemas Available At:

http://www.medbiq.org/working_groups/competencies/index.html

Open license, membership not required

Competencies – individual competency definition

A standard format for a single competency definition, including an identifier, categories and references.

Competency Object

Working draftBeing implemented by TuftsMay be implemented by VA

Essential for tying curricula to competencies and enabling competency based education.

Competencies – an interrelated set of competencies

A framework for establishing the relationships among competencies in a single framework

Competency Framework

Working draftBeing implemented by TuftsMay be implemented by VA

Essential for integrating existing competency sets into curriculum management systems.

Competencies – evidence of achievement

A format for documenting achievement of a milestone or competency.

Educational Achievement

Work to begin in July 2010

This will be important for schools tracking competence in a portfolio and for residency programs tracking competence.

Competencies – cross-mapping

A format for mapping competencies from one framework to competencies from another framework.

Informal proposal from competency working group (planned work)

Should we arrive at the point of having a national competency framework, this would be important for schools that want to see how local competencies fit into the national picture.

Competencies – mapping to external resources

A way of mapping competencies to learning activities and resources, curricula, and potentially performance data.

Informal proposal from competency working group (planned work)

Could be important for the curriculum inventory portal should it be competency based.

Spec needs Analysis of the Problem from all perspectives Student View

How am I doing? What do I have to do? How can I show my learning

Faculty view How is student doing Grade/Rate/Comment on student View proof of learning See full range of expected learning outcomes

Registrar Track grades

Admin View What are the competencies Where are competencies taught -

content/courses/themes/rotations/clinics Where are the gaps

Need tools to Enter, Store, view, Grade

Medbiquitous Competency Object Standard

Copyright Medbiquitous consortium 2010 All rights reserved May 20 2010 p.11

What Info Needs to be brought Together

School wide competencies Course level competencies/learning

objectives Content addressing competencies

and learning objectives Element of time – when and how

much How are competencies assessed

Where does TUSK get the Data

Schedule Content linked to schedule (upon

upload) Curriculum hours pulled from

schedule Keywords linked to content

School WideCompetencies:Competency Framework

School WideCompetencies:Competency Framework

Course competenci

es

Course competenci

esCourse competenci

es

Course competenci

es

Content repository

w/metadata

Content repository

w/metadata

Contentcompetenc

ies

Contentcompetenc

ies

Competency

Assessment

Competency

Assessment

skillsskills knowledge

knowledge attitudesattitudes

Where are they taught ?

How are they taught?

Where are they assessed?

ExternalCompetency Framework

ExternalCompetency Framework

lectures

lectures

PBLPBL

rotations

rotations

When?(Links to schedule)

Competencies

Schedule – Curriculum View

First year courses

Course Summary view

Methods to Display Complex Datacomplex data displays

References Medical Teacher: Outcomes-Based Education Themed Issue

Vol 29 Number 7 September 2007 Lessons Learned from the Competency-Based Curriculum

Initiative at Baylor College of Dentistry. McCann, Ann L.; Babler, William J.; Cohen, Peter A., Journal of Dental Education, v62 n2 p197-207 Feb 1998

The new formal competency based curriculum at Indiana University School of Medicine, Litzleman and Cottingham, Academic Medicine Vol 82, No. 4 April 2007

An educational blueprint for the Brown Medical School, Smith MD

Learning objectives for medical student education--guidelines for medical schools: report I of the Medical School Objectives Project. AAMC Acad Med. 1999 Jan;74(1):13-8.