SC04_Kern_David_201191

7/30/2019 SC04_Kern_David_201191

1/41

32ndSGIM Annual Meeting May 13, 2009

The Six-Step Approach to Curriculum DevelopmentDavid E. Kern, MD, MPH

Patricia A. Thomas, MDMark T. Hughes, MD, MA

L. Randol Barker, MD

Eric B. Bass, MD, MPH

Joseph A. Carrese, MD, MPHLeah Wolfe, MD

LEARNING OBJECTIVESBy the end of this precourse, participants will be able to:

A. Describe the six-steps of curriculum development:1. Problem identification and general needs assessment2. Targeted needs assessment3. Goals and objectives

4. Educational Strategies5. Implementation6. Evaluation and Feedback

B. Apply the steps in curricular planning exercisesC. Identify additional resources that are available to help them improve their curriculum

development skills.

AGENDA

1:30-2:15 Lecture:

Step 1. Problem Identification and General Needs Assessment

Step 2. Targeted Needs AssessmentStep 3. Goals & Objectives

2:15-3:00 Facilitated Small Group Exercise #1: Applying Steps 1 to 3 to a curricular project.

3:00-3:15 Break

3:15-4:00 Lecture:

Step 4. Educational StrategiesStep 5. Implementation

Step 6. Evaluation

4:00-4:45 Facilitated Small Group Exercise #2: Applying Steps 4 to 6 to a curricular project.

4:45-4:55 Summary

DisseminationAdditional Resources

4:55-5:00 Evaluation

7/30/2019 SC04_Kern_David_201191

2/41


SMALL GROUP EXERCISE #1

Step 1: Problem Identification and General Needs Assessment

Step 2: Targeted Needs Assessment

Step 3: Goals and Objectives

1. Based on your knowledge of the literature and your targeted learners, build a short logicalargument for the curriculum you are developing, moving from problem identification to the

gap between the current versus ideal approaches of addressing the problem, to specific

information you have about your targeted learners and targeted learning environment. Thelogical argument should lead to the objective you propose in the second question. It could

also serve as the introduction to a paper on your curriculum. (See Tables 2.1, 2.2, and 3.1.)

2. Please write one cognitive (knowledge), affective (attitude), or psychomotor (skill or

performance) objective for your curriculum, whichever is most relevant. Remember: Who

will do how much of what by when? (See Tables 4.1-4.3.) (Preview: This objective willdetermine your educational and evaluation methods, which you will detail in the next small

group exercise.)

7/30/2019 SC04_Kern_David_201191

3/41


Table 2.1 Identification and Characterization of the Health Care Problem

Whom does it affect?Patients

Health care professionalsMedical educatorsSociety

What does it affect?Clinical outcomesQuality of lifeQuality of health careUse of health care and other resourcesMedical and non-medical costsPatient and provider satisfactionWork and productivitySocietal function

What is the quantitative and qualitative importance of the effects?

Table 2.2 The General Needs Assessment

What is currently being done by the following?PatientsHealth care professionals

Medical educatorsSociety

What personal and environmental factors impact upon the problem?PredisposingEnablingReinforcing

What ideally should be done by the following?PatientsHealth care professionalsMedical educatorsSociety

What are the key differences between the current and ideal approaches?

Tables from: Kern DE, Thomas PA, Hughes MT, eds. Curriculum Development for Medical Education: ASix-Step Approach. 2

nded. Baltimore (MD): Johns Hopkins University Press; 2009. In Press.

7/30/2019 SC04_Kern_David_201191

4/41


Table 3.1: Content Potentially Relevant to a Needs Assessment of Targeted Learners

Content about Targeted LearnersPrevious training and experiences relevant to the curriculum

Already planned training and experiences relevant to the curriculumExisting characteristics / proficiencies / practices:

Cognitive: knowledge, problem-solving abilitiesAffective: attitudes, values, beliefs, role expectationsPsychomotor: skills / capabilities (e.g. history, physical examination, procedures, counseling);

current behaviors / performance / practicesPerceived deficiencies and learning needsPreferences and experiences regarding different learning strategies

Synchronous (educator sets time, such as with noon lecture)Asynchronous (learner decides on learning time, such as with computer learning)Duration (amount of time learner thinks is needed to learn or that they can devote to learning)Methods (e.g. lectures, readings, web-based computer interactive modules, case-based

discussions, group learning, role plays / simulations, supervised experience)

Content about Learning EnvironmentRelated existing curriculaNeeds of stakeholders other than the learners (course directors, clerkship directors, residency program

directors, accrediting bodies, others)Barriers, enabling, and reinforcing factors that affect learning by the targeted learners

The informal and hidden curriculumIncentivesResources

Patients and Clinical Experiences

FacultyInformation ResourcesComputers

Audio-visual EquipmentRole Models, Teachers, MentorsOther

Table from: Kern DE, Thomas PA, Hughes MT, eds. Curriculum Development for Medical Education: ASix-Step Approach. 2


7/30/2019 SC04_Kern_David_201191

5/41


Table 4.1. Verbs Open to More and Fewer Interpretations

Verbs Open to More Interpretations Verbs Open to Fewer Interpretations

Verbs that frequently apply to Cognitive Objectives:

Taxonomy of cognitive

objectives (2,3) Verb

knowRemember(recall offacts)

identifylistrecitedefinerecognizeretrieve

understand Understand definecontrastinterpretclassifydescribesortexplainillustrate

Apply Implementexecuteuse (a model, method)complete

Analyze differentiatedistinguishorganizedeconstruct

discriminate

be ableknow howappreciate

Evaluate detectjudgecritiquetest

know how Create designhypothesizeconstructproduce

Verbs that frequently apply to Affective Objectives:

appreciategrasp the significance of

rate as valuable,rank as important

believe identify, rate, or rank as a belief or opinion

enjoy rate or rank as enjoyable

internalize use one of above terms

Verbs that frequently apply to Psychomotor Objectives:

7/30/2019 SC04_Kern_David_201191

6/41


Skill / Competence:be ableknow how

demonstrateshow

Behavior / PerformanceInternalize use or incorporate into performance (as measured

by)

Other Verbs:learn (use one of the above terms)

teach (use one of the above terms; do not confuse theteacher and the learner in writing learner objectives)

Table from: Kern DE, Thomas PA, Hughes MT, eds. Curriculum Development for Medical Education: A Six-Step

Approach. 2nded. Baltimore (MD): Johns Hopkins University Press; 2009. In Press.

7/30/2019 SC04_Kern_David_201191

7/41


Table 4.2. Examples of Less Well Written and Better Written Objectives

Less Well Written Objectives Better Written Objectives

Residents will learn the techniques of joint

injections. [The types of injections to belearned are not specified. The types ofresidents are not specified. It is unclearwhether cognitive understanding of thetechnique is sufficient, or whether skills mustbe acquired. It is unclear by when the learningmust have occurred, and how proficiencycould be assessed. The objective on the rightaddresses each of these concerns.]

By the end of their residency, each family

practice resident will have demonstrated atleast once (according to the attached protocol)the proper techniques of:- subacromial, bicipital, and intra-articular

shoulder injection;- intra-articular knee aspiration and/or injection;- injections for lateral and medial epicondylitis;- injections for deQuervain's tenosynovitis; and- aspiration and/or injection of at least one new

bursa, joint, or tendinous area, usingappropriate references and supervision.

By the end of their internal medicine clerkship,each 3rd year medical student will be able to

diagnose and manage common ambulatorymedical disorders. [This objective specifies"who" and "by when", but is vague about whatit is the medical students are to achieve. Thetwo objectives on the right add specificity tothe latter.]

By the end of their internal medicine ambulatorymedicine clerkship, each 3rd year medical

student will have achieved cognitive proficiencyin the diagnosis and management ofhypertension, diabetes, angina, chronicobstructive pulmonary disease, hyperlipidemia,alcohol and drug abuse, smoking, andasymptomatic HIV infection, as measured byacceptable scores on interim tests and the finalexamination.

By the end of their internal medicine clerkship,each 3rd year medical student will have seenand discussed with their preceptor, ordiscussed in a case conference with

colleagues, at least one patient with each of theabove disorders.

Physician practices, whose staff complete the3-session communications skills workshops,will have more satisfied patients. [Thisobjective does not specify the comparisongroup or what is meant by "satisfied". Theobjective on the right specifies more preciselywhich practices will have more satisfiedpatients, what the comparison group will be,and how satisfaction will be measured. Itspecifies one aspect of performance as well as

satisfaction. One could look at the satisfactionquestionnaire and telephone managementmonitoring instrument for a more precisedescription of the outcomes being measured.]

Physician practices, which have 50% of theirstaff complete the 3 session communicationsskills workshops, will have lower complaintrates, higher patient satisfaction scores on theyearly questionnaire, and better telephonemanagement as measured by randomsimulated calls, than practices that have lowercompletion rates.

Table from: Kern DE, Thomas PA, Hughes MT, eds. Curriculum Development for Medical Education: A Six-StepApproach. 2nded. Baltimore (MD): Johns Hopkins University Press; 2009. In Press.

7/30/2019 SC04_Kern_David_201191

8/41


Table 4.3. Types of Objectives. Examples from a Smoking Cessation Curriculum for Residents

INDIVIDUAL LEARNER AGGREGATE OR PROGRAM

LEARNER

Cognitive(Knowledge)

By the end of the curriculum, eachresident will be able to list the 5-step approach to effective smokingcessation counseling.

By the end of the curriculum, 80%of residents will be able to list the5-step approach to effectivesmoking cessation counseling,and 90% will be able to list the 4critical (asterisked) steps.

Affective(Attitudinal)

- - - - - - - - - - - - - - - - - - - - - - -By the end of the curriculum, each

primary care resident will ranksmoking cessation counseling asan important and effective

intervention by primary carephysicians (3 on a 4-point scale).

- - - - - - - - - - - - - - - - - - - - - - -By the end of the curriculum there

will have been a statisticallysignificant increase in how primarycare residents rate the importance

and effectiveness of smokingcessation counseling by primarycare physicians.

Psychomotor(Skill or Competence)

- - - - - - - - - - - - - - - - - - - - - - -During the curriculum, each primary

care resident will demonstrate inrole play a smoking cessationcounseling technique thatincorporates the attached 5-steps.

- - - - - - - - - - - - - - - - - - - - - - -During the curriculum, 80% of

residents will have demonstratedin role play a smoking cessationcounseling technique thatincorporates the attached 5-steps.

Psychomotor(Behavioral orPerformance)

- - - - - - - - - - - - - - - - - - - - - - - -By 6 months after completion of the

curriculum, each primary careresident will have negotiated aplan for smoking cessation with60% of his/her smoking patients,or have increased the percentageof such patients by 20% frombaseline.

- - - - - - - - - - - - - - - - - - - - - - -By 6 months after completion of the

curriculum, there will have been astatistically significant increase inthe % of GIM residents who havenegotiated a plan for smokingcessation with their patients.

PROCESSEach primary resident will have

attended both sessions of thesmoking cessation workshop.

80% of primary care residents willhave attended both sessions ofthe smoking cessation workshop.

PATIENT OUTCOME

By 12 months after completion of thecurriculum, the smoking cessationrate (for6 months) for thepatients of each primary careresident will increase 2-fold frombaseline or be 10%.

By 12 months after completion of thecurriculum, there will have been astatistically significant increase inthe % of primary care residents'patients who have quit smoking(for6 months).



7/30/2019 SC04_Kern_David_201191

9/41



Step 4: Educational Strategies

Step 5: Implementation

Step 6: Evaluation

1. Choose 2 educational strategies that you could use in your curriculum that would helplearners achieve the learner objective that you identified. Are the strategies congruent with

your objective? (See Tables 5.2-5.3.)

2. Identify one or two evaluation methods that you could use for assessing learner achievementof your learner. (See Table 7.3.) Is (are) your evaluation method(s) congruent with your

educational objective and methods?

3. Are your proposed educational strategies and evaluation plans feasible in terms of available

resources (personnel, equipment, space, time, funding)? (See Table 6.1)

7/30/2019 SC04_Kern_David_201191

10/41


Table 5.2 Matching Educational Methods to Objectives*

Type of Objective

Educational Method

Cognitive:Knowledge

Cognitive:Problem-Solving

Affective:Attitudinal

Psychomotor:Skills or

Competence

Psychomotor:Behavioral orPerformance

Readings +++ + + +

Lectures +++ + + +

Programmed learning +++ ++ +

Discussion ++ ++ +++ + +

Reflection on experience +++ +++ +++

Feedback on performance + ++ ++ +++ +++

Small-group learning ++ ++ ++ + +

Problem-based learning ++ +++ + +

Team-based learning +++ +++ ++ + +

Learning projects +++ +++ + + +

Role models + ++ + ++

Demonstration + + + ++ ++

Role plays + + ++ +++ +

Artificial models andsimulation

+ ++ ++ +++ +

Standardized patients + ++ ++ +++ +

Real life experiences + ++ ++ +++ +++

Audio or video review oflearner

+ +++ +

Behavioral /environmentalinterventions**

+ + +++

*blank = not recommended; + = Appropriate in some cases, usually as an adjunct to other methods; ++ =good match; +++ = excellent match (consensus ratings by author and editors).

** = Removal of barriers to performance; provision of resources that promote performance;reinforcements that promote performance.



7/30/2019 SC04_Kern_David_201191

11/41


TABLE 5.3 Summary of Advantages & Limitations of Different Educational Methods

Educational Method Advantages Disadvantages

Readings Low costCover fund of knowledge

Little preparation time

Passive learningLearners must be motivated to

complete

Lectures Low costAccommodate large numbers of

learnersStructured presentation of

complicated topics

Passive learningTeacher-centeredQuality depends on

speaker/audiovisual material

Programmed Learning Active learningDon't need clinical material at handSafe simulations for learnersImmediate feedback on knowledge,

sequencing, efficiency, clinicaldecision-making

Learner applies new knowledge

Developmental costs if notcommercially available

Discussion Active learningPermits assessment of learner needs

Allows learner to apply newly acquiredknowledge

Suitable for higher order cognitiveobjectives: problem-solving andclinical decision-making; canaddress affective objectives

Exposes students to differentperspectives

More faculty intensive than readingsor lectures

Cognitive/experience base required oflearners

Group dependentUsually facilitator dependent

Reflection on experience Promotes learning from experiencePromotes self-awareness /

mindfulnessCan be built into discussion / group

learning activitiesCan be done individually through

assigned writings / portfolios

Requires protected timeUsually requires scheduled interaction

time with another / others.Often facilitator dependent

Feedback on performance Promotes learning from experienceCan be used with role play, artificial

models / simulation, standardizedpatients, clinical experience, andaudio/video review

Requires observer who is a skilledprovider of feedback

Small group learning Active learningResources usually available

Allows multidisciplinary approachesSuitable for team-based and problem-

based learning, clinical decision-making, community-based projects

Encourages cooperation, team-workamong learners.

Incorporates discussion.

Group should have some training ingroup process skills, conflictmanagement, etc.

May require faculty facilitators withtraining in above

Time required for successfulfunctioning

7/30/2019 SC04_Kern_David_201191

12/41


Problem-based leaning(PBL)

Active learningFacilitates higher cognitive objectives:

problem-solving and clinicaldecision-making; can incorporateobjectives which cross domains:ethics, humanism, cost-efficiency

Developmental costsRequires faculty facilitators and small

groupsLess efficient for transferring factual

information

Team-based learning(TBL)

Active learningFacilitates higher cognitive objectives;

constructive knowledgeStudents take responsibility for

learningCollaborativeUses less faculty than PBL / other

small group learning methods

Developmental costs for ReadinessAssurance Test (RATs) andapplication exercises

Students need to be self-directedRequires orientation of students to

process of team work and peerevaluation

Learning projects Active learningPromote, teach self-directed learningLearners sets individual learning

objectives

Suitable for higher-order cognitiveobjectives

Learners need motivationLearners need basic skills to access

and optimally use learningresources

Requires effective faculty mentor

Role models Faculty often availableImpact often seems profound

Require valid evaluation process toidentify effective role models

Specific interventions usually unclearImpact depends on interaction

between specific faculty memberand learner

Outcomes multifactorial and difficult toassess

Demonstration Efficient method for demonstrating

skills/procedures

Passive learning

Teacher-centeredQuality depends on

teacher/audiovisual material

Role plays Suitable for objectives which crossdomains: knowledge, attitudes, andskill

EfficientLow costCan be structured to be learner-

centeredSafe environment for skills practice

Require trained faculty facilitatorsLearners need some basic knowledge

or skillsCan be resource intensive if large

numbers of learners

Artificial models andsimulation

Safe environments to practice skillsLearners can use at own pace; less

faculty supervision required

May not be available for specificcurriculum

Can be expensive

Standardized patients Ensure appropriate clinical materialApproximate "real life" more closely

than role playsSafe environment for skills practiceCan give feedback to learners on

performanceCan be reused for ongoing curricula

CostExpertise required to develop and

train standardized patients

7/30/2019 SC04_Kern_David_201191

13/41


Clinical experience "Real life"Promotes learner motivation and

responsibilityPromotes higher level cognitive,

attitudinal, skill, and performancelearning

Requires clinical material whenlearner is ready

Requires faculty to supervise and toprovide feedback

Learner needs basic knowledge orskill

Needs to be monitored for case mix,appropriateness

Requires reflection, follow-up

Audio or video reviews oflearner

Provides accurate feedback onperformance

Provides opportunity for self-observation

Requires trained faculty/facilitatorsRecording can be awkward or

intrusive, and pose logistic problemsRequires patient permission

Behavioral /environmentalinterventions*

Influence performance Assume competenceRequire control over learners real-life

environment

* Removal of barriers to performance; provision of resources that promote performance; reinforcements

that promote performance.



7/30/2019 SC04_Kern_David_201191

14/41


Table 7.3. Uses, Strengths, and Limi tations of Commonly Used Evaluation Methods

METHOD USES STRENGTHS LIMITATIONSGlobal Rating Forms(separated in timefrom observation)

Cognitive, affective, orpsychomotor attributes;real life performance

EconomicalCan evaluate anythingOpen ended questions can

provide information forformative purposes

SubjectiveRater biasesInter and intra-rater

reliabilityRaters frequently have

insufficient data uponwhich to base ratings

Self-assessmentforms

Cognitive, affective,psychomotor attributes;real life performance

EconomicalCan evaluate anythingPromotes self-assessmentUseful for formative

evaluation

SubjectiveRater biases

Agreement with objectivemeasurements often low

Limited acceptance asmethod of summativeevaluation

Essays onRespondent'sExperience

Attitudes, feelings,description of respondentexperiences, perceivedimpact

Rich in textureProvides unanticipated as

well as anticipatedinformation

Respondent-centered

SubjectiveRater biasesRequires qualitative

evaluation methods toanalyze

Focus varies fromrespondent to respondent

Written or computer-interactive tests

Knowledge; higher levelcognitive ability

Often economicalObjectiveMultiple choice exams can

achieve high internalconsistency reliability,

broad samplingGood psychometric

properties, low cost, lowfaculty time, easy toscore

Widely AcceptedEssay type questions or

computer-interactive testscan assess higher levelcognitive ability,encourage students tointegrate knowledge,reflect problem solving

Constructing tests of higherlevel cognitive ability, orcomputer-interactivetests, can be resourceintensive

Reliability and validity varywith quality of test (e.g.questions that are notcarefully constructed canbe interpreted differentlyby different respondents,there may be aninsufficient number ofquestions to validly test adomain)

7/30/2019 SC04_Kern_David_201191

15/41


Oral Examinations Knowledge; higher levelcognitive ability; indirectmeasure of affectiveattributes

Flexible, can follow-up andexplore understanding

Learner-centeredCan be integrated into case

discussions

Subjective scoringInter and intra-rater

reliabilityReliability and validity vary

with quality of test (e.g.questions that are notcarefully constructed canbe interpreted differentlyby different respondents,there may be aninsufficient number ofquestions to validly test adomain)

Faculty intensiveCan be costly

Questionnaires Attitudes; perceptions;suggestions forimprovement

Economical SubjectiveConstructing reliable and

valid measures ofattitudes requires time

and skill

Individual Interviews Attitudes; perceptions;suggestions forimprovement

Flexible, can follow-up andclarify responsesRespondent-centered

SubjectiveRater biasesConstructing reliable and

valid measures ofattitudes requires timeand skill

Requires interviewers

Group Interviews /Discussions

Attitudes; perceptions;suggestions forimprovement

Flexible, can follow-up anddevelop / exploreresponses

Respondent-centeredEfficient means of

interviewing several atonce

Group interaction canenrich or deepeninformation

Can be integrated intoteaching sessions

SubjectiveRequires skilled interviewer

or facilitator to control

group interaction andminimize facilitatorinfluence on responses

Does not yield quantitativeinformation

Information may not berepresentative of allparticipants

7/30/2019 SC04_Kern_David_201191

16/41


Direct Observation Skills; performance First hand dataCan provide immediate

feedback to observedDevelopment of standards,

use of observation checklists, and training ofobservers can increasereliability and validity.The Objective StructuredClinical Examination(OSCE) (62,63) andObjective Structured

Assessment of TechnicalSkills (OSATS) (64-66)combine directobservation withstructured checklists toincrease reliability andvalidity.

Rater biasesInter and intra-rater

reliabilityPersonnel intensiveUnless observation covert,

assesses capabilityrather than real-lifeperformance

Performance Audits Record keeping; provisionof recorded care (e.g.tests ordered, provisionof preventive caremeasures, prescribedtreatments)

ObjectiveReliability and accuracy

can be measured andenhanced by the use ofstandards and thetraining of raters

Unobtrusive

Dependent on what isreliably recorded; muchcare is not documented

Dependent on available,organized records or datasources



7/30/2019 SC04_Kern_David_201191

17/41


Table 6.1: Checklis t For Implementation

____ Identify resources____ Personnel: faculty, AV, computing, secretarial and other support staff, patients____ Time: faculty, support staff, learners____ Facilities: space, equipment, clinical sites, virtual space (servers, content management

software)____ Funding/costs: direct financial costs, hidden or opportunity costs

____ Obtain support____ Internal

from: those with administrative authority (dean's office, hospital administration,department chair, program director, division director, etc.), faculty,learners, other stakeholders

for: personnel, resources, political support____ External

from: government, professional societies, philanthropic organizations or foundations,accreditation bodies, other entities (e.g. managed care organizations), individualdonors

for: funding, political support, external requirements, curricular or faculty

development resources

____ Develop administrative mechanisms to support the curri culum____ Administrative structure: to delineate responsibilities and decision-making____ Communication

content: rationale; goals and objectives; information about the curriculum, learners,faculty, facilities & equipment, scheduling; changes in the curriculum; evaluationresults; etc.

mechanisms: websites, memos, meetings, syllabus materials, site visits, reports, etc.____ Operations: preparation and distribution of schedules and curricular materials;

collection, collation and distribution of evaluation data; curricular revisions andchanges, etc.

____ Scholarship: plans for presenting and publishing about curriculum;

human subjects protection considerations; IRB approval, if necessary____ Anticipate and address barriers

____ Financial & Other Resources____ Competing Demands____ People: attitudes, job/role security, power & authority, etc.

____ Plan to introduce the curricu lum____ Pilot____ Phase-in____ Full implementation

Table from: Kern DE, Thomas PA, Hughes MT, eds. Curriculum Development for Medical Education: ASix-Step Approach. 2nd

ed. Baltimore (MD): Johns Hopkins University Press; 2009. In Press.

7/30/2019 SC04_Kern_David_201191

18/41

1

32nd SGIM Annual Meeting

THE 6-STEP APPROACHTO CURRICULUM DEVELOPMENT

David E. Kern, MD, MPH

Patricia A. Thomas, MD

Mark T. Hughes, MD

L. Randol Barker MD, ScM

Eric B. Bass, MD, MPH

Joseph A. Carrese, MD, MPH

May 13, 2009

No conflicts of interest

to disclose.

Goals of Workshop

By the end of the workshop, participants willdemonstrate their ability to

aDescribe the principles of curriculumdevelopment

aApply these principles to enhance their work ineducational program development

aApply these principles to benefit learners,society, and their own career advancement aseducational scholars

CURRICULUM: DEFINITION

aA planned educational experience

CURRICULUM DEVELOPMENT:

UNDERLYING ASSUMPTIONS

aEducational programs have goals or aims,whether articulated or not

aMedical educators have professional and ethical

obligations to meet the needs of their learners,patients and society

aMedical educators should be held accountablefor the outcomes of their interventions

aA logical, systematic approach to CD will helpachieve these goals

Rudolf Virchow

Medical instruction does not exist toprovide individuals with an opportunity

of learning how to make a living, but inorder to make possible the protection ofthe health of the public.

7/30/2019 SC04_Kern_David_201191

19/41

2

CURRICULUM DEVELOPMENT

PRECOURSE: OVERVIEW

a6 Steps

RRC REQUIREMENTS FOR A

Written Curriculum (U.S.)

For each rotation or major learning experience,must include:

aEducational purpose (Goals & Objectives)

aTeaching methods (Educational strategies)

aMethod of evaluation of resident performance(Evaluation)

LCME (U.S.)

aSimilar requirements

ACGME OUTCOME PROJECT (U.S.)(http://www.acgme.org/Outcome)

a6 Competencies1. Patient Care

2. Medical Knowledge

3. Practice-Based Learning & Improvement

4. Interpersonal and Communication Skills5. Professionalism

6. Systems-Based Practice

aFocus on:

objectives

evaluation

continuous improvement

Glassick*

Criteria for Scholarship

1. Clear goals and aims

2. Adequate preparation

3. Appropriate methods

4. Significant results

5. Effective dissemination

6. Reflective critique

*Scholarship Assessed:Evaluation of the Professoriate,1997

7/30/2019 SC04_Kern_David_201191

20/41

3

STEP 1:PROBLEM IDENTIFICATION

AND

GENERAL NEEDS ASSESSMENT

building the foundation for

meaningful objectives

Glassick Criteria

1. Clear goals and aims2. Adequate preparation

3. Appropriate Methods

4. Significant Results

5. Effective Dissemination

6. Reflective Critique

PROBLEM IDENTIFICATION &

GENERAL NEEDS ASSESSMENT: WHY?

aBuilds a rationale for your curriculum

aFocuses a curriculums goals andobjectives

aWhich in turn focus the educationaland evaluation strategies

aMakes you an expert and a scholar

STEP 1:

PROBLEM IDENTIFICATION

aIdentify and Characterize the Health CareProblem That Will Be Addressed by theCurriculum

Whom Does the Problem

Affect?

aPatients

aSociety

aHealth Care Professionals

aTrainees

What Does the Problem Affect?

aClinical Outcomes

aQuality of Life

aQuality of Health Care

aUse of Health Care and Other Resources

aMedical and Non-medical Costs

aPatient and Provider Satisfaction

aWork and Productivity

aSocietal Function

7/30/2019 SC04_Kern_David_201191

21/41

4

Example: Problem Identification

We need a curriculum in communication skillsfor our residents becomes: Why is itimportant for residents to be effectivecommunicators?`What is the impact on the process of care?

`What is the impact on clinical outcomes?

`What is the impact on malpractice?

`What is the impact on utilization and costs?

`What is the impact on patient and physiciansatisfaction?

Example: Problem Identification

We need a curriculum in professionalism for ourresidents becomes: What is professionalism,and why is it important for our residents tobehave professionally?

`What are the critical, evidenced-based or agreed uponcomponents of professionalism?

`How do physicians professional behavior impact onpatients (satisfaction, clinical care outcomes),functioning of the health care team, and society?


aWhat is currently being done aboutthe problem?

`By patients?

`By practitioners?

`By medical educators?

`By society at large


aWhat is the ideal approach tothe problem?

`By patients?

`By practitioners?

`By medical educators?

`By society at large


General Needs Assessment =

Ideal Approach - Current Approach

Example: Communication Skills(Kern DE et.al. Re sidency training in interviewing skills and the psychosocial domain of medicalpractice. J Gen Intern Med 1989; 4:421-431 .)

a CS critical to diagnosis, patient education, trust, patientsatisfaction, clinical decision-making

a CS related to patient outcomes: satisfaction, compliance,diabetes control, malpractice

a Physicians are hypo-competent

a Physician education often ignored or deficient at medicalstudent and resident level

a Examples of effective education exist

a Effective education uses: effective educationalmethodologies which includes 2 experiential methods,same specialty role models, and reinforcement

7/30/2019 SC04_Kern_David_201191

22/41

5

Example: Professionalism

a Generally agreed upon components of professionalism include:

altruism, respect, cross-cultural sensitivity, accountability,confidentiality, communication and shared-decision making,integrity, compassion / empathy, duty, competence, recognizingand managing conflicts of interest, self-awareness, andcommitment to excellence and ongoing professional development

a Frequent lapses in professionalism have been documented inresidents, faculty, and practicing physicians. There is someevidence of erosion of professionalism during training.

a Explicit teaching and assessment of professionalism is uncommonin medical schools and residency programs, although exceptionsexist.

a Patient and societal trust of physicians is related to theirperceptions and experiences of physicians professionalism.

PI AND GNA:

OBTAINING THE INFORMATION

aReview of Available Information`Published Literature: PubMed, ERIC

Accreditation Bodies: AAMC, LCME, ACGME

`Professional Societies: IOM

`BEME

`Government Databases and Reports

aUse of Consultants / Experts

aCollection of New Information

Example: Musculoskeletal Medicine(Houston TK et. al. A primary care musculoskeletal clinic for residents: success and sustainability. J GenIntern Med 2004;19:524-529.

aMS disorders common and major cause of disability

a Patients desire quick access

a Training increases PMD confidence and referrals

a FCIM, ACGME, COGME recommend training in physicalexamination, diagnosis, and management of commonMS disorders (including joint aspiration and injection,when appropriate)

a Preferred training modality is supervised clinical practicein settings similar to those in which the trainees willeventually practice, i.e. in primary care settings withtrained preceptors possessing the desired expertise.

Example: Resident Teaching Skills(Morrison EH et.al. The effect of a 13-hour curriculum to improve residents teaching skills: a randomizedtrial. Ann Intern Med 2004;141:257-263.

aRecognizing the crucial roles that resident physicianteachers fulfill in medical education, the LCME (1) andother professional bodies (2) are calling upon residencytraining programs to ensure residents competence asclinical teachers.

aMore residency programs today are offering teachingskills training to their housestaff (3,4), but the evidencefor how this training should be accomplished is limited. .. .

Example: Smoking Cessation (1)(Cornuz J et.al. Efficacy of resident training in smoking cessation: a randomizedcontrol trial.. Ann Intern Med. 2002;136:429437.

aMedical advice and pharmacotherpay are effectiveinterventions in clinical practice to help patients stopsmoking (1,2).

aAlthough primary care physicians can play a key role in

promoting smoking cessation to their patients whosmoke (3), they miss many opportunities to advisesmokers (4-7), mainly because they lack skills incounseling about smoking cessation (8).

a Residency training in ambulatory care is an ideal settingin which to learn the attitudes and skills of preventivemedicine, including smoking cessation (9).

Example: Smoking Cessation (2)(Cornuz J et.al. Efficacy of resident training in smoking cessation: a randomizedcontrol trial.. Ann Intern Med. 2002;136:429437.

aTraining programs in smoking cessation improve thefrequency and quality of smoking cessation interventionsadministered by physicians (10-17). . .

aMost training programs mainly use didactic teachingrather than such potentially effective methods as activelearning of practical skills (20,21). .

aA few training programs based on active learningmethods effectively improve counseling skills, self-efficacy, and attitudes (22-25), but their effect on ratesof smoking cessation remains unknown. . .

7/30/2019 SC04_Kern_David_201191

23/41

6

STEP 2:

TARGETED NEEDS ASSESSMENT

refining the foundation

TARGETED NEEDS ASSESSMENT:

DEFINITION

aA needs assessment of ones

`Targeted learners

`Targeted learning environment

whose needs may differ from the needsof learners and learning environments ingeneral.

IMPORTANCE

aIdentifies the specific needs and preferences oftargeted learners and other stakeholders, whichmay be different from learners and stakeholdersin general.

aAssesses the environment (including the hiddenand informal curriculum) which will likelyinfluence behavioral / performance outcomes.

aPermits tailoring the educational intervention tospecific needs.

aIncreases efficiency, prevents duplication.

aBuilds relationship with stakeholders.

aAligns resources with strategy

TARGETED NEEDS ASSESSMENT:

THREE STEPS

1. Choose your targeted learners.

2. Find out information about thetargeted learners.

3. Determine characteristics of thelearning environment.

INFORMATION ABOUT

TARGETED LEARNERS

aPrevious training & experience

aAlready planned training & experience

aExisting proficiencies: knowledge / attitudes/ skills

aCurrent performance / behaviors

aPerceived deficiencies and learning needs

aPreferences

INFORMATION ABOUT

TARGETED ENVIRONMENT

aRelated existing curricula

aHidden / informal curriculum

aSpecific enabling and reinforcing factors /barriers

aResources

aStakeholders

aPolitics / factors related to institutionaladministration, policy and procedure

7/30/2019 SC04_Kern_David_201191

24/41

7

CONSIDERATIONS IN

COLLECTING INFORMATION

aAre data sources already available?

aHow many resources should be devoted tothis process?

aWhat are long term plans for using theinformation that is gathered? (Need forIRB Approval?)

METHODS FOR

COLLECTING INFORMATION

aInventory of existing trainingaInformal discussion

aFormal interviews

aFocus groups

aQuestionnaires

aObservation

aTests

aAudits

aStrategic Planning Sessions

EXAMPLE:ETHICS CURRICULUM

FOR IM PGY-2 AND 3 RESIDENTS

Methods for Collecting Information:

aInventory of previous curricula to which theresidents had been exposed in the residency

aInformal interviews with several residents

aSurvey of all targeted residents about theirprevious training, perceived competencies, andperceived needs.



Findings:

aResidents had considerable training related toautonomy, beneficence, substituted judgment,advance directives, and end-of-life decisions

aResidents had no training related to clinicaldecision making in the context of competinginterests such as patient vs. family vs. societalneeds, or to reimbursement structures forambulatory healthcare.

aAll of the residents' training had centered around

inpatient cases.



Response:

aThe curriculum developers decided to focustheir curriculum on clinical decision making,

with an emphasis on the ambulatorysetting.

EXAMPLE:

MUSCULOSKELETAL CURRICULUM


aReview of existing training

aSenior resident exit interview

aFocus group of residents at noon conference

aSurvey of current residents

7/30/2019 SC04_Kern_David_201191

25/41

8

EXAMPLE:


Findings:aWrong case mix in Rheumatology and

Orthopedics

aLow self-rated proficiency

aLow levels of training and clinical experience

aStrong desire for training

aPreferred educational method was directsupervision of patient care by primary carepractitioners with expertise in MS medicine.

EXAMPLE:


Response:

a Development of workshops and syllabus materials ondiagnosis and management, including injection therapy, forthe musculoskeletal disorders commonly presenting toprimary care practices.

a Institution of a new primary care musculoskeletal clinicsupervised by Internal Medicine preceptors with a specialinterest in musculoskeletal medicine, to which other primarycare practitioners referred patients for diagnosis andinjection. The clinic enabled residents to gain supervisedexperience in diagnosing and managing commonmusculoskeletal disorders, such as back, shoulder, elbow,hip, knee and foot pain, to perform procedures, and to seerole models in action.

Houston TK, Connors RL, Cutler N, Nidiry MA.A primary care musculoskeletal clinic forresidents: success and sustainability. J Gen Intern Med. 2004 May;19(5 Pt 2):524-9.

EXAMPLE:

PALLIATIVE CARE CURRICULUM


aFocus group of trainees (residents)

aFocus group of nurses

aInformal interviews with selected faculty.

EXAMPLE:


Findings:

aWhile the trainees conceptually understood theselection and dosing of opioid analgesics, therewas resistance in prescribing them, especially inhigh doses or to those patients with a history ofsubstance abuse disorders.

aThe resistance was prevalent among nursingand attending staff as well.

EXAMPLE:


Response:

aOne of the objectives of the curriculum becameaddressing attitudinal barriers in prescribingopiates.

aNurses and attendings became targeted learners

as well.

At the conclusion of Steps 1 & 2:

aYou have a strong argument for the need foryour curriculum.

aSet the stage for generalizability and

dissemination of your curriculum.aUnderstand the particular needs of your

targeted learners and institution(s)

aIdentified potential resources and support.

aHave the introduction and elements of adiscussion for a manuscript

aYou are now the expert!

7/30/2019 SC04_Kern_David_201191

26/41

9

QUESTIONS?

STEP 3:

GOALS & OBJECTIVES

focusing the curriculum

the reason for teaching

Glassick Criteria







GOALS

aGoalsare broad educational objectives, thegeneral ends toward which an effort is

directed. They are usually not measurable aswritten.

aExample: The purpose of themusculoskeletal curriculum is to prepareresidents to evaluate and managemusculoskeletal conditions commonly seen inGeneral Internal Medicine practice.

OBJECTIVES

aObjectivesare specific & measurable.

aExamples: By the end of the curriculum,residents will demonstrate their ability to:`Correctly label shoulder anatomy on a diagram.

`List the 4 most common causes of shoulder pain.`Perform an appropriate physical examination of the

shoulder.

`Diagnose the 4 most common causes of shoulderpain, based on history and physical examination.

Appropriately manage these 4 conditions.

Appropriately perform subacromial and intra-articularinjections.

7/30/2019 SC04_Kern_David_201191

27/41

10

GOALS VS. OBJECTIVES

aGoals: visionary, lofty, expansive

aObjectives: precise, measurable

IMPORTANCE OF OBJECTIVES

aHelp prioritize

aDirect content

aIdentify learning methods (congruity)

aEnable and direct evaluation

aPermit clear communication to learners,faculty, and other stakeholders

aRequired by ACGME/ LCME

TYPES OF OBJECTIVES

aLearner Objectives

cognitive

affective

psychomotor (skill/competence vsbehavior/performance)

aProcess Objectives

curriculum implementation measures

aPatient / Healthcare Outcome Objectives

effects beyond those delineated in learner and

process objectives,

e.g. patient outcomes, career choice

LEVELS OF OBJECTIVES

aIndividual Learner

aAggregate or Program

HOW TO WRITE OBJECTIVES

1. Who

2. will do

3. how much / how well

4. of what

5. by when?


By the end of the gynecology curriculum(BY WHEN), each IM resident (WHO) willhave demonstrated (WILL DO), at least

once (HOW MUCH), the appropriatetechnique (HOW WELL), as defined on acheck sheet, for obtaining a Pap smearand cervical cultures (OF WHAT).

7/30/2019 SC04_Kern_David_201191

28/41

11


(Table 4.1)

Words Open to More

Interpretationsa know or understand

a be able

a know how, internalize

a appreciate

a grasp the significance of

a believe

a learn

Words Open to Fewer

Interpretationsa list, recite, present, define,

describe, give an example of

a demonstrate (as measured by)

a use, incorporate

a rate as valuable

a rank as important

a identify or rate as an opinion

a (use one of the above terms)


aAchieve balance between specificity andreadability

aHave someone else read them andexplain them to you

aHave a manageable number of objectives

EXAMPLE: PROFESSIONALISM

aPoor: Residents will be able to obtain informed consent.

aBetter: By the end of PGY-2, residents will routinelyobtain informed consent that includes the followingcritical elements:

`natural course without treatment

àlternative treatments

`risks and benefits of the alternative treatments

àssessment of patient understanding

`sensitivity to patient needs and preferences

ànswering of patient questions

EXAMPLE:Communication Skills

aCognitive Objective: By the end of the rotation, residentswill be able to list the critical components of effective patienteducation: assessing patients knowledge, beliefs, needs;tailoring education to needs; giving information clearly andeffectively; checking patients comprehension and agreement.

aAffective Objective: By the end of the rotation, residentswill believe that it is a physicians role to effectively educate

patients.

aPsychomotor Objective: By the end of the rotation,residents will have demonstrated their proficiency in the abovepatient education skills. By the end of residency, patient surveys

will reveal the implementation of of these skills in practice.

EXAMPLE:Communication Skills

aProcess Objective: By the end of the rotation, eachresident will have reviewed 3 videotapes of their actualpatient interactions with their colleagues and a facilitator.

aPatient / Healthcare Outcome Objective: Two monthsafter the end of the rotation, patients of trained residents willbe more satisfied with their physicians and be morecompliant with their prescribed medication regimen thanpatients of untrained residents.

REMEMBER

aGoals provide overall direction

aA manageable number of objectives shouldìnterpret the goals

`focus and prioritize curricular components

aMost curricula encompass more than thesum of their written objectives

aObjective can be written to encouragecreativity, flexibility, and nonspecifiedlearning relevant to curricular goals

7/30/2019 SC04_Kern_David_201191

29/41

12

Example: Goal & Objective for

a Ward Rotation

aGoal: Students and residents will become self-directedlearners.

aObjective that encourages creativity,flexibility, and

nonspecified learning:

`Each week during a ward rotation, each student andeach resident will identify a question relevant to the

care of one of their patients, and briefly report duringmorning rounds the sources used, the search timerequired, and the answer to their question.

QUESTIONS?


aStep 1: Problem Identification & General NeedsAssessment

aStep 2: Targeted Needs Assessment

aStep 3: Goals & Objectives

STEP 4:

EDUCATIONAL STRATEGIES

accomplishing educational objectives

Glassick Criteria







7/30/2019 SC04_Kern_David_201191

30/41

13

STEP 4:

EDUCATIONAL STRATEGIES

aContent of the Curriculum

aEducational Methods

EDUCATIONAL METHODS

Education is not the filling of a pail,but the lighting of a fire.

William Butler Yeats

EDUCATIONAL METHODS:

GENERAL GUIDELINES

aMaintain congruence between objectives andmethods

aUsed multiple educational methods

aChoose educational methods that are feasible

aRemember that assessment can drive learning(internalization of assessment criteria)

Congruence: Educational Methods

for Achieving Cognitive Objectives

aReading

aLecture

aAudio-visual Materials

aDiscussion

aInteractive Programmed Learning

Congruence: Educational Methods

for Achieving Affective Objectives

aExposure (readings, discussions,experiences)

aFacilitation of openness, introspection,& reflection

aRole models

Congruence: Educational Methods for

Achieving Psychomotor Objectives

aSkill or Competency Objectives`Supervised clinical experience

`Simulations

Audio or visual review of skills

aBehavioral or Performance Objectives`Removal ofbarriersto performance

`Provision ofresourcesthat facilitate performance

`Provision ofreinforcementsfor performance

7/30/2019 SC04_Kern_David_201191

31/41

14

NEW CHALLENGES

What are the educational methods that willfoster the attainment of the ACGMEcompetencies?

`Professionalism

`Practice-based learning and improvement

`Systems-based practice

Educational Methods for

Achieving PBLI

aTraining in skills related to self-directed learning`Self assessment, including audits of own care

`Information searching

`Critical appraisal

`Clinical decision-making

aIndependent learning projects

aPersonal learning plans or contracts

aUse of learning portfolios

aEncouraging learners to formulate and answer theirown questions

aRole modeling

Educational Methods for Systems-

Based Practice / Teamwork

a Providing feedback on costs of care

a Case conferences focused on cost effectiveness andquality of care

a Training in team skills

a Collaborative learning experiences

a Opportunities to work in disease management programs

a Participation in quality improvement and safety teams

a Redesigning teaching services towards multidisciplinaryintegration / work environments that model effectiveteamwork

a Focused curricula on team functioning and related skills

a Reflection on / assessment of team function

Educational Methods for

Promoting Professionalism

aFaculty role modeling

aFacilitated reflection on experience

aParticipation in writing professionalism goals

aPeer evaluations

aParticipation in patient advocacy groups

aService learning and volunteerism

aAttention of institutional and program leaders tothe policies and culture of the training institution.

EXAMPLE: Primary Care GYN

a By the end of the gynecology curriculum, each residentwill have demonstrated, at least once, the appropriatetechnique, as defined on a check sheet, for obtaining aPap smear and cervical cultures.` Lecture / demonstration on proper communication /

procedural skills

` Practice with genitourinary teaching associate (GTA)` Observed pelvic exams in resident clinic

a By the end of residency, patient surveys will reveal theimplementation of these procedures in their continuityclinic.

` Pap exam trays in clinic

` Provision of trained MAs to assist in clinic

`Provision of audit feedback to residents on # of the patientsthey have seen eligible for Pap / Culture, # of these who havereceived the tests, and location of testing.

EXAMPLE: MusculoskeletalCurriculum(Houston TK et.al. A primary care musculoskeletal clinic for residents: success and sustainability. J GenIntern Med. 2004 May;19(5 Pt 2): 524-9.)

aBy the end of the curriculum, residentswill be able to perform subacromial andintra-articular corticosteroid injections of

the shoulder, using proper technique.`Didactic discussion, with demonstration

`Supervised practice with simulated models

`Supervised practice with real patients in aspecially designed musculoskeletal clinic

7/30/2019 SC04_Kern_David_201191

32/41

15

EXAMPLE: Informed Consent

aBy the end of PGY-2, residents will routinelyobtain informed consent that includes thepreviously listed critical elements:`Didactic / handout on components of informed consent, and

relevant information on 10 most common procedures.

`Demonstration by role model physicians.

`Supervised practice, with explicit reflection and feedback.

`Feedback of patient survey information and nurse evaluations.

`Faculty and nurse development that addresses the informal andhidden curriculum.

`General and procedure specific forms that enable informedconsent.

True teaching is not an

accumulation of knowledge;

it is an awakening of

consciousness which goes

through successive stages.

from a temple wall

inside an Egyptian pyramid

Education is what survives

when what has been learned

has been forgotten.

-B.F. Skinner

QUESTIONS?

QUESTIONS?

STEP 6:

EVALUATION AND FEEDBACK

assessing the achievement of objectives

and stimulating continuous improvement.

7/30/2019 SC04_Kern_David_201191

33/41

16

Glassick Criteria






EVALUATION AND

FEEDBACK: WHY?

aTo determine if goals and objectives met

aTo provide information for improvement

aTo assess individual achievement

aTo satisfy external requirements (e.g., ACGME)

aTo document accomplishments of curriculumdevelopers

aTo maintain and garner support

aTo serve as a basis for presentations/publications

ACGME REQUIREMENTS (U.S.)

Resident Evaluation

aEvaluate the 6 competencies

patient care, medical knowledge, practice-based learning,

interpersonal & communication skills, professionalism,

systems based practice

aInclude observation

aInclude humanistic qualities in each observation

aInclude chart audit

aEvaluate in writing

aMaintain permanent records


Resident Evaluation (continued)

aProvide regular and timely feedback

aHave a process in place to achieve progressive

improvements in residents competence &performance


Program Evaluation

aProgram and faculty members should beevaluated by residents

aThere should be a process in place to reviewand improve the program that includes reviewof: evaluations by residents; aggregateevaluations of residents; patient mix; quality ofsupervision

7/30/2019 SC04_Kern_David_201191

34/41

17

LCME: FUNCTIONS AND STRUCTURE

OF A MEDICAL SCHOOL (U.S.)

The medical school faculty must establish

principles and methods for the evaluation of

student achievement, and make decisions

regarding promotion and graduation. The

evaluation of student achievement must employ a

variety of measures of knowledge, competence

and performance, systematically and sequentially

applied throughout medical school. Each

accredited program must utilize methods for

determining the quality of its program and the

level of achievement of its students compared to

national norms.

THE 10 TASKS OF EVALUATION

I. Identify UsersII. Identify Uses

III. Identify Resources

IV. Identify EvaluationQuestions*

V. Choose EvaluationDesigns*

VI. Choose MeasurementMethods* and

Construct Instruments

VII. Address EthicalConcerns

VIII. Collect Data

IX. Analyze Data

X. Report Results

IV. IDENTIFY

EVALUATION QUESTIONS

aEnsure that some evaluation questions arecongruent with learner objectives.

aInclude some evaluation questions that do notrelate to specific learner objectives (programevaluation).

aInclude some that are open-ended in nature.

aPrioritize and select key evaluation questions,based upon user needs and feasibility.

EXAMPLE: MED-PSYCH

aDo residents communication skills improvefollowing training? Are they superior to those ofuntrained residents?

aHow do residents rate the curriculum and itsvarious components?

aWhat are its strengths?

aHow can it be improved?

V. CHOOSE EVALUATION DESIGNS

aChoose an evaluation design congruent with theevaluation question.

aChoose an evaluation design that is feasible in

terms of resources.aConsider internal validity : history, maturation,

testing, instrumentation, selection bias, dropoutbias, statistical regression.

aConsider external validity or generalizabilty toother populations and settings.

V. COMMON EVALUATION DESIGNS

aPosttest Only X ---- O

aPretest Posttest O1---- X ---- O2

aControl Group E (O1---) X ---- O2(R)

C (O1---)------- O2

7/30/2019 SC04_Kern_David_201191

35/41

18

CONGRUITY

EXAMPLE: MED-PSYCH

aDo residents communication skills improvefollowing training?

O1---- X ---- O2

aAre they superior to those of untrainedresidents?

E X ---- O2R

C ------- O2

CONGRUITY

EXAMPLE: MED-PSYCH

aHow do residents rate the curriculum and itsvarious components?

aWhat are its strengths?

aHow can it be improved?

X ---- O

VI. CHOOSE MEASUREMENT METHODS

AND CONSTRUCT INSTRUMENTS

aGlobal rating forms

aSelf-assessmentforms

aDirect observation

aWritten orcomputerized tests

aOral Examinations

aPerformance audits

aPatient Questionnaires

aLearner Questionnaires

aIndividual interviews

aGroup interviews /discussions

aEssays

(see Table 7.3)

VI. CHOOSE MEASUREMENT METHODS

AND CONSTRUCT INSTRUMENTS

aChoose a measurement method that iscongruent with the evaluation question.`i.e. written tests for knowledge objectives;

observation session for skill objectives

aChoose a measurement method that isfeasible in terms of available resources.

VI: Choose Measurement

Methods: Congruence

K no wl ed ge A tt it ud e Sk il l/Per fo rm an ce

Learner Oral examWritten exam/QaireCase discussionGlobal rating scales

Learner interviewQuestionnaireSelf-evaluationGlobal rating scales

Direct observationAudio/video

observationRecord audit

Outcomes of careSelf-evaluationPatient interviewGlobal rating scales

Program Aggregated scoresfrom above methods

Aggregated scoresfrom above methods

Aggregated scoresfrom above methods

EXAMPLE: MED-PSYCH

aDo residents communication skills improve

following training?

`Pre-post self-assessments

`Faculty observation during experiential learning exercises

`consensus global rating formaAre they superior to those of untrained residents?

`Randomized controlled trial using a rigorously evaluated

videotape of a resident-standardized patient interaction.

aHow do residents rate the curriculum and itsvarious components? / What are its strengths? /How can it be improved?

`End-of-rotation questionnaire

7/30/2019 SC04_Kern_David_201191

36/41

19

EXAMPLE:

MUSCULOSKELETAL

a Can residents perform an appropriate physical

examination of the shoulder? perform subacromial andintra-articular injections using proper technique?

Òbserved physical examinations, preceptor checklists.Òbserved shoulder injections, preceptor checklists.

ÒSCEsa Can residents appropriately diagnose and manage the 4

most common causes of shoulder pain?

`Closed and open-ended case-based test.ÒSCEs

a How do residents rate the curriculum and its variouscomponents? / What are its strengths? / How can it beimproved?Ènd-of-rotation questionnaire

EXAMPLE:

INFORMED CONSENT

a Do residents know the essential components of informed

consent?`Written exam.

`Demonstration of inclusion of components in observedpatient interactions.

aAre residents capable of obtaining informed consent thatincludes the essential components for the 5 mostcommon procedures and for an unfamiliar procedure?

`Supervised observation and documentation, with rater usingchecklist; or

`Rater evaluation of audio or videotaped patient interactions.a Do residents routinely include the essential component

of informed consent in practice?`Nurse survey form.

`Patient survey form.

QUESTIONS?

TAKE HOME MESSAGE:

CONGRUENCY

OBJECTIVES

EDUCATIONAL

METHODS

EVALUATION

METHODS

STEP 5: IMPLEMENTATION

making the curriculum a reality

converting a good plan into an

accomplishment.

STEP 5: IMPLEMENTATION

aIdentify Resources

aObtain Support (Institutional, External)

aDevelop Administrative Mechanisms toSupport the Curriculum

aAnticipate and Address Barriers

aHave a Plan for Introducing the Curriculum

7/30/2019 SC04_Kern_David_201191

37/41

20

IDENTIFY RESOURCES

a Personnel: faculty, secretarial, and othersupport staff; patients; other

a Time: faculty, support staff, learners

a Facilities: space, equipment, clinical sites

a Funding/Costs: direct financial costs, releasetime, hidden or opportunity costs

OBTAIN SUPPORT

aInternal`From: administrators (dean, department chair,

program director, division director, hospitaladministrator), faculty, learners, other stakeholders

`For: personnel, resources, political support

aExternal

`From: government, donors and philanthropicorganizations, accrediting bodies, professionalsocieties

`For: funding, requirements, political support,curricular or faculty development resources

DEVELOP

ADMINISTRATIVE MECHANISMS

aAdministrative Structure: delineate responsibilitiesand decision making

aCommunication:

`Content: rationale; goals & objectives; informationabout the curriculum, schedules, learners, faculty,facilities and equipment; changes; evaluation results;etc.

`Mechanisms: memos, meetings, syllabus, website,site visits, reports, news articles, etc.

aOperations: preparation and distribution of materials;

collection, collation, and distribution of evaluation data,curricular revisions and changes; etc.

ANTICIPATE AND ADDRESS

BARRIERS

aFinancial and Other Resources

aCompeting Demands

aPeople: attitudes, job/role security,power and authority

PLAN FOR

INTRODUCING THE CURRICULUM

aPilot

aPhase-In

aFull Implementation

EXAMPLE: MUSCULOSKELETAL

aEstablishment of a musculoskeletal clinicthat provided a concentration of casesthat represented the epidemiology of MSproblems in primary care practice.

`3 of 4 curriculum developers worked in CBP

`Financial analysis and pilot demonstrated feasibility

Administrative support was obtained

`PI, GNA, TNA provided evidence for need

Advance communication with Rheumatology andOrthopedics

`Evaluation that demonstrated success & popularity

7/30/2019 SC04_Kern_David_201191

38/41

21

EXAMPLE: MED-PSYCH

aExternal funding for GIM Residency thatrequired CS and PS training

aStarted with external experts

aFaculty development / co-teaching

aPhase-in with interested residents

aDocumented resident satisfaction

aPresented evaluations and examples todepartment chair.

aExpanded to entire residency program

QUESTIONS?


aStep 4: Educational Strategies

aStep 6: Evaluation

aStep 5: Implementation

CURRICULUM DEVELOPMENT :

OVERVIEW

a1. Problem ID & Genl Needs Assessment

a2. Needs Assessment of Targeted Learners

a3. Goals & Objectives

a4. Educational Strategies

a5. Implementation

a6. Evaluation & Feedback

a7. Curriculum Maintenance & Enhancement

a8. Dissemination

LONGITUDINAL PROGRAM IN


a9 1/2 Months

aWorkshops on Each Curricular Step

aSessions on Literature Searching, IRB, Searchingfor Funding, Simulation Center, Dissemination

aMentored Project

aIndividual Meetings with Facilitators, WrittenFeedback on Each Step

aWork-in-Progress Sessions

aWritten Paper / Curriculum and Oral Presentation

REFERENCES

a Kern DE, Thomas PA, Howard DM, Bass EB. Curriculum Developmentfor Medical Education: A Six-Step Approach. Baltimore (MD): JohnsHopkins University Press; 1998. / Kern DE, Thomas PA, Hughes MT,eds. Curriculum Development for Medical Education: A Six-Step

Approach. 2nd ed. Baltimore (MD): Johns Hopkins University Press;2009. In Press.

a Thomas PA, Kern DE. Internet resources for curriculum developmentin medical education: an annotated bibliography. J Gen Intern Med.2004; 19(5): 598-604.

a ACGME Outcome Project: http://www.acgme.org/Outcome/a ACGME: http://www.acgme.org/a AAMC/LCME: http://www.lcme.org/a Kern DE, Branch WT, Green ML, et.al. Making it count twice: how to

get curricular work published. SGIM Workshop, 2005. Available athttp://www.sgim.org/userfiles/file/AMHandouts/AM04/Workshops/WB09.pdf .

a David E. Kern, MD, MPH [email protected]

7/30/2019 SC04_Kern_David_201191

39/41

22

NEXT STEPS:

aReflect on the main things you havelearned from the CD Workshop.

aThink concretely about how you can applythem to the work you envision.

aArticulate next steps that will ensure thatyou apply what you have learned tosomething that is meaningful to you.

SUMMARY

TAKE HOME MESSAGE

OBJECTIVES

EDUCATIONAL

METHODS

EVALUATION

METHODS

CURRICULUM DEVELOPMENT :

OVERVIEW

a1. Problem ID & Genl Needs Assessment

a2. Needs Assessment of Targeted Learners

a3. Goals & Objectives

a4. Educational Strategies

a5. Implementation

a6. Evaluation & Feedback

a7. Curriculum Maintenance & Enhancement

a8. Dissemination

LONGITUDINAL PROGRAM IN


a9 1/2 Months

aMentored Project

aWorkshops on Each Curricular Step

aIndividual Meetings with Facilitators, WrittenFeedback on Each Step

aWork-in-Progress Sessions

aWritten Paper / Curriculum and Oral Presentation

aSessions on Literature Searching, Survey Design,IRB, Searching for Funding, Dissemination

7/30/2019 SC04_Kern_David_201191

40/41

23

DISSEMINATION / PUBLICATION

making it count twice

Glassick*

Criteria for Scholarship


3. Appropriate methods

4. Significant results

5. Effective dissemination

6. Reflective critique

*Scholarship Assessed:Evaluation of the Professoriate,1997

DISSEMINATION

aPublication in Peer Reviewed Journals

aElectronic Publication

aPresentation

`Local

`Regional Professional Meetings

`National and International ProfessionalMeetings

Needs Assessments

a Important information

a New information or systematic review ofexisting data

a Methodologically sound

a Generalizable information

e.g. Ratanawongsa N, Bolen S, Howell EE, et.al. Residents perceptions ofprofessionalism in training and practice: barriers, promoters, and duty hourrequirement s. J Gen Intern Med. 2006;21(7):758-763.

e.g. Clark JM, Houston TK, Kolodner K, et.al. Teaching the teachers: a national survey of

faculty development in departments of medicine of U.S. teaching hospitals. J GenIntern Med 2004;19:205-214.

Goals and Objectives

aTopic of demonstrated importance

aWell described, methodical, sound process ofdeveloping the goals and objectives

aA broad consensus supports the goals andobjectives.

aProfessional organizations, authoritative bodiessupport the goals and objectives.

aE.g., CDIM/SGIM Core Medicine Clerkship,FCIM

Educational Strategies

aTopic of demonstrated importance

aEducational strategies innovative and add to theexisting literature

aEducational strategies are adaptable to other

settings.aEvaluation provides evidence regarding efficacy (If

very innovative and topical, less rigorous evaluationmay be acceptable.)

e.g. Houston TK, Connors RL, Cutler N, Nidiry MA. A primary care musculoskeletal clinicfor residents: success and sustainability. J Gen Intern Med. 2004 May;19(5 Pt 2):524-9.

e.g. Branch WT Jr., Kern DE, Gracey K, Haidet P, Weissmann P, Mitchell G, Inui T, NovakTL. Teaching the human dimensions of care in clinical settings. JAMA.2001;286:1067-1074

7/30/2019 SC04_Kern_David_201191

41/41

Evaluations of

Educational Interventions (1)

a Important topicaGeneralizable sample: multi-institutional

aStrong evaluation design: sufficient numbers / power,control group, long-term f/u, blinded raters

a Important outcomes: clinical outcomes >behaviors/performance > skills,attitudes > knowledge> satisfaction

aStrong assessment methods: objective, reliability,content and other measures of validity

aAppropriate data analysis: when appropriate, accountfor confounding variables with multivariate analysis

aEvaluation strategy innovative

aContribution to the existing literature

Evaluations of

Educational Interventions (2)

e.g. Roter DL, Hall JA, Kern DE, Barker LR, Cole KA, Roca RP. Improving physiciansinterviewing skills and reducing patients emotional distress: a randomized clinical

trial. Arch Intern Med 1995; 155: 1877-1884.e.g. Smith RC, Lyeles JS, Mettler J, et al. The effectiveness of intensive training for

residents in interviewing: a randomized, controlled trial. Ann Intern Med.1998;128:118-126.

e.g.Watkins RS, Moran WP. Competency-based learning: the impact of targetedresident education and feedback on Pap smear adequacy rates. J Gen Intern Med2004;19:545-548.

e.g. Morrison EH, Rucker L, Boker JR, et. al. The effect of a 13-hour curriculum toimprove residents teaching skills: a randomized trial Ann Intern med 2004;141:257-263.

e.g. Cornuz J, Humair JP, Seematter L, et. Al. Efficacy of resident training in smokingcessation: a randomized control trial of a program based on application of behavioraltheory and practice with standardized patients. Ann Intern Med 2002:429-437.

e.g. Sisson SD, Hughes MT, Levine D, Brancati FL. Effect of an internet-based curriculumon postgraduate education: a multicenter intervention. J Gen Intern Med2004;19:505-509.

e.g. . Windish DM, Gozu A, Bass EB, et.al. A ten-month program in curriculumdevelopment for medical educators: 16 years of experience. J Gen Intern Med.2007;22:655-61. / Gozu A, Windish DM, Knight AM, et.al. Long-term follow-up of aten-month programme in curriculum development: a cohort study. Med Educ.2008;42:684-692.

NEXT STEPS:

aReflect on the main things you havelearned from the CD Workshop.

aThink concretely about how you can applythem to the work you envision.

aArticulate next steps that will ensure thatyou apply what you have learned tosomething that is meaningful to you.

REFERENCES

a Kern DE, Thomas PA, Howard DM, Bass EB. Curriculum Developmentfor Medical Education: A Six-Step Approach. Baltimore (MD): JohnsHopkins University Press; 1998. / Kern DE, Thomas PA, Hughes MT,eds. Curriculum Development for Medical Education: A Six-Step

Approach. 2nd ed. Baltimore (MD): Johns Hopkins University Press;2009. In Press.

a Thomas PA, Kern DE. Internet resources for curriculum developmentin medical education: an annotated bibliography. J Gen Intern Med.2004; 19(5): 598-604.

a ACGME Outcome Project: http://www.acgme.org/Outcome/a ACGME: http://www.acgme.org/a AAMC/LCME: http://www.lcme.org/a Kern DE, Branch WT, Green ML, et.al. Making it count twice: how to

get curricular work published. SGIM Workshop, 2005. Available athttp://www.sgim.org/userfiles/file/AMHandouts/AM04/Workshops/WB09.pdf .

a David E. Kern, MD, MPH [email protected]

THE ENDTHANK YOU

Date post:	14-Apr-2018
Category:	Documents
Upload:	sivapathasekaran
View:	216 times
Download:	0 times

SC04_Kern_David_201191

Documents