+ All Categories
Home > Documents > Competency Based Training

Competency Based Training

Date post: 12-Feb-2016
Category:
Upload: dacey
View: 33 times
Download: 0 times
Share this document with a friend
Description:
Enhancing Competency Based Training Through the Screening, Selection, and Performance Evaluation of Psychology Interns. Paul Robins, Ph.D 1,2 ., Thomas Power, Ph.D 1,2 ., Mary Rourke,Ph.D 1 ., Melissa Alderfer, Ph.D 1,2 ., Cathi Grus, Ph.D 3 ., & Marley Watkins, Ph.D 4 - PowerPoint PPT Presentation
Popular Tags:
29
Enhancing Competency Based Enhancing Competency Based Training Through the Training Through the Screening, Selection, and Screening, Selection, and Performance Evaluation of Performance Evaluation of Psychology Interns Psychology Interns Paul Robins, Ph.D Paul Robins, Ph.D 1,2 1,2 ., Thomas ., Thomas Power, Ph.D Power, Ph.D 1,2 1,2 ., Mary ., Mary Rourke,Ph.D Rourke,Ph.D 1 ., Melissa Alderfer, ., Melissa Alderfer, Ph.D Ph.D 1,2 1,2 ., Cathi Grus, Ph.D ., Cathi Grus, Ph.D 3 ., & ., & Marley Watkins, Ph.D Marley Watkins, Ph.D 4 1 The Children’s Hospital of Philadelphia, The Children’s Hospital of Philadelphia, 2 University of Pennsylvania, University of Pennsylvania, 3 APA, APA, 4 Arizona Arizona State University State University
Transcript
Page 1: Competency Based Training

Enhancing Competency Based Enhancing Competency Based Training Through the Screening, Training Through the Screening,

Selection, and Performance Selection, and Performance Evaluation of Psychology InternsEvaluation of Psychology Interns

Paul Robins, Ph.DPaul Robins, Ph.D1,21,2., Thomas Power, ., Thomas Power, Ph.DPh.D1,21,2., Mary Rourke,Ph.D., Mary Rourke,Ph.D11., Melissa ., Melissa

Alderfer, Ph.DAlderfer, Ph.D1,21,2., Cathi Grus, Ph.D., Cathi Grus, Ph.D33., & ., & Marley Watkins, Ph.DMarley Watkins, Ph.D44

11The Children’s Hospital of Philadelphia, The Children’s Hospital of Philadelphia, 22University of University of Pennsylvania, Pennsylvania, 33APA, APA, 44Arizona State UniversityArizona State University

Page 2: Competency Based Training

Competency Based TrainingCompetency Based Training

Forms a basis for defining and measuring Forms a basis for defining and measuring trainee learning outcomestrainee learning outcomesCompetencies conceptualized as elements or Competencies conceptualized as elements or components of competence, and consist of components of competence, and consist of discrete discrete knowledge, skills, and attitudes knowledge, skills, and attitudes (Kaslow (Kaslow et al., 2004). et al., 2004). Cube model: twelve core competencies Cube model: twelve core competencies conceptualized as either conceptualized as either foundational foundational or or functional functional competencies (Rodolfa, et. al, 2005) competencies (Rodolfa, et. al, 2005)

l

Page 3: Competency Based Training

Today’s PresentationToday’s Presentation1.1. Describe process of developing a blueprint of Describe process of developing a blueprint of

foundational and functional competenciesfoundational and functional competencies2.2. Describe competency based intern evaluation Describe competency based intern evaluation

instrument instrument 3.3. Describe process of developing an intern Describe process of developing an intern

screening tool that reflects core mission screening tool that reflects core mission prioritiespriorities

4.4. Present reliability and validity data of the intern Present reliability and validity data of the intern screening toolscreening tool

5.5. Future directions/learnings Future directions/learnings

Page 4: Competency Based Training

Developing Core CompetenciesDeveloping Core Competencies

Foundational documentsFoundational documents– Hospital missionHospital mission– Behavioral Health Center missionBehavioral Health Center mission– Internship program missionInternship program mission– Funding priorities: GPE and LEND missions Funding priorities: GPE and LEND missions

Translate documents to core training domainsTranslate documents to core training domains– Iterative processIterative process– Responsive to multiple missions and our Responsive to multiple missions and our

understanding of training psychologists as leaders in understanding of training psychologists as leaders in the fieldthe field

Page 5: Competency Based Training

Foundational Documents:Foundational Documents:Hospital MissionHospital Mission

The Children’s Hospital of Philadelphia, the oldest hospital in the United States dedicated exclusively to pediatrics, strives to be the world leader in the advancement of health care for children by integrating excellent patient care, innovative research and quality professional education into all of its programs.

Page 6: Competency Based Training

Behavioral Health Center MissionBehavioral Health Center Mission

The Children’s Hospital of Philadelphia’s Behavioral Health The Children’s Hospital of Philadelphia’s Behavioral Health Center (BHC) strives to be among the nation’s top Center (BHC) strives to be among the nation’s top leadersleaders in in children’s behavioral health. The BHC provides an children’s behavioral health. The BHC provides an inter-inter-disciplinary approachdisciplinary approach to improving the behavioral health and to improving the behavioral health and development of children and their families and advancing development of children and their families and advancing knowledge through excellent clinical care, research, training knowledge through excellent clinical care, research, training and advocacy. The BHC promotes innovative research and and advocacy. The BHC promotes innovative research and the the integration of science and practiceintegration of science and practice, focusing on , focusing on prevention and interventionprevention and intervention. We partner with others within . We partner with others within the hospital, regionally, and nationally to increase the the hospital, regionally, and nationally to increase the availability of availability of evidence-based behavioral servicesevidence-based behavioral services for all for all patients at CHOP. The BHC is committed to patients at CHOP. The BHC is committed to outstanding outstanding scientist-practitioner trainingscientist-practitioner training and to supporting the and to supporting the competence of caregivers and staff. Our endeavors competence of caregivers and staff. Our endeavors integrate disciplines and systems of care in children’s health. integrate disciplines and systems of care in children’s health.

Page 7: Competency Based Training

Internship Program MissionInternship Program Mission

The Internship Program aims to prepare advanced The Internship Program aims to prepare advanced doctoral-level students to take doctoral-level students to take leadership rolesleadership roles in developing, providing, evaluating, and in developing, providing, evaluating, and disseminating effective psychological services disseminating effective psychological services for the for the diverse population of children and diverse population of children and familiesfamilies with with health and/or mental health needshealth and/or mental health needs. . The Program is committed to preparing The Program is committed to preparing psychologists as leaders in the field, engaged in psychologists as leaders in the field, engaged in clinical practice, advocacy, interdisciplinary clinical practice, advocacy, interdisciplinary collaboration, and scientific investigation.collaboration, and scientific investigation.

Page 8: Competency Based Training

Funding AgenciesFunding AgenciesHRSA: BHP (GPE) & MCHB (LEND)HRSA: BHP (GPE) & MCHB (LEND)

Vision Vision The Health Resources and Services Administration The Health Resources and Services Administration (HRSA) envisions optimal health for all, supported by a (HRSA) envisions optimal health for all, supported by a health care system that assures access to health care system that assures access to comprehensive, culturally competent, quality care. comprehensive, culturally competent, quality care.

Mission Mission HRSA provides national leadership, program resources HRSA provides national leadership, program resources and services needed to and services needed to improve accessimprove access to to culturally culturally competentcompetent, quality health care. , quality health care.

Page 9: Competency Based Training

Development of Core Development of Core CompetenciesCompetencies

Functional Competency Domains (6)Functional Competency Domains (6)1.1. AssessmentAssessment2.2. Intervention/Intervention/prevention/advocacyprevention/advocacy3.3. ConsultationConsultation4.4. ResearchResearch5.5. Professional developmentProfessional development6.6. Diversity effectivenessDiversity effectiveness

Page 10: Competency Based Training

Identify Unique Strengths of Identify Unique Strengths of Program—the “Big Five” Program—the “Big Five”

Foundational Competency DomainsFoundational Competency Domains1.1. Prevention/advocacy (BHP/GPE, future)Prevention/advocacy (BHP/GPE, future)2.2. Ecological/systems underpinning (our Ecological/systems underpinning (our

unique history, PCGC)unique history, PCGC)3.3. Leadership training (Hospital, MCHB/LEND)Leadership training (Hospital, MCHB/LEND)4.4. Interdisciplinary collaboration (Hospital—Interdisciplinary collaboration (Hospital—

family centered care, systems)family centered care, systems)5.5. Community based, medically underserved Community based, medically underserved

populations (BHP/GPE)populations (BHP/GPE)

Page 11: Competency Based Training

Content Validity--BlueprintContent Validity--Blueprint

6 Functional Competency Domains X 5 6 Functional Competency Domains X 5 Foundational Competency Domains—55 Foundational Competency Domains—55 itemsitemsProvided blueprint of core skills and Provided blueprint of core skills and outcomes outcomes (content validity)(content validity)

Formed basis of required intern skills and Formed basis of required intern skills and outcomesoutcomes

Page 12: Competency Based Training

Assessing Intern CompetenciesAssessing Intern Competencies

Core competencies rated by rotation Core competencies rated by rotation supervisors along a 3 point scale supervisors along a 3 point scale (scaling (scaling being revised to reflect APA CoA standards)being revised to reflect APA CoA standards)

““Based upon Based upon end of the internshipend of the internship performance performance expectations, rate this intern on each item using the expectations, rate this intern on each item using the following scale:following scale:

n/a = not applicable1 = does not yet meet expectations2 = meets expectations3 = exceeds expectations (second semester only)

Page 13: Competency Based Training

Assessing Intern CompetenciesAssessing Intern CompetenciesItem ExamplesItem Examples

AssessmentAssessment– Demonstrates effective clinical interviewing Demonstrates effective clinical interviewing

skills with children and familiesskills with children and families– Demonstrates ability to effectively collect and Demonstrates ability to effectively collect and

integrate data from multiple sources and/or integrate data from multiple sources and/or disciplinesdisciplines

Page 14: Competency Based Training

Assessing Intern CompetenciesAssessing Intern Competencies

ResearchResearch– Effectively searches for and applies most Effectively searches for and applies most

relevant empirical findings to inform clinical relevant empirical findings to inform clinical practicepractice

– Demonstrates skills in developing research Demonstrates skills in developing research designsdesigns

Page 15: Competency Based Training

Intern SelectionIntern Selection

Develop screening method/instrument Develop screening method/instrument which assists us in identifying interns who which assists us in identifying interns who are a strong match based on our mission are a strong match based on our mission and valuesand values– Across functional and foundational Across functional and foundational

competency domains, our blueprintcompetency domains, our blueprint

Page 16: Competency Based Training

Description of Screening Description of Screening InstrumentInstrument

Iterative process, revisions, & trainingIterative process, revisions, & training13 items total, anchored Likert scale (1-4)13 items total, anchored Likert scale (1-4)Breath of clinical training (5 items), across:Breath of clinical training (5 items), across:– Systems of care Systems of care (e.g., clinic, hospital, community, school)(e.g., clinic, hospital, community, school)

– Treatment modalities Treatment modalities (e.g., individual, group, family)(e.g., individual, group, family)

– Age/demographics Age/demographics (e.g., infants, toddlers, school age, (e.g., infants, toddlers, school age, adolescents)adolescents)

– Presenting diagnoses Presenting diagnoses (e.g., mental health, medical)(e.g., mental health, medical)

– Clinical activities Clinical activities (e.g., assessment, intervention, consultation)(e.g., assessment, intervention, consultation)

Page 17: Competency Based Training

Description of Screening ToolDescription of Screening Tool

Depth of clinical training (3 items)Depth of clinical training (3 items)– Amount/type of clinical supervisionAmount/type of clinical supervision– Direct treatment hours (adults and children)Direct treatment hours (adults and children)– Number of comprehensive assessment Number of comprehensive assessment

reports (adults and children)reports (adults and children)

Page 18: Competency Based Training

Description of Screening ToolDescription of Screening Tool

““Big Five” items (Foundational)Big Five” items (Foundational)– Developmental-ecological modelDevelopmental-ecological model– Diversity effectivenessDiversity effectiveness– Interdisciplinary collaborationInterdisciplinary collaboration– Research potentialResearch potential– Leadership potentialLeadership potential

Page 19: Competency Based Training

The Children’s Hospital of PhiladelphiaThe Children’s Hospital of PhiladelphiaPsychology Internship Training ProgramPsychology Internship Training Program

Pre-Doctoral Internship Selection – Screening Form Pre-Doctoral Internship Selection – Screening Form (Class of 2008-2009)(Class of 2008-2009)

1.1. Rate Rate breadth of training…across multiple systems of breadth of training…across multiple systems of care care (e.g.,mental health, medical, school, and community (e.g.,mental health, medical, school, and community systems)systems)

44 1 or more training experiences within 4 systems of care1 or more training experiences within 4 systems of care3 3 1 or more training experiences within 3 systems of care1 or more training experiences within 3 systems of care2 2 1 or more training experiences 2 systems of care1 or more training experiences 2 systems of care

1 1 1 training experience within 1 system of care1 training experience within 1 system of care

Page 20: Competency Based Training

with regard to various with regard to various treatmenttreatment modalities (e.g., individual modalities (e.g., individual therapy, family therapy, multiple family therapy, parent training, therapy, family therapy, multiple family therapy, parent training, child group intervention {i.e. classroom-based} and group therapy)child group intervention {i.e. classroom-based} and group therapy)

44 Training experience in 4 or more treatment Training experience in 4 or more treatment modalitiesmodalities

33 Training experience in 3 different treatment Training experience in 3 different treatment modalitiesmodalities

22 Training experience in 2 different treatment Training experience in 2 different treatment modalitiesmodalities

11 Training experience in only 1 treatment modalityTraining experience in only 1 treatment modality

Page 21: Competency Based Training

6. 6. Rate level of ethnic, linguistic, and culturalRate level of ethnic, linguistic, and cultural diversity awareness and diversity awareness and effectivenesseffectiveness, taking into consideration experience in and commitment to learn , taking into consideration experience in and commitment to learn in this area as demonstrated through the following domains: research, in this area as demonstrated through the following domains: research, coursework, teaching experiences, seeking practicum placements, case coursework, teaching experiences, seeking practicum placements, case conceptualization, and committee membership.conceptualization, and committee membership.

44 Diversity effectiveness is a clear, defining priority, guiding graduate Diversity effectiveness is a clear, defining priority, guiding graduate training, as evidenced in more than 3 of the above domainstraining, as evidenced in more than 3 of the above domains

33 Diversity effectiveness informs graduate training and is important as Diversity effectiveness informs graduate training and is important as demonstrated in at least 3 of the above domainsdemonstrated in at least 3 of the above domains

22 Diversity effectiveness is a part of graduate training as Diversity effectiveness is a part of graduate training as demonstrated by awareness and exposure in at least 2 of the above demonstrated by awareness and exposure in at least 2 of the above domainsdomains

11 Diversity effectiveness is reflected in only 1 domain or is not evident Diversity effectiveness is reflected in only 1 domain or is not evident in trainingin training

Page 22: Competency Based Training

8. 8. Rate research potentialRate research potential in child-oriented psychology in child-oriented psychology. . 44 Extramural pre-doctoral grant and 1+ first author peer-Extramural pre-doctoral grant and 1+ first author peer-

reviewed journal publication (excluding submitted for reviewed journal publication (excluding submitted for publication)publication)

22 1 first author publication (excluding submitted for publication) 1 first author publication (excluding submitted for publication) or a pre-doctoral grant; plus at or a pre-doctoral grant; plus at least 2 national poster or least 2 national poster or paper presentations as a first authorpaper presentations as a first author

22 1 or more publications (including in press) on which applicant 1 or more publications (including in press) on which applicant is not the first author; plus 1 or more national poster or paper is not the first author; plus 1 or more national poster or paper presentationspresentations

11 No publications; may have some national poster or paper No publications; may have some national poster or paper presentationspresentations

Page 23: Competency Based Training

Screening ProcessScreening Process

Administrative screening: ~210 Administrative screening: ~210 applications/yearapplications/yearEach of ~190 applications reviewed by 2 Each of ~190 applications reviewed by 2 psychology staff using screening psychology staff using screening instrument instrument (early Nov-mid Dec)(early Nov-mid Dec)Total score obtainedTotal score obtainedPreliminary rank order determinedPreliminary rank order determinedHand reviewed by TDHand reviewed by TD35 on -site interviews35 on -site interviews

Page 24: Competency Based Training

Psychometrics:Psychometrics:Construct ValidityConstruct Validity

Factor Analysis—Reviewer 1Factor Analysis—Reviewer 1– N = 382N = 382– 2 factor solution2 factor solution– Factor 1 alpha = .69 (acceptable)Factor 1 alpha = .69 (acceptable)– Factor 2 alpha = .70 (acceptable)Factor 2 alpha = .70 (acceptable)Factor Analysis—Replication (Reviewer 2)Factor Analysis—Replication (Reviewer 2)– N = 382N = 382– 2 factor solution2 factor solution– Factor alphas .69 and .72, respectivelyFactor alphas .69 and .72, respectivelyFactor 1 & 2 congruence coefficients = .94 & .98 Factor 1 & 2 congruence coefficients = .94 & .98 (good to excellent factor similarity between sets of raters)(good to excellent factor similarity between sets of raters)F1 and F2 F1 and F2 r r = .44 (moderate)= .44 (moderate)

Page 25: Competency Based Training

Pattern MatrixPattern MatrixItem 1 2Interdisciplinary collaboration .70 -.08

Developmental ecological model .68 .01

Diversity effectiveness .50 -.03

Leadership effectiveness .48 -.00

Research potential .42 -.01

Practicum hours .39 .10

Supervision quality .17 .13

Number assessment reports .15 .02

Number intervention cases -.13 .69

Breath across age groups -.00 .65

Breath across treatment modalities .02 .62

Breath of clinical activities .27 .41

Systems of care training .22 .24

Page 26: Competency Based Training

ReliabilityReliability

Alphas:Alphas:– Factor 1 (Reviewers 1 & 2) = .69 &.72Factor 1 (Reviewers 1 & 2) = .69 &.72– Factor 2 (Reviewers 1 & 2) = .70 & .70Factor 2 (Reviewers 1 & 2) = .70 & .70– ““acceptable”acceptable”

Inter-rater reliability, factor levelInter-rater reliability, factor level– F1 = .62F1 = .62– F2 = .61F2 = .61– ““marginal”marginal”

Page 27: Competency Based Training

Results suggest:Results suggest:

• Two factors makes sense—capture Two factors makes sense—capture breath of clinical experience and breath of clinical experience and foundational items foundational items (but still only capture (but still only capture 27.4% of variance)27.4% of variance)

• Use inter-rater item correlationsUse inter-rater item correlations

Page 28: Competency Based Training

Foundational or Functional?Foundational or Functional?

Page 29: Competency Based Training

ConclusionsConclusions

Attempt to operationalize core Attempt to operationalize core competencies—competencies—step in the right directionstep in the right direction

Reflects our mission, our unique identityReflects our mission, our unique identityOngoing process, iterativeOngoing process, iterativeScreening tool:Screening tool:

Clarify items: better anchor response choicesClarify items: better anchor response choicesIncrease training (e.g., develop scoring manual with Increase training (e.g., develop scoring manual with examples)examples)


Recommended