6/6/19 1 Quality Assurance Collaboration and Innovation Mark LaCelle-Peterson, President/CEO Association for Advancing Quality in Educator Preparation AAQEP Workshop Mount Saint Mary College Newburgh, New York June 7, 2019 Collaboration and Innovation in Quality Assurance 1. Welcome and Introductions 2. Overview: AAQEP’s Origins and Commitments 3. Standard 1: Candidate/Completer Performance: Aspects and Evidence 4. Evidence Base: Expectations for Standards 1 and 2 5. Standard 2: Completer Professional Competence and Growth: Aspects and Evidence Lunch — Clear the Parking Lot 6. Standard 3: Program Capacity Fundamentals: Process Documentation and Improvement Break — 7. Standard 4: Program Engagement: Setting Agendas and Engaging Partners 8. Closing Thoughts and Questions.
Transcript
6/6/19
1
Quality AssuranceCollaboration and Innovation
Mark LaCelle-Peterson, President/CEOAssociation for Advancing Quality in Educator Preparation
AAQEP WorkshopMount Saint Mary CollegeNewburgh, New YorkJune 7, 2019
Collaboration and Innovation in Quality Assurance1. Welcome and Introductions2. Overview: AAQEP’s Origins and Commitments3. Standard 1: Candidate/Completer Performance: Aspects and Evidence4. Evidence Base: Expectations for Standards 1 and 25. Standard 2: Completer Professional Competence and Growth: Aspects and Evidence
Lunch — Clear the Parking Lot6. Standard 3: Program Capacity Fundamentals: Process Documentation and Improvement
Break —7. Standard 4: Program Engagement: Setting Agendas and Engaging Partners8. Closing Thoughts and Questions.
6/6/19
2
Parking lot(s)
1. What questions do you have about evidence that the standards are met?
2. What questions do you have about the accreditation process?
3. What questions do you have about (or for) AAQEP?
Accreditation as a Professional Conversation
Accreditation: a profession’s conversation with internal and external stakeholders on key questions about quality:
QUA
LITY
6/6/19
3
43 individuals:
v34 faculty, deans, coordinators
v2 PK-12 educators
v4 reps from 3 SEAs (HI, MI, UT)v3 AAQEP initial staff
v39 institutions in 14 states
Working in three groups: Expectations, Process, Consistency
5
What’s new? Renewing a conversation, 2017
What informed this renewed conversation?
• Regulatory change: post-NCLB, post-RttT, post-Title II Regulation
• Evidence improvement: direct performance measures
• Emphasis change: respect for state/local contexts (e.g. CHEA)
• Strategy change: collaboration among providers, states, accreditor
• Orientation change: responsive innovation required to meet needs
6
6/6/19
4
Association’s Vision and
Mission
VISIONExcellent, effective, innovative educator preparation that is committed to evidence-based improvement, engages with the P-20 system, and holds high public confidence.
MISSIONTo promote and recognize quality educator preparation that strengthensthe education system’s ability to serve all students, schools, and communities.
Association’s Operating Principles
• Promote provider collaboration• Focus on improvement and innovation• Partner with providers and state authorities• Recognize the value and significance of
context and mission• Serve all providers and all types of programs
equitably• Seek efficiencies everywhere• Share ideas and innovations broadly
6/6/19
5
Quick pace: milestones in AAQEP’s growth• 2015 - 2017: Desire for option expressed by institutions & states• July 2017: AAQEP Board convened to incorporate• August-September 2017: Working Groups drafted framework• October-December 2017: Framework circulated for feedback• January 2018: Standards finalized and posted on aaqep.org• May 2018: New York State Board of Regents opened option• Spring 2019: Nine site visits completed in 8 states (MN had 2!)• May 2019: Five accreditation decisions completed• May 2019: First state agreement signed (others in the works)• Summer 2019: (planned): Reflection/refinement w stakeholders 9
AAQEP Now: 80 Members, 9 site visits completed
10
Members in 14 states,Guam, & Canada
Diverse membership:• Initial• Advanced• Traditional• Alternative • Online• Large, medium, &
small• Public• Independent• Proprietary• Community colleges
AAQEP’s Four Standards1. Program completers perform as professional educators with the
capacity to support success for all learners.
2. Program completers adapt to working in a variety of contextsand grow as professionals.
3. The program has the capacity to ensure that its completers meetstandards 1 and 2.
4. Program practices strengthen the P-20 education system in light of local needs and in keeping with the program’s mission.
14
6/6/19
8
Candidate/Completer Performance: Standards 1 & 2Confidence that completers perform as effective educators and continue to grow
Key Questions for Standards
1 and 2
1. Program completers perform as professional educators with the capacity to support success for all learners.
At the end of the program, are completers ready to fill their target professional role effectively?
2. Program completers adapt to working in a variety of contexts and grow as professionals.
Were completers prepared to work in diverse contexts, have they done so successfully, and are they growing as professionals?
6/6/19
9
Standard 1 — Candidate/Completer Performance
Program completers perform as professional educators with the capacity to support success for all learners
Candidates and completers exhibit the knowledge, skills, and professional dispositions of competent, caring, and effective professional educators. Successful candidate performance requires knowledge of learners, context, and content. Candidates demonstrate the ability to plan for and enact and/or support instruction and assessment that is differentiated and culturally responsive.
17
Standard 1Candidate/Completer Performance: Aspects and Evidence
6/6/19
10
Standard 1 — Candidate/Completer Performance
Program completers perform as professional educators with the capacity to support success for all learners
Candidates and completers exhibit the knowledge, skills, and professional dispositions of competent, caring, and effective professional educators. Successful candidate performance requires knowledge of learners, context, and content. Candidates demonstrate the ability to plan for and enact and/or support instruction and assessment that is differentiated and culturally responsive.
19
Standard 1 — Candidate/Completer PerformanceEvidence for this standard will include:•multiple measures, •multiple perspectives, including
program faculty, p-12 partners, program completers, graduates’ employers,
• direct measures, and evidence of • performance in clinical setting appropriate to program
Note that the evidence includes in-program & post-program measures.20
6/6/19
11
Aspects of Standard 1 (dimensions of the whole)
21
Evidence of Completer Performance includes
Content/pedagogical/professional knowledge (relevant to license)
Learners / learning theory, including SEL
Cultural competence
Assessment and data literacy
Positive learning/work environment
Professional dispositions/behaviors
Each standard includes six aspects of performance or practice that must be part of the evidence set. Evidence for each standard is evaluated holistically.
Standard 1 — Candidate/Completer PerformanceEvidence for this standard will include:•multiple measures, •multiple perspectives, including
program faculty, p-12 partners, program completers, graduates’ employers,
• direct measures, and evidence of • performance in clinical setting appropriate to program
Note that evidence includes in-program & post-program measures and that data quality will be investigated and documented in a variety of ways! 22
6/6/19
12
Standard 2 — Completer ProfessionalCompetence and Growth
Program completers adapt to working in a variety of contextsand grow as professionals.
Program completers engage in professional practice in educational settings and show that they have the skills and abilities to do so in a variety of additional settings and community/cultural contexts.
For example, candidates must have broad and general knowledge of the impact of culture and language on learning, yet they cannot, within the context of any given program, experience working with the entire diversity of student identities, or in all types of school environments.
23
Aspects of Standard 2 (dimensions of the whole)Evidence of Completer Professional Competence and Growth
Engage local school and cultural community, caregivers and familiesCulturally responsive practice with diverse learnersCan develop productive learning environments in diverse contextsSupport their P12 students’ increasing global perspectivesGrow professionally throughout their careersCollaborate for professional learning with colleagues
24
Each standard includes six aspects of performance or practice that must be part of the evidence set. Evidence for each standard is evaluated holistically.
6/6/19
13
Standard 2 — Completer ProfessionalCompetence and Growth
Evidence for this standard will show that program completers:
• have engaged successfully in relevant professional practice,
• are equipped with strategies and reflective habits that enable them to serve effectively in a variety of educational settings appropriate to the credential or degree sought,
• have, in fact, done so—using whatever evidence can be gathered.25
Thinking About EvidencePotentially Overlapping Evidence: Expectations for Standards 1 and 2
6/6/19
14
Evidence Requirements and PrioritiesMultiple measures with reasonable continuity
Quality of evidence must be investigated and shared
Priority is given to direct performance measures
Down-stream data is considered for improvement purposes
Differentiate evidence by license, level, location, delivery
Assessments appropriate to specific programs
Standard 1: Questions and AnswersProgram completers perform as professional educators with the capacity to support success for all learners.
At the end of the program, were completers ready to fill their target professional role effectively? What evidence supports this claim?
Multiple measures across the standard, preferably per aspectMultiple perspectives for standard, not all perspectives per aspect
(Perspectives: program faculty, P12 partners, completers, employers)
Direct performance measures, including in the final field experience 28
6/6/19
15
Engagement in groups
Standard 1 Aspects:
Content/pedagogical/professional knowledge (relevant to certificate/license)
Learners / learning theory, including SEL — RED
Cultural competence — GREEN
Assessment and data literacy — YELLOW
Positive learning/work environment — BLUE
Professional dispositions/behaviors
Standard 1 – Aspects
1. What measures inform you about this aspect?2. What new options might you consider?3. What perspectives are/aren’t represented?4. What particular measurement challenges do you note?
Table dialogue:• Select a presenter• Whole group presentations to follow
6/6/19
16
Describe your practices in groups: Standard 1Consider evidence requirements:• Multiple measures overall• Multiple perspectives across the evidence set for the Standard• Which perspectives have you captured?
• Program faculty• P12 Partners• Completers• Completer’s Employers
• Direct measures of performance including in appropriate field setting
Standard 2Completer Professional Competence and Growth: Aspects and Evidence
6/6/19
17
Standard 2 — Completer ProfessionalCompetence and Growth
Program completers adapt to working in a variety of contextsand grow as professionals.
Program completers engage in professional practice in educational settings and show that they have the skills and abilities to do so in a variety of additional settings and community/cultural contexts.
For example, candidates must have broad and general knowledge of the impact of culture and language on learning, yet they cannot, within the context of any given program, experience working with the entire diversity of student identities, or in all types of school environments.
33
Aspects of Standard 2 (dimensions of the whole)Evidence of Completer Professional Competence and Growth
Engage local school and cultural community, caregivers and familiesCulturally responsive practice with diverse learnersCan develop productive learning environments in diverse contextsSupport their P12 students’ increasing global perspectivesGrow professionally throughout their careersCollaborate for professional learning with colleagues
34
Each standard includes six aspects of performance or practice that must be part of the evidence set. Evidence for each standard is evaluated holistically.
6/6/19
18
Standard 2 — Completer ProfessionalCompetence and Growth
Evidence for this standard will show that program completers:
• have engaged successfully in relevant professional practice,
• are equipped with strategies and reflective habits that enable them to serve effectively in a variety of educational settings appropriate to the credential or degree sought,
• have, in fact, done so—using whatever evidence can be gathered.35
Two questions, some common dataAt the end of the program, are completers ready to fill their target professional role effectively? What evidence supports this claim?
What’s needed? Direct evidence of performance in program and internship/field placements; data from multiple perspectives—including faculty, P12 partners, completers, and their employers.
Were completers prepared to work in diverse contexts? Have they done so successfully? Are they continuing to grow as professionals? What evidence supports this claim?
What’s needed? Evidence from preparation and from practice.36
6/6/19
19
Making the case for Standards 1 and 2 may draw on some of the same evidence sources to answer distinct questions
• Evidence from program measures• Direct performance in clinical setting• Completers’ and their employers’ views
Completer performance
• Base provided in concepts and strategies in program, and experienced in practice
• Application, adaptability, growth throughout professional experience
Professional growth
Engagement in groups
Standard 2 Aspects
RED — Engage local school & cultural community, caregivers & families
RED — Culturally responsive practice with diverse learners
GREEN — Can develop productive learning environments in diverse contextsYELLOW — Support increasing global perspectives in their P12 students
BLUE — Grow professionally
BLUE — Collaborate with others for professional learning
6/6/19
20
Standard 2 – Aspects
1. How do you currently measure this aspect?2. What new options might you consider?3. What particular measurement challenges do you note?
Table dialogue:• Select a presenter• Whole group presentations to follow
Parking lot(s)
1. What questions do you have about evidence that the standards are met?
2. What questions do you have about the accreditation process?
3. What questions do you have about (or for) AAQEP?
6/6/19
21
Lunch…and conversation
It’s never too early to start planning ahead!
6/6/19
22
Follow AAQEP on Social Media
twitter.com/aaqep1
linkedin.com/company/aaqep
Process elementsCohorts, proposals, writing the Quality Assurance Report, planning for the visit
6/6/19
23
Review process is familiar• Self-study based on standards (Quality Assurance Report)• Off-site review completed 2 months prior to visit• Provider meets team and fields questions for 30 minuts
• On-site review by peer review team of reasonable size• Team sizes vary from 3 to 5 (or more in complex cases)• Each team includes a local practitioner to ensure understanding
• Opportunity to provide factual corrections to team report
Decision process is familiar• Accreditation decision made by the Accreditation Commission• For transparency and accuracy, Provider observes entire session
• Full term of accreditation is 7 years• Problems (smaller and larger) will be noted• ‘Concerns’ and ‘Conditions,’ respectively
• Shorter term (2 years) will signal significant problems• ‘Conditions’ on more than one standard lead to 2-year term• Full or focused review required to ‘clear’ conditions
• Denial (or revocation) for failure on one or more standards
6/6/19
24
Process Innovation: Cohorts & Liaisons Ensuring Communication, Facilitating Collaboration
Providers participate in a Cohort of 10-15 members based either on site visit date or their own choice (program size, mission, location, existing networks)
Each cohort is assigned two Liaisons (AAQEP staff members) who facilitates monthly calls and provides assistance throughout the process.
Process Innovation: the Proposal What’s in the proposal?• Overview of the provider and its context• Identification of assessments linked to aspects of standards 1 & 2• Explanation of how validity, reliability, fairness will be established• Description of innovations/ contextual challenges
How is the proposal reviewed and what role does it play?• Peer reviewers provide 1 or 2 rounds of feedback
• AAQEP reviews final proposal for completeness• Proposal and summary of feedback becomes part of case record
48
6/6/19
25
Program practices: Standards 3 &4Confidence that quality will be sustained, enhanced, improved
Accreditation’s two larger
decision questions
• Is there credible and sufficient evidence of program quality?
• Is there sufficient evidence of capacity and commitment to conclude that the level of quality will persist?
6/6/19
26
Key Questions for Standards
3 and 4
3. The program has the capacity to ensure that its completers meet standards 1 and 2.
Does the program have the capacity (internal & with partners) to ensure that completers are prepared and succeed professionally?
4. Program practices strengthen the P-20 education system in light of local needs and in keeping with the program’s mission.
Is the program engaged in strengthening the education system?
Standard 3Program Capacity Fundamentals: Process Documentation and Improvement
6/6/19
27
Standard 3 — Quality Program PracticesThe program has the capacity to ensure
that its completers meet standards 1 and 2.
Preparation programs ensure that candidates, upon completion, are ready to engage in professional practice, to adapt to a variety of professional settings, and to grow throughout their careers. Effective program practices include: consistent offering of coherent curricula; high quality, diverse clinical experiences; dynamic, mutually-beneficial partnerships with stakeholders; and comprehensive and transparent quality assurance processes informed by trustworthy evidence. Each aspect of the program is appropriate to its context and to the credential or degree sought.
53
Aspects of Standard 3Evidence of Quality Program Practices will include
Coherent curriculum (part of Appendix C)
Quality clinical experiences
Stakeholder engagement (part of Appendix D)
Admission/monitoring process linked to success (Appendix A)
Capacity for quality and institutional commitment (Appendix C)
54
Each standard includes six aspects of performance or practice that must be part of the evidence set. Evidence for each standard is evaluated holistically.
6/6/19
28
Standard 3 — Quality Program Practices
Evidence related to this standard will include documentation of program practices and resources as well as the program’s rationale for its structure and operation.
Documentation of some aspects of Standard 3 is presented in specific Appendices to the Quality Assurance Report (self-study)
55
Program Practices: Focused AppendicesAppendices describe and document key program practices:A: Candidate recruitment, selection, and monitoring (Standard 3)B: Completer support and follow-up (Standard 4)C: Capacity: faculty, resources, institutional commitment (Standard 3)D: Internal Audit: describe/test internal quality control system (Stnd 3)E: Jurisdictional obligations specified in state agreement (IF ANY)F: Missional commitments and distinct contributions (Standard 4)G: Data quality: reliability, validity, fairness, trustworthiness (Standard 3)
56
6/6/19
29
Admissions and Monitoring: Appendix AYour case study of candidate recruitment, selection, monitoring, success*
Describes the cycle of…• Recruitment/selection (admissions), • Monitoring/candidate support throughout the program and
Evaluates the…• effectiveness of the system with attention to the • appropriateness of the admissions criteria and candidate supports.
The report must be based on empirical evidence of the your own choosing.
* Success can include program completion or counseling to appropriate alternative 57
• Part of Standard 3: program practices that ensure outcomes
• Empirical process required but measures, criteria not mandated
Note: This matrix represents the substance of the Appendix. Both tables and narrative will be needed.
Quality Assurance Report: Appendix C
60
Institutional Commitment should be evident in parity for program
Capacity AspectPreparation Program
Institutional Norm or Comparable Program
Analysis of Difference
Curriculum
Faculty (ranks, load)
Facilities
Fiscal support
Student support
Student feedback
Note: This matrix represents the substance of the Appendix. Both tables and narrative will be needed.
6/6/19
31
Internal Audit: Appendix DHaving specified admissions and monitoring processes (Appendix A), described completer support and follow up efforts (Appendix B), and documented dimensions of program capacity (Appendix C)….Developing Appendix D invites you to describe the quality control processes by which you evaluate and improve program quality and to audit that system to ensure that it is working as intended.For example, a completer sample of 10 to 20 program graduates should suffice; these graduates’ records are ‘audited’ to learn whether their experiences match the program as designed.Results help shape the program’s improvement agenda 61
Appendix D: Internal Audit Map (sample)
62
6/6/19
32
Data Quality: Appendix GEvidence used to make the case for accreditation must be:• Valid—How have/might (content) validity be established?• Reliable and Reliably Collected—How are raters calibrated?• Fair & Address Bias—Do measures work well for all candidates?• Trustworthy— For qualitative data, how is analysis done
and how is consistency assured?
Evidence regarding assessments should be appropriate to type and to the purpose of the measure and the use to which data is put.
63
Data Quality: Appendix GFor all elements that are part of the evidence base, data quality characteristics must be considered. Questions to consider:1. Have you followed a formal process to establish validity,
reliability, fairness (bias mitigation), and trustworthiness?2. What process will you follow to engage program faculty as well
as internal and external stakeholders in evaluating instruments?3. Who will lead the team, and who will be on the team, to
investigate data quality characteristics?
Note: expectations regarding evidence should be reasonable.64
6/6/19
33
Standard 3 - AspectsIn small groups….
• Choose one main topic for table dialogue (Appendix A; Appendix D –Audit Maps; Proposal development and Process)• Choose second topic for table dialogue (in case you solve the first
challenge really quickly:)• Pick a diligent recorder and intrepid reporter
1. What have you learned about your own quality assurance processes?2. What new actions actions or strategies have you identified?
65
Digging In: Aspects of Program PerformanceSelect aspects of standard 3 to discuss:
1. Do admissions and monitoring processes lead to successful completion or transfer to ‘better fit’ program? (Evidence?) How can you study that meaningfully and usefully?
2. Who are your stakeholders, and how are they engaged in evidence-based continuous improvement efforts?
3. How do internal quality control processes work, and have they been investigated?
4. How does the program define and ensure quality of clinical experiences and the coherence of the curriculum? 66
6/6/19
34
Standard 4Program Engagement: Setting Agendas and Engaging Partners
Standard 4 — Program Engagement inSystem Improvement
Program practices strengthen the P20 education system in light of local needs and in keeping with the program’s mission.
The program is committed to and invests in strengthening and improving the education profession and the P20 education system. Each program’s context (or multiple contexts) provides particular opportunities to engage the field’s shared challenges and to foster and support innovation. Engagement with critical issues facing the field is essential and must be contextualized. Sharing results of contextualized engagement and innovation support the field’s collective effort to address education’s most pressing challenges through improvement and innovation.
68
6/6/19
35
Aspects of Standard 4
Evidence of Program Engagement in System Improvement
Engages stakeholders to support schools and reduce disparitiesSupports diverse educator workforce and address state and local needsSupports completer career entry and growth (Appendix B)Uses available evidence on completers for program improvementMeets relevant regulatory requirements (general, and Appendix E)Investigates effectiveness vis-à-vis institutional mission (Appendix F)
69
Each standard includes six aspects of performance or practice that must be part of the evidence set. Evidence for each standard is evaluated holistically.
Standard 4 — Program Engagement inSystem Improvement
Evidence for this standard addresses the identified issues in light of local and institutional context.
70
6/6/19
36
Digging In: Aspects of Program EngagementSelect one aspect of standard 4 to discuss:1. How do or might you engage with local schools to address local
challenges such as achievement disparities?2. How do or could you respond to local and state workforce needs
and help diversify the educator workforce?3. How do or might you support completers as they enter the
workforce and grow as professionals?4. How is evidence from completer performance used for program
improvement/innovation?71
Program Practices: Making the CaseEvidence that program practice standards are met is made both in sections of the quality assurance report and in key appendices.
Outline of Quality Assurance Report (the institutional self-study)
Introduction and Overview of the Program and the StudySection 1: the case for Standard 1: Completer Professional PerformanceSection 2: the case for Standard 2: Completer Adaptation and GrowthSection 3: the case for Standard 3: Quality Program PracticesSection 4: the case for Standard 4: Program Engagement in ImprovementAppendices A through G (see following slide)
72
6/6/19
37
Program Practices: Focused AppendicesAppendices describe and document key program practices:A: Candidate recruitment, selection, and monitoring (Standard 3)B: Completer support and follow-up (Standard 4)C: Capacity: faculty, resources, institutional commitment (Standard 3)D: Internal Audit: describe/test internal quality control system (Stnd 3)E: Jurisdictional obligations specified in state agreement (IF ANY)F: Missional commitments and distinct contributions (Standard 4)G: Data quality: reliability, validity, fairness, trustworthiness (Standard 3)
73
Completer Support and Follow-up: Appendix B•Describes services provided to completers to
ease their transition into the professional workplace as well as follow-up methods the program engages to monitor completer readiness and performance and provide ongoing support.
• Some providers have well-developed practices and collaborate with partners in the P-12 system to support and monitor their completers.
74
6/6/19
38
Digging In: Aspects of Program EngagementSelect one aspect of standard 4 to discuss:1. How do or might you engage with local schools to address local
challenges such as achievement disparities?2. How do or could you respond to local and state workforce needs
and help diversify the educator workforce?3. How do or might you support completers as they enter the
workforce and grow as professionals?4. How is evidence from completer performance used for program
improvement/innovation?75
Professional judgment and evidence
There are things that can be measured. There are things that are worth measuring. But what can be measured is not always what is worth measuring: what gets measured may have no relationship to what we really want to know. … And measurement [alone] may provide us with distorted knowledge—knowledge that seems solid but is actually deceptive. (Muller, 2018, p 3)
76
6/6/19
39
If what is actually measured is a reasonable proxy for what is intended to be measured, and if it is combined with judgment, then measurement can help practitioners to assess their own performance, both for individuals and for organizations. (Muller, 2018, p7)
77
A strong quality assurance system can assure accountability, support innovation
and improvement, and foster professional collaboration.
78
www.aaqep.org
6/6/19
40
Questions?
79
• Visit www.aaqep.org for updates and events• Questions or ideas? Contact us at:
