Designing a Cross Model Quality Assurance System:
Mission Impossible?1
Introductions
Julia HeanyPrincipal Investigator
Cynthia ZagarMHVQAS LeadHFA Reviewer
Jennifer TorresStudy Coordinator
Tiffany KostelecHome Visiting Unit Manager
2
Agenda for Today
Provide a broad understanding of the evidence base behind this work
Discuss the laws relative to this project
Share the story of the journey
Discuss the study and preliminary findings
Share initial thinking about what next steps are to make use of the tool moving forward
3
The Research Base: Our Guiding Light
4
Finding clarity: What’s QA & what’s QI & why do we need both?
Quality Assurance:Reviewing performance against a set of standards
Quality Improvement:Adjusting processes to better meet the needs of customers, both internal and external
Determines whether a program meets expectations for quality
Determines whether adjustments to a program result in improvements
Uses staff expertise to ensure the program provides quality services
Uses staff experience to improve the processes by which the program is implemented
Relies on external review and internal monitoring
Relies on internal teams
Responds to expectations established by external bodies
Responds to needs of customers and stakeholders
Expectation of MIECHV grant recipients Expectation of MIECHV grant recipients
Expectation of Michigan’s Home Visiting Law (PA 291)
5
Current QA & QI Activities
Gaps in quality assurance: Does not communicate consistent expectations for home visiting
program implementation from the State’s perspective Does not monitor the quality of home visiting implementation across
Michigan’s HV system Does not provide a foundation for quality improvement Does not meet legislative or funding requirements
Quality Assurance Quality ImprovementModel reviews MHVI Learning Collaborative
Contractual monitoring Participation in HV Collaborative Innovation and Improvement Network (CoIIN)
Benchmark reporting (under MIECHV) MHVI LIA, Local, and State CQI projects
Training & TA
6
The Law and You
7
Building a System that Supports Quality
Quality Assurance Expectations for MI are established through: Michigan’s Home Visiting Law - PA 291 of 2012
Defines the Home Visiting System as the “infrastructure and programs that support and provide home visitation.”
Built on the idea that “evidence-based programs have comprehensive home visitation standards that ensure high-quality service delivery and continuous quality improvement...”
Requires that state funded HV programs “operate with fidelity to the program or model.”
MIECHV Funding Requirements “States must provide a plan for ongoing monitoring of implementation
quality” Includes “State’s overall approach to home visiting quality assurance” State’s approach to “maintaining quality assurance”
8
The Journey of the Development of the Quality System
9
The Shock:10
Building a System that Supports Quality
To build an integrated approach to assuring and improving the quality of Evidence-Based and Promising Home Visiting Programs in Michigan: Formed a workgroup, including model representatives and
state partners Explored the evidence base and existing tools Received consultation from national & model experts Identified domains of quality in home visiting implementation Developed cross-model standards and measures under each
domain Developed a tool and process for assessing quality Received funding to field test the tool
11
Quality Domains
Selected domains based on indicators of quality and fidelity that were supported by the research and common across models
Domains include: Recruitment and Enrollment Home Visitor Caseloads Assessment of Family Needs and Referral to Services Dosage and Duration Home Visit Content Staff Qualifications and Supervision Professional Development Organizational Structure and Support
12
Quality Standards and Measures13
Quality Standards and Measures14
MHVQAS: Supplementary Materials
Model Specific Guidance Provides guidance to reviewers for measures where models have specified requirements.
Worksheets Reviewers – assists with reviewing site documentation prior to site visit Sites – assists with preparing data for site review
Document Checklist List of self-assessment materials to be sent prior to site visit
Document Gathering Questions Documentation of issues with gathering materials
Site Visit Document Review List of materials to have ready on day of site visit
Interview Questions Questions you will be asked during the interview portion of the site visit
15
Quality Report
On-site review summary – rating for each standard
16
Quality Report
• Ratings for each measure, including opportunities for improvement and/or special recognition
17
The Study Framework and Preliminary Findings
18
Study Questions
o Do the tool & procedure produce reliable and valid results? o Are the results similar when the tool is completed by different reviewers?o Do the tool & procedure really measure implementation quality?
o Can the tool & procedure be applied across models implemented in Michigan?
o What are the costs to the state and to local home visiting programs associated with preparing for and completing the review procedure?
o How can the results of the tool be used to improve quality in home visiting at the local and state levels?
19
Participants
Home visiting implementation experts (n=6) Provide feedback on whether the tool captures key concepts in quality
through an online survey
Reviewers (n=5) Conduct site reviews, track time and resources, and complete satisfaction
surveys
Home visiting sites (n=8) Participate in site reviews, coordinate home visit observations, provide
recent model review, track time and resources, and complete satisfaction surveys
20
Study Tools
Michigan’s Home Visiting Quality Assurance System Tool Implementation Expert Survey Home Visit Rating Scale Satisfaction Surveys Reviewer satisfaction Site satisfaction 6-month follow-up
Cost Tracking Tool Reviewer Site
21
Procedures
Training
• Reviewers• LIAs
Scheduling
• On site review• HOVRS
observations
Review Preparation
• LIA collects documentation
• Submit 3 weeks prior to review
On-site Review
• Documentation• Interview• Tour• HOVRS
completed within a month of the review
Quality Report
• Developed through reviewer consensus process
• Provided to LIA
• Phone conference to discuss
Cost Tracking
• Completed throughout process
• Submitted 1 week after review online
Surveys
• LIAs and reviewers -day after quality report is sent
• LIAs - 6 months after review
• Experts
22
Findings
Focus on research question #2 Can the tool and procedure be applied across models
implemented in Michigan? To what degree do model review guidance overlap with MHVQAS
standards? To what degree were model review findings consistent with MHVQAS
findings? Were there areas of the MHVQAS tool where models performed
differently? Did models differ in their perceptions of the MHVQAS tool’s reliability and
validity, or in their satisfaction with the tool? Did models differ in the cost of preparing for and completing the MHVQAS
process? What successes and challenges were associated with testing a cross-
model set of standards and measures?
23
To what degree do model review guidance overlap with MHVQAS standards?
Domains Standards MHVQAS MIHP HFA EHS PAT
Recruitment and Enrollment
Recruitment and Enrollment X X X X
CaseloadsCaseloads
X X X
Assessment of Need and Tailoring of Services
Assessment of Need and Referrals
X X X X X
Developmental ScreeningX X X X X
Care CoordinationX
Design of Services to Address Needs
X X X X
Goal Setting and Visit Planning X X X X
24
Domains Standards MHVQAS MIHP HFA EHS PAT
Dosage and Duration
Dosage X X X X X
Retention X X X
Family Exit and Transition Plans X X X X
Home Visit Content
Use of a Curriculum X X X X X
Nutrition Services X
Documentation of Home Visits X X X
Family Rights
Family Feedback X X X X X
Accommodations X X
Voluntary Services X
Consent X X
25
To what degree do model review guidance overlap with MHVQAS standards?
Domains Standards MHVQAS MIHP HFA EHS PAT
Staff Qualifications and Supervision
Staff QualificationsX X X X X
SupervisionX X X
Professional DevelopmentTraining
X X X
Organizational Structure and Support
Environmental Health & Safety X
InfrastructureX X X
Quality AssuranceX X X X
Integration into Service System X X X
Advisory GroupX X X X
26
To what degree do model review guidance overlap with MHVQAS standards?
To what degree do model standards overlap with MHVQAS standards?
There is not a perfect overlap between MHVQAS standards and any other model’s standard
The MHVQAS tool does not include any standards that are not included in at least one model
Take Home Message:The MHVQAS standards are comprehensive by design, overlapping
with and supplementing each of the other models reviewed.
27
To what degree were model review findings consistent with MHVQAS findings?
Model review findings were compared with MHVQAS findings to see if the two reviews found consistent areas of strength and opportunities for improvement Reviewed findings at the standard level rather than at the
measure level
BIG limitations: The reviews were conducted at different times, so there could
have been improvement or drift between each review EHS home based and center based are reviewed against the
same standards so a review comparison was not possible for EHS sites
28
To what degree were model review findings consistent with MHVQAS findings?
Model Number of standards that overlapped
Review 1 - # of standards with different findings
Review 2 - # of standards with different findings
MIHP 18 1 2HFA 20 2 7PAT 12 1 1
29
Take Home Message:There were some differences in ratings between reviews, but there were no
standards that were systematically different.
Were there areas of the MHVQAS tool where models performed differently?
Scores on each standard were compared by model to identify if some models performed better on some standards than others There was not a statistically significant difference in ratings on
any of the standards between different home visiting models Used a non-parametric test – Kruskal-Wallis H
Take Home Message:One model is not more likely to perform well on any part of the tool
than any other model.
30
Did models differ in their perceptions of the MHVQAS tool’s reliability and validity, or in their satisfaction with the tool?
Perceptions of reliability 13 items on a 1-6 scale focused on whether the tool could be applied
consistently (e.g., “The measures assessed are clear”) LIA Mean = 4.49 Differences between models were not statistically significant
Perceptions of validity 3 items on a 1-6 scale focused on if the tool was measuring what it
intended to measure (e.g., “The standards reflect key drivers of quality in home visiting”)
LIA Mean = 4.44 Differences between models were not statistically significant
31
Did models differ in their perceptions of the MHVQAS tool’s reliability and validity, or in their satisfaction with the tool?
Satisfaction 17 items on a 1-6 scale focused on the review process (e.g., “The
onsite review process was efficient”) LIA Mean = 4.81 Differences between models were not statistically significant
Take home message:Perceptions of the MHVQAS tool’s reliability and validity were positive, and satisfaction with the process was high, and reliability, validity, and
satisfaction did not differ by model.
32
Did models differ in the cost of preparing for and completing the MHVQAS process?
Site costs preparing for and conducting the review Mean = $2,722 Differences between models were not statistically significant
33
$2,685
$4,204
$2,958 $2,869
$2,291
$1,166
$3,052 $2,553
$0$500
$1,000$1,500$2,000$2,500$3,000$3,500$4,000$4,500
MIHP 1 MIHP 2 HFA 1 HFA 2 EHS 1 EHS 2 PAT 1 PAT 2
Cost of Preparing for and Completing the Process by Site (Total n=62)
Did models differ in the cost of preparing for and completing the MHVQAS process?
Most of the cost was in staff time spent preparing for the review Home visitors spent the fewest hours (mean = 6.05) Supervisors (mean = 22.83) and directors (mean = 23) spent the
most hours Differences by model were not statistically significant
34
10.6612.79 13.24
8.88 8.38
2.43
4.46 1.83
2.08 1.87
02468
101214161820
Total MIHP HFA EHS PAT
Mean Hours Staff Spent on the Review Process (Total n=68)
Mean hours preparing per staff Mean hours day of review
Did models differ in the cost of preparing for and completing the MHVQAS process?
There was a broad range in the amount of time spent on the total process by LIA Mean hours by LIA = 117.32, Min = 76.00, Max = 160.55 Differences between models were not statistically significant
Take Home Message:On average sites spent 117 total staff hours on the process and it cost
an average of $2,722 per site, and while some sites spent more time than others, there were not differences in time or money spent by
model.
35
What opportunities for making the MHVQAS work better across models were identified?
36
Scope: Balance achieving a comprehensive view of quality against the
burden of documentation Specify a requirement for each MHVQAS measure so that
expectations are clear for models without a requirement under a particular measure
Translation: Illustrate connections between MHVQAS standards and model
standards Identify alignment between MHVQAS documentation requirements
and program documents Clarify areas of the tool where there was more misalignment with
model requirements (e.g., advisory groups, tracking referrals, timeframes for data)
Use of the QA System Moving Forward
37
Michigan Home Visiting Initiative
Validation of the tool, once achieved: MHVI will incorporate this process into sub-recipient
monitoring of MHVI grantees to assist in assuring model fidelity as required by both MIECHV and PA 291 of 2012.
Allows MHVI to provide documentation of ongoing monitoring. Can be shared with EBHV Model Representatives.
LIAs will be trained on the process to ensure they understand all components.
MHVI will work with state partners to spread the use of the tool across models and funding streams.
38
Questions and Wrap-up39
Contact Information40