PENNSYLVANIA SYSTEM OF CARE PARTNERSHIP
Standard VIII: Evaluation and CQI
The PA System of Care Partnership is a cooperative agreement funded by the Substance Abuse & Mental Health Services Administration (SAMHSA).
Monica Walker Payne, M.A.Lead EvaluatorMarch 25, 2014
Monica Walker Payne, MA – Lead Evaluator
William McKenna, BS – Data Coordinator & Analyst
Thomas Hahner, MS – Project Coordinator and Interviewer for Crawford, Venango, and Fayette
Local Interviewers:› Wendi Buzzanco (Erie, Crawford, and Venango)› Amanda Clouse (York and Northumberland)› Jessica Keller (Montgomery and other East)› Edward McKenna (Chester and other East)› Jill Santiago (Philadelphia and other East)› Sue Soriano (Montgomery and other East)
English and Spanish speaking› Recovery Partnership (Lehigh)
6 interviewers
Oversight of the Evaluation Team by: Amy Herschell, Ph.D. Assistant Professor of Psychiatry and Psychology Director of Outcomes Research and Evaluation for Child Behavioral Health University of Pittsburgh
Meet the Evaluation Team!
Webinar Agenda
• Discussion of the Evaluation and Continuous Quality Improvement Standard
• Brief overview of the PA SOC Evaluation Requirements
• Introduction to the new PA County Progress Assessment
• Presentation and discussion of some public state and county data resources
• Additional questions and discussion
3
Evaluation and CQI Standard
5
A. The County Leadership Team, partnering agencies, and/or providers collaborate to gather data for the purpose of continuous quality improvement (CQI).
B. The County Leadership Team actively engages and participates in the implementation of a statewide System of Care evaluation and CQI plan.
C. As a part of the PA System of Care Partnership, the county consistently submits required data to the SOC Evaluation Team as mandated by the federal cooperative agreement with SAMHSA.
Evaluation and CQI Standard
6
D. The County Leadership Team provides opportunities for youth, family, system, and community leaders to participate in the Evaluation Subcommittee of the Youth and Family Training Institute Advisory Board.
E. The County Leadership Team works with the System of Care Evaluation Team and the Evaluation Subcommittee to develop and establish protocols for data sharing and analysis for ongoing CQI purposes.
Evaluation Subcommittee• Lisa Caruso – System/Provider Partner – Northumberland County
• Mark Durgin – System Partner – York County
• Dr. Gordon Hodas – System Partner – OMHSAS
• Andy Kind‐Rubin – Family Partner – Delaware County
• Corey Ludden – Youth Partner – Luzerne County
• Bryon Luke – System/Provider Partner – Montgomery, Delaware, Bucks Counties
• Gina Lutz – Youth/Provider Partner – Montgomery County
• Sandy Mantsch – System Partner – Erie County
• Maria Silva – Family Partner – Allegheny County
• Karan Steele – Family Partner – Westmoreland County
• Tim Truckenmiller – System/Provider Partner – Fayette County
7
If you are interested in joining please email me for details at [email protected] kick‐off event for the subcommittees is April 8, 2014 at the Child Welfare Resource Center in Mechanicsburg, PA from 10am‐3pm. Regular subcommittee meetings will be held via phone/webinar.
Evaluation and CQI Standard
8
F. The County Leadership Team collaborates with child‐serving systems to reduce barriers related to confidentiality and data sharing between systems.
G. The County Leadership Team collaborates with child‐serving systems to collect data related to cost‐effective services and supports.
Evaluation and CQI Standard
9
H. The County Leadership Team participates in the annual PA County System of Care Progress Assessment to monitor and assess the implementation of the PA System of Care Standards.
Two levels of data collection
• System Level Data
– Evaluates the progress the county has made in implementing the 8 standards of System of Care.
• Individual Youth and Family Level Data
– Evaluates individual outcomes, satisfaction, fidelity, and progress that youth and families experience because the county is implementing the System of Care standards.
– This is collected through the Youth and Family Services and Supports Planning Process that your county uses. For most counties, this is High Fidelity Wraparound.
Required Data Collection Information
• There are specific requirements for counties who are part of the first cooperative agreement using High Fidelity Wraparound
• There are specific requirements for counties who are part of the Expansion‐Implementation cooperative agreement using other approved Youth and Family Services and Supports Planning Processes
• Descriptions of required evaluation activities for both cooperative agreements can be found at:– http://www.pasocpartnership.org/resources/evaluation
11
Overview of Individual Youth and Family Level Data Collection
• TRAC‐NOMS Study for both SOC agreements– Brief Outcomes and Satisfaction Study
• Descriptive Study (EDIF information) for High Fidelity Wraparound– Demographic and Descriptive Information
• Longitudinal Outcome Study for High Fidelity Wraparound– Detailed Outcomes and Satisfaction Study
• WFI‐EZ Fidelity Monitoring for High Fidelity Wraparound– Fidelity, Satisfaction, and Outcomes Information
CQI Reports• Status of data collection statewide (for High Fidelity Wraparound):
Descriptive Study – 374 families
TRAC‐NOMS Outcomes/Satisfaction Study – 113 families
Longitudinal Outcomes/Satisfaction Study – 43 families
WFI‐EZ Fidelity/Satisfaction/Outcomes Study – 44 families
• State overall reports are updated in April, August, and December and can be requested by any SOC county for review and discussion
• Counties with 10 or more families in a study can request their own county reports as well as state overall reports for comparison
• Counties can request the Evaluation Team to attend a County Leadership Team meeting to present and discuss county, state, and national comparison reports
• In addition, we have High Fidelity Wraparound graduate survey summaries for 2012 and 2013.
13
PA County Progress Assessment
• The assessment will monitor progress made in each county in implementing the System of Care approach.
• Annual use of the tool can help a county to determine strengths, identify challenges, and show progress over time.
• It is intended to be used as a technical assistance tool both at the county level and for the PA System of Care Partnership staff to offer targeted technical assistance.
The survey will cover 5 main areas of SOC implementation:
1. Development and use of a strategic plan
2. Implementation of the PA System of Care Standards
3. County Services and Supports
4. County Infrastructure based on the System of Care approach
5. Commitment to the System of Care philosophy and approach
Who can take the survey?
The survey is completed by County Leadership Teams and other stakeholders involved in the System of Care planning and implementation:
Youth Partner Family Partner Policy and Decision Maker Partner (County Commissioner, County Council
Member, Human Services Administrator, Court Administrator, etc.) Provider Organization Partner (Director, Manager, Direct Service Staff, etc.) Managed Care Organization Partner (Behavioral Health MCO, Physical Health
MCO) Community Partner (YMCA, United Way, Big Brothers/Big Sisters,
Spiritual/Religious, etc.) System Partner (Mental Health, Child Welfare, Juvenile Justice, Education, Drug
and Alcohol, Physical Health, Individuals with Developmental Disabilities, etc.)
How many people per county should take it?
• The tool is designed for completion by at least five respondents per county.
• The minimum number and type of respondents is a youth partner, a family partner, and three system partners from the County Leadership Team.
• It is extremely important to capture the perspective of a wide variety and number of county stakeholders so we encourage you to invite anyone who has been involved with System of Care planning and/or implementation to participate.
• The ratings of all respondents are averaged together to determine the overall county scores.
When and how will we take the survey?
• The survey takes 30‐60 minutes to complete
• It is completed by stakeholders once per year in April (paper or online depending on individual county preference).
• You can access the survey in several ways:
– Through the PA SOC Partnership Website www.pasocpartnership.org
– You can request an email link from the Evaluation Team to distribute to county stakeholders
– You can receive a PDF file of the survey that you can print on your own
– You can request paper copies to be mailed to you along with postage paid envelopes for returning it to the Evaluation Team for data entry
Email me at [email protected] to request any of the last 3 options
Assessment Logistics
• The online survey will open on April 1st and close on April 30th of each year.
• You must log in to the site with an email address and password.
• We do not collect your name or any other personal information aside from your county and the type of partner that you are.
• You can log in and change your answers at any point during the month of April.
Website only
The PA County Progress Assessment Website only ‐ Registration
• E‐mail and password...
• County and Partner type…
The PA County Progress Assessment Website only – Main Menu
• Complete the assessment or change your password
The PA County Progress Assessment Sections to be completed:
Your answers are automatically saved when you complete each section. You will see an asterisk * when a section is completed and saved.
The PA County Progress Assessment II. Implementation of PA System of Care Standards
B. Youth Driven
• A “Don’t Know” response is not counted in the final score.
The PA County Progress Assessment Completing the Assessment
• When an asterisk appears next to every section completed, you will know that you are done.
• Your answers are automatically saved at the end of each section and there is no final “submit” button.
The PA County Progress Assessment Paper version ‐ Completing the Assessment
• When finished with the County Progress Assessment, please mail to:
Monica Walker Payne, M.A.Corporate One Office Park Building One, Suite 4384055 Monroeville Blvd.Monroeville, PA 15146
• If this is inconvenient for anyone, we can provide self‐addressed postage paid envelopes to a county contact person for distribution (email me at [email protected])
• We could also give you a link to a secure website where you can log in and upload a scanned form or set of scanned forms (PDF files) to us.
Reports• Counties will receive regular reports every June that include county data as well as overall PA data for comparison.
• Reports will be emailed to our county contacts for distribution.
• All individual ratings will be kept confidential and scores will only be reported as an aggregate.
Reports• If there is more than one respondent in each category (Youth Partner, Family Partners, System Partners, County Policy and Decision Maker Partners, Provider Partners, Community Partners, and/or Managed Care Partners) you will be able to receive summaries of each type of partner to compare perspectives.
• You will need to have a minimum of 7 respondentsto receive a report broken down by partner (2 youth, 2 family, and 3 system partners.)
State and County Data Resources
• The following slides will contain resources to find public data and reports from PA systems
• This document, called “PA County Data Resources” (with clickable links) can be found at:
www.pasocpartnership.org/resources/evaluation
33
United States Census Bureau
Access to national, state, county, and city data including:
– Population estimates and projections – Geographical mobility/migration – Age, race, ethnicity, language, education, income – Industry and occupation – Data visualizations and interactive maps
http://www.census.gov/main/www/access.html
SAMHSA:Substance Abuse and Mental Health Services Administration
Access to the latest national research, resources and reports through 2013
– Data on drug use/abuse (detailed list A to Z) – Drug use in correlation with mental health disorders
– Treatment data http://samhsa.gov/data/
2013 PA Behavioral Health Barometer Report: http://store.samhsa.gov/shin/content//SMA13‐4796/SMA13‐4796PA.pdf
KIDS COUNT Data Center City, county, state and national data/rankings on indicators of child well‐being
– Demographics– Economic Well‐being– Education– Family and Community– Health– Safety and Risky Behaviors
• Data available from 2008‐2012• Custom printable reports can be generated of any/all rankings and
indicators
http://datacenter.kidscount.org/
40
County Health Rankings and Roadmaps
Access to health data, rankings, and roadmaps (Data from 2010‐2013) for every county in the nation including:
– Premature deaths – Children in poverty – Access to safe drinking water – Rate of sexually transmitted diseases – Obesity – Number of fast food restaurants
http://www.countyhealthrankings.org/app/#/pennsylvania/2013/rankings/outcomes/overall/by‐rank
43
County Snapshot and PA Comparison of Health Factors
CountyErrorMargin PA National Trend Rank
(of 67)
City‐data.com
Community statistics (race, income, ancestries, education, language, religion, employment, geographical data, crime data, cost of living, housing, businesses, state profiles, etc.)
www.city‐data.com
Diversity Data
• Child demographic and wellbeing indicators (profiles, rankings, and maps) available to:– Analyze by race/ethnicity– Compare data across states, counties, cities, and large school districts
– Compare policy indicators across states• Demographics, Education, Neighborhoods, Health, Economic, Policy, etc.
• Neighborhood‐level Child Opportunity Index• Launched on March 12, 2014• www.diversitydatakids.org
51
Child Opportunity Index Lehigh / Northampton
53The Child Opportunity Index is a measure of relative opportunity across a metropolitan area calculated based on indicators of Educational Opportunity, Health and Environmental Opportunity, and Social and Economic Opportunity.
Office of Mental Health and Substance Abuse Services
Access to data from PA’s behavioral health system: – HealthChoices goals and performance reports – Statewide initiatives – Resources – County behavioral health data profiles – Performance outcomes
http://www.parecovery.org
PA Department of Public Welfare Child Welfare Services Data
• Annual Progress and Service Reports • Child Abuse Reports • Child Fatality / Near Fatality Reports
http://www.dpw.state.pa.us/forchildren/childwelfareservices/
PA Child Stat
Community Outcomes for Pennsylvania's Children and Families including:
– Uninsured children – Juvenile Crime Arrests – Child suicide – New child welfare placements
https://www.humanservices.state.pa.us/CommunityLevelOutcomeIndicators/PAChildStatFlex.html
58
Juvenile Court Judges Commission
Statistics about juvenile court dispositions and juvenile probation
http://www.jcjc.state.pa.us/portal/server.pt/community/statistics/5040
61
Pennsylvania Commission on Crime and Delinquency
County level crime data including: – Corrections– Courts– Delinquency Prevention– Juvenile Justice
http://pacrimestats.info/About.aspx#
PA Uniform Crime Reporting Statistics
Crime statistics sortable by county and jurisdiction
http://ucr.psp.state.pa.us/ucr/reporting/ruaware/ruawarecountyui.asp
Pennsylvania Department of Corrections
• Reports on Recidivism (2013 study just released)• Motivational Boot Camp• Annual Statistical Reports• State Intermediate Punishment Reports
http://www.cor.state.pa.us/portal/server.pt/community/researchstatistics/10669/reports/1069947
EPISCenter
Implementation and assessment of proven‐effective prevention and intervention programs
– Evidence‐Based Programs – training, resources, research, and tools of 11 programs
– Juvenile Justice ‐ Standardized Program Evaluation Protocol and other assessments
– Communities that Care and PA Youth Survey ‐information and resources
http://www.episcenter.psu.edu/
72
PA Department of Education
Access to all Pennsylvania public school statistics including:
– Dropout, enrollment, and graduation information – College‐bound projections and post‐secondary information – District, school, and state report cards – Early childhood information – PSSA, SAT and ACT scores – Home School Statistics
http://www.pde.state.pa.us/portal/server.pt/community/data_and_statistics/7202
PA Bureau of Special Education
PaTTAN (Pennsylvania Training and Technical Assistance Network)
– Training, Educational Initiatives, Resources, Publications http://www.pattan.net/
PennData – Early Intervention and Special Education data and reports by district and state overall http://penndata.hbg.psu.edu/index.aspx
Discussion
• What have you been doing in your county around evaluation and CQI?
• What are your biggest challenges with this standard?
• What additional tools or technical assistance do you need from our Evaluation Team?
• Any additional comments or questions that we can talk about today?
79
Please give us your feedback
• If you registered for this webinar, you will receive a link to a brief feedback form that you can take online via Survey Monkey
• We always appreciate your honest feedback about our events and feel free to add any comments and/or suggestions for the future.
• Thank you so much for participating!!
80
For additional information contact:
Monica Walker Payne, M.A.Lead EvaluatorPennsylvania System of Care PartnershipYouth and Family Training InstituteCorporate One Office Park – Building One, Suite 4384055 Monroeville Blvd., Monroeville, PA 15146Office: (412) 856‐2890 / 1‐866‐462‐3292 (Ext. 2)Cell: (724) 858‐9019Fax: (412) 856‐8790Email: [email protected]: www.pasocpartnership.org