Court Related Research in Maryland: Lessons Learned from Research Conducted with the Administrative Offices of the Courts. Llewellyn J. Cornelius, Ph.D.- University of Maryland, Ruth Young Center (RYC) David Crumpton, Ph.D. Administrative Office of the Courts - PowerPoint PPT Presentation
33
1 Court Related Research in Maryland: Lessons Learned from Research Conducted with the Administrative Offices of the Courts Llewellyn J. Cornelius, Ph.D.- University of Maryland, Ruth Young Center (RYC) David Crumpton, Ph.D. Administrative Office of the Courts Jeanne Bilanin, PhD. Institute for Governmental Service and Research (IGSR)- University of Maryland We would like to acknowledge both our collaborators in the Maryland Judiciary Research Consortium, our Colleagues here at the Ruth Young Center and our stakeholders across the state as such a complex endeavor would not have been possible without the concerted involvement of the many people who are part of this collaboration.
Transcript
1
Court Related Research in Maryland: Lessons Learned from Research Conducted with the Administrative Offices of the Courts
Llewellyn J. Cornelius, Ph.D.- University of Maryland, Ruth Young Center (RYC)David Crumpton, Ph.D. Administrative Office of the CourtsJeanne Bilanin, PhD. Institute for Governmental Service and Research (IGSR)- University of Maryland
We would like to acknowledge both our collaborators in the Maryland Judiciary Research Consortium, our Colleagues here at the Ruth Young Center and our stakeholders across the state as such a complex endeavor would not have been possible without the concerted involvement of the many people who are part of this collaboration.
2
Overview
It is safe to say that this is a different type of presentation as the focus of this presentation is more about the collaboration of a large number of organizations and partners in the process of conducting evaluation research, than it is about presenting findings from one study.
3
Overview
The following areas will be covered in this presentation: A short history of the collaboration process An overview of the roles of the collaborators An example of a cluster of projects that have
been supported under this collaboration Discussion of the challenges of balancing
scientific methods with ever changing needs in the policy arena.
4
Goals of the presentation
Expose the audience to a multi-campus, multi-discipline collaboration model
Highlight the challenges of balancing scientific methods with ongoing policy and programmatic issues.
Highlight the complexities of working with a large range of stakeholders.
5
History
The University of Maryland has been involved in a multitude of academic state partnerships including work with the Department of Human Resources and Department of Health and Mental Hygiene
6
History
Much of this work has been influenced by the vision of Judge Robert M. Bell, Chief Judge of the Maryland Court of Appeals who issued an challenge to the Maryland Judiciary in 1996 to increase its focus on public outreach. This had led to the development of wide range of programs and the development of inter-disciplinary teams that uses multiple paradigms to assess the delivery of equitable services across the state.
7
History
This vision had set the stage of programs that pre-date the involvement of the Ruth Young Center in Court based evaluations.
The RYC collaboration with the Administrative Office of the Courts began in 2005 with an evaluation of the Foster Care Court Improvement Program that was conducted by Bruce DeForge.
By the summer of 2010, this Collaboration had expanded to an evaluation of 13 programs by the RYC and some 40+ programs by the Maryland Judiciary Research Consortium
8
History
The Maryland Judiciary Research Consortium is comprised of the Ruth Young Center here at the School of Social Work, the University of Maryland Law School, the Institute for Governmental Service and Research at the University of Maryland College Park, the University of Baltimore, Salisbury University, Morgan State University, Coppin State University and Bowie State University.
9
History
The Ruth Young Center collaboration team included: Kieva Bankins, MSW Charlotte Bright, PhD Jacqueline Booth, MA Leigh Casey, BA Trenette Clark, Ph.D Clara Daining, Ph.D. Rebecca Sander,
PhD. Elizabeth Greeno,
PhD.
Corey Shdaimah, Ph.D.
Marie Bailey Kloch, MSW
Darnell Morris-Compton, MSW
Naeem Shaikh, MSW Ninoosh Sadeghi, BA Llewellyn Cornelius,
PhD. Bruce DeForge, Ph.D.
10
History
This presentation focuses on a subset of those evaluations: Evaluations of the continuum of
Truancy Programs across the State (the Truancy Reduction Pilot Program (TRPP) in the Lower Eastern Shore), the Truancy Court Program in Baltimore City and the BSMART (Baltimore Students: Mediation About Reducing Truancy) program in Baltimore City.
11
Collaborator Roles
MJRC Program evaluators Multi-disciplinary- social work,
law, public health, sociology, psychology, business administration
Multi-method research designs- Qualitative,, Quantitative Mixed methods, Community involved and Community Based Participatory Research.
Examine key program concepts, purposes and goals to determine what can be evaluated within a given scope of work.
Focus on what can and cannot be generalized within the limits of the social science paradigm
AOC Program Administration Connect evaluations to locate,
state and federal policy issues Use of stakeholders from the
judiciary to advise the RYC regarding the design and implementation of the project evaluations
Provide funding from the State of Maryland General Fund or through Federal grants.
12
Case Example: The Continuum of Truancy Research
The premise: The AOC expressed an interest in learning more about the processes that were involved in the administration of the continuum of Truancy Intervention/Prevention Programs across the state.
They were also interested in learning more about what factors contributed to the success of these interventions.
13
Case Example: The Continuum of Truancy Research
This led to development of a three year program evaluation (that is ending in December 2010) of the three types of truancy interventions.
This is a multi-institutional evaluation involving partners across the MJRC.
14
Case Example: The Continuum of Truancy Research
In particular they were interested in learning about three such programs: A one session school based mediation program
that focuses on school tardiness called BSMART Baltimore (Students: Mediation About Reducing Truancy)
A ten week early truancy intervention program called the Truancy Court Program (TCP) that is targeted to youth with 5-20 absences, and
A ninety day Truancy “Episode” based intervention called the Truancy Reduction Pilot Program (TRPP).
15
Case Example: The Continuum of Truancy Research
BSMART BSMART is operated by the University of Maryland
School of Law’s Center for Dispute Resolution (“C-DRUM”). Established in 2006, BSMART targets students who exhibit early patterns of five or more unexcused absences or instances of tardiness. BSMART program mediators work with students, parents and schools to improve communication and develop strategies in a unified effort to address factors that may be contributing to student truancy. Mediation is strictly voluntary and occurs only when the parent/guardian and teacher agree to participate. In conjunction with BCPSS, C-DRUM identified three schools as initial sites for the program.
16
Case Example: The Continuum of Truancy Research
TCP TCP is operated by the University of Baltimore
School of Law’s Center for Families, Children and the Courts (“CFCC”) at selected schools.
TCP involves weekly in-school sessions with a volunteer judge or master, a team of school representatives, a CFCC staff person, a law student, the student, and his/her family.
The model is based on an early intervention approach and targets students referred by the Baltimore City Public School System who are “soft” truants.
17
Case Example: The Continuum of Truancy Research
TCP TCP targets students who have from 5 to 20
unexcused absences – in the belief that this group still has academic, social, and emotional connections to the school.
TCP participation lasts for ten weeks and is strictly voluntary on the part of the student and his/her family. Every week, the judge or master reviews each TCP participant’s file, speaks privately with the child and his/her parent/caregiver about the student’s school attendance, and inquires about difficulties encountered during the week.
In addition to the team that meets weekly with the student and his/her parent/caregiver, CFCC has established a TCP Mentor Program and a TCP Volunteer Program.
18
Case Example: The Continuum of Truancy Research
TCP The TCP mentor program, coordinated by a
full-time consultant, provides mentors and role models for both students and family members. The mentor coordinator works with the students during the TCP session and conducts phone calls each week with parents/guardians.
TCP currently operates in eight Baltimore City public schools. Eleven Circuit and District Court judges and masters have volunteered to participate in TCP in Baltimore City. TCP is also expanding to other jurisdictions in Maryland.
19
Case Example: The Continuum of Truancy Research
TRPP (in the 1st Judicial Circuit) The TRPP was authorized and funded under the
provisions of Chapter 551 of the Acts of 2004 and House Bill 1443. TRPP is a court-based truancy reduction intervention in the juvenile courts. As a ‘problem solving court”, the TRPP includes a special document of truancy cases that are heard by the same master/judge each month. At these monthly hearings, factors contributing to truant behavior are identified through comprehensive family assessments and it is as a result of these assessments that students and their families are referred to community resources to address these factors. Students in this program are monitored by the court for a minimum of 90 days. Students who demonstrated improved school attendance during these 90 days are “graduated” from the program.
20
Case Example: The Continuum of Truancy Research
In general, the evaluations of these program focused on: Collecting data regarding school attendance, school
absences, school behavior, academic performance and juvenile justice experience- pre and post intervention.
Interviewing program administrators in the school system, the courts and the department of social services regarding the program operated and the factors they believed contributed to program success.
Interviewing participants and their families regarding their satisfaction with the services that they received, barriers to participation in the programs and perceptions regarding what worked or did not work in these programs.
David Crumpton
Note revised language
21
Case Example: The Continuum of Truancy Research
In general, the evaluations of these programs focused on: Collecting data which described the availability
of school and legal services on the community level to determine geographic barriers to obtaining services.
Interviews with senior program administrators to compare the intended goals of the programs vs. the actual program activities.
22
Case Example: The Continuum of Truancy Research
In general, evaluation methods included: Conducting focus groups and key informant
interviews to capture the views of stakeholders and senior administrators- using grounded theory to capture the patterns of themes that emerged from these interviews.
The administration of surveys to program participants to record their perceptions regarding program activities.
The abstraction/retrieval of school attendance and outcome data from the local school system, along with program participation data from the intervention site.
The abstraction, where relevant, of program data from the Department of Juvenile Services for youth that were adjudicated to DJS.
23
Case Example: The Continuum of Truancy Research
Detailed Example: The Process Evaluation of the Truancy Reduction Pilot Program in the 1st Judicial Circuit
The process evaluation (Daining, et al, 2008) focused on describing the extent to which the program has operated as intentioned. The evaluation focused on:
Describing the program including its goals, organization and responsibilities of program personnel.
Describing the characteristics of the youth who participated in the program.
Describing the services that were delivered in the program Presenting the perceptions of the stakeholders regarding what
did or did not work well in the program; and Describing the role of the courts in the TRPP operation.
24
Case Example: The Continuum of Truancy Research
Detailed Example: The Process Evaluation of the Truancy Reduction Pilot Program in the 1st Judicial Circuit Four data collection strategies were used for
this process evaluation: Administrative data relating to participant
demographics, participation rates and court proceedings,
Stakeholder interviews with court personnel, and personnel from collaborating community agencies
Observations of truancy court hearing in each county, and archival data review.
25
Case Example: The Continuum of Truancy Research
Detailed Example: The Process Evaluation of the Truancy Reduction Pilot Program in the 1st Judicial Circuit Using a semi-structured interview guide, a total
of 41 stakeholder interviews were conducted of judges and masters in the First Judicial Circuit, court administrators, family services departments personnel, offices of the clerk personnel, boards of education personnel, community service providers, DJS case management specialist and representatives from local management boards and health departments.
26
Case Example: The Continuum of Truancy Research
Detailed Example: The Process Evaluation of the Truancy Reduction Pilot Program in the 1st Judicial Circuit Administrative process data was abstracted
from a centralized Ms Access database, where data was routinely collected over the course of the program regarding the demographics of the program participants, the number of days between truancy petition file and adjudication hearings, the length of time the youth participated in TRPP, the types of services recommended by the court (type of disposition at the TRPP hearing) and the reason for closing the case.
27
Case Example: The Continuum of Truancy Research
Detailed Example: The Process Evaluation of the Truancy Reduction Pilot Program in the 1st Judicial Circuit Court observations: a total of 72
truancy hearings were heard across the four counties (Somerset, Dorchester, Worcester and Wicomico) including initial TRPP hearings, ongoing case hearing and final review hearings.
28
Case Example: The Continuum of Truancy Research
Detailed Example: The Process Evaluation of the Truancy Reduction Pilot Program in the 1st Judicial Circuit Archival data review: archival data
review included program reports, flow charts and TRPP procedural forms.
29
Case Example: The Continuum of Truancy Research
Detailed Example: The Process Evaluation of the Truancy Reduction Pilot Program in the 1st Judicial Circuit
Key Findings: 61 percent of the students who complete TRPP were deemed
successful as indicated by improved school attendance in compliance with court recommendations.
Successful students were younger at the time of the petition filing on average than unsuccessful students.
There was no statistically significant difference in the length of TRPP involvement between successful students and unsuccessful students.
Grade point averages of middle and high school students completer increased from the time of petition to case closure.
30
Lessons learned
The lenses of evaluation influences what is collected and the analyses that are conducted- Saying the primary outcome of a Truancy
reduction program is improvement in school attendance- leads to the construction of one type of evaluation- saying that it also leads to improvements in behavior with peers, or positive self esteem leads to the construction of an alternative evaluation design.
31
Lessons learned
Early involvement of stakeholders and program partners is critical in minimizing measurement error and observation bias in program evaluations. One of the challenges we faced was
differences in operating paradigms- the paradigm of science in the design and implementation of studies and the paradigm of the active policy and program development. A change in a program design in the midst of the study may increase its program relevance but limit its generalization.
32
Lessons learned
Early involvement of stakeholders and program partners is critical in minimizing measurement error and observation bias in program evaluations. A second challenge we worked through was
ensuring that the team was clear regarding what was being evaluated- as terminology across disciplines has different meanings- for example- mandated may have very specific definition from a legal perspective as it is tied to a particular ruling or law, whereas mandated from a social science perspective may reflect more general guidance and principles.
33
Lessons learned
Available Resources Govern what can be evaluated. All evaluations are governed by
balancing scientific precision (for example, in terms of striving for designs that can measure cause and effect, designing sampling strategies that lead to the generation of statistically significant findings) with the resources at hand.
