Information

TO: Juvenile Welfare Board Members

Item D Regular Meeting 9/9/99

FROM: James E. Mills,~xecutive Director~ Krista I. Win~A~ector, Quality Ma-;,-"\ement and Evaluation

Neighborhood Family Center ~-utcome Study- Interim Report

This'is the first of several reports that will be produced as products of the Neighborhood Family Center (NFC) outcome study that began in March, 1999. The purpose of the study is to design and implement an appropriate design to measure effects and benefits of implementing JWB's Neighborhood Family Center Initiative. The outcome study is conducted by Rowlette Research Associates. $40,000 each year is allocated to this project in FY 98-99 and in FY 99-00.

As required by the project proposal and research contract, this report lays the foundation for the study. It provides a historical context, describes the development of the Neighborhood Family Centers in relationship to the JWB Strategic Plan, and describes each of the existing and emerging Neighborhood Family Centers. Descriptive information includes demographics of participants, methods of implementing core services, operating principles or "theories of change", and governance. Social indicator data is provided for the ZIP codes from which the NFC's drew participants in 1997-1998. This report also provides a review of current research recommendations for measuring the impact of complex community initiatives. A framework and design is proposed for the implementation phase. Activities are identified at each center that show promise of yielding measurable results to be tracked through the JWB SAM IS (Services and Activity Management Information System). Recommended "next steps" include developing consistent working definitions for data collection with input from JWB and the NFC Directors, and consideration of a "results mapping" methodology described in the report.

This phase of the outcome study was intended to develop background and methodology for the collection and analysis of participant level and neighborhood level outcome and impact data. An assessment of currently available data for use in outcome analysis is in progress, and will be summarized in the FY 1998-99 end of year report. Result of the outcome analysis will not be available until 1999-00; however, findings during the design phase provide evidence that the NFC's are valued and supported by the communities in which they are located. Such evidence, included in the report, is indicative of community effectiveness. Examples are: financial and other support from municipalities, businesses and other community agencies, recognition and awards from peer organizations and civic groups, and collaborative relationships with a variety of community partners.

This report provides important information for NFC stakeholders, including NFC staff and boards, participants and community members, as well as JWB staff and board members. The information may be useful to others engaged in implementing neighborhood development activities. A copy of the Executive summary is attached. As recommended by the NFC outcome study planning and advisory committee and the NFC Coalition, a copy of the full report is available for each Board member.

Staff resource: Judith Simpson

, Executive Summary

JWB Board Meeting 9/9/99 Attachment to Information Item D

Introduction. This is the first interim report of the Neighborhood Family Center Outcome Study summarizing evaluation research activities conducted during the first 1 0-week period of Phase I. The first phase, March 23 to September 30, 1999, is centered on pre-evaluation planning research activities leading to the design of an overall framework for an outcome evaluation of the eight (8) established and three (3) emerging Neighborhood Family Centers. Phase 11--0ctober 1,1999 to September 30, 2000--involves the implementation and refinement of the evaluation plan and presentation of the first outcome data in the concluding report.

The report provides a detailed description of each of the Neighborhood Family Centers. The evaluability of the Neighborhood Family Center concept, variations in their philosophies of change, core services, neighborhood governance, programs offerings, achievement of measurable objectives, and evidence of their value in the community at large is examined and discussed. The requirements and challenges of evaluating community-based initiatives, the methodology standards, and the importance of interim outcomes from national applied research literature has been thoroughly reviewed and presented. An overall preliminary evaluation research design that also encompasses cross site measures has been proposed and discussed. Key components for both the historical and ongoing lessons learned are identified and the methods for collecting the stakeholders' perspectives and compiling the reports are explained. Recommendations relevant to Phase I are presented, and finally, the next evaluation research steps to conclude Phase I are identified .

Purpose. This study, among others, was developed as a result of a series of decisions following a referendum passed by Pinellas County voters in September 1990. In December 1992, The Juvenile Welfare Board (JWB) established a new direction, "Family Empowerment and Support," producing the "Neighborhood and Community Development Initiative 1991-1996." The plan provided for the development of seven Neighborhood Family Centers throughout Pinellas County between 1993 and 1996.

Background. Both the Community Development Plan and the Neighborhood Family Center initiative are grounded in a body of research about community initiatives that indicate a need to address the wholeness of individuals and family lives through integrated, comprehensive strategies. The approach takes into account the influence the neighborhoods have on the lives of children and families and upholds the premise that active participation of members of the neighborhood is a necessary ingredient for the kind of community change that can improve outcomes for children and families. An underlying assumption is that by strengthening families and communities, the conditions that contribute to poor youth outcomes, such as substance abuse, violence, and teen pregnancy, will be reduced, and that over time, community indicators related to youth outcomes will improve. The Neighborhood Family Center model is based on a model design found successful nationally. The

1

research on this approach has identified two predictors of success: the extent to which the core services are provided for the neighborhood; and the development of the level of local governance. These two key points have been made central in the evaluation framework design.

Evaluation Research Activities. Research activities during this period included historical record and document review, key informant interviews, review of national applied evaluation research, literature review of lessons learned in community-based initiatives, site visits to the Neighborhood Family Centers, and meetings with the Neighborhood Family Center Planning and Advisory Committee and the Neighborhood Family Center Coalition.

Objectives. The objectives of the evaluation research activities conducted during this first 1 0-week period of Phase I are:

• a description of each Neighborhood Family Center which includes the geographical neighborhood or community of interest served;

• variations in theories or philosophy of how they will change the lives of the participants, families, and the neighborhood;

• variations in how each center implements the nine (9) core services; • variations in other community priorities addressed by each center; • a description of related activities that are offered; and, • types of activities and data currently collected at each center that could be

used in an outcome study.

Development of Theories/Philosophy of Change. One of the most important tasks in the pre-evaluation planning stage is to identify how each Neighborhood Family Center proposes to change the lives of the families they serve. Once identified, these change strategies are then translated into the operating hypotheses from which measurable objectives can be developed and linked to interim outcomes. A theories­of-change approach to evaluation reduces, but does not eliminate problems associated with attributing outcomes to specific causes. This approach contends that the more the events predicted by the theory actually occur during the intervention, the more confidence evaluators can have in the theory's validity. The theory of change model is particularly suited to answer evaluation questions about an initiative's impact, using multiple methods for measuring change that are united by a common framework or research strategy.

Challenges. One difficulty associated with theory-based evaluation is that consensus on what are good interim outcomes for children and youth nationally is an area that is still evolving or there are gaps in the research. However, there are developmental, stage-specific tasks that researchers have found as predictors of subsequent stage outcomes or can be linked to positive adult outcomes. Potential interim indicators are presented from the research literature and discussed.

ii

'

;

/

Findings. The evaluation team found that most Neighborhood Family Centers are providing programs that are very sound for an outcome evaluation study with basic measurement and recording-keeping already in place. These program activities are extremely relevant to mutual goals and objectives of NFCs and The Juvenile Welfare Board's Strategic Plan. The Neighborhood Family Centers also have a history of achieving, and in some cases consistently exceeding, most of the measurable objectives annually. In addition, the evaluation team has identified evidence of strong support and recognition of the Neighborhood Family Centers in their communities by City governments, civic organizations, faith communities, businesses, and other social service agencies.

Evaluation Design Framework. The overall evaluation framework (provided in graphic form on the following page) provides for measurement of outcomes for individuals, families, and the community. It is a results-driven model that postulates that Neighborhood Family Centers create both effects and benefits for the community and that the outcomes of both will be measured. Effects relate to the outcomes produced in the areas of increasing social capital (family cohesion, building assets and reducing risks, improving intra/intergroup relations) and human capital (building skills, increasing academic achievement, and providing opportunities for meaningful participation). The design measures benefits to the community through the enrichment of the neighborhood environment, mobilization of the neighborhood community, and responsiveness to neighborhood assets and needs.

Key Components of Lessons Learned. A separate report will be compiled at the conclusion of Phase I (September 30, 1999) of the study that encompasses the historical Lessons Learned in the implementation of the Neighborhood Family Center initiative. An ongoing Lessons Learned section will be included in each report in Phase II encompassing the lessons learned during the implementation of the evaluation study.

Recommendations. (1) Develop agreed upon procedures between the agencies and organizations that Neighborhood Family Centers collaborate with regarding access to participant data and informed consent. (2) Schedule training for the Results Mapping methodology recommended. (3) Schedule a meeting of Neighborhood Family Center Directors and relevant Juvenile Welfare Board staff to review the definitional terms and measures listed in Section VI of the report. (4) Complete the program specifications for each Neighborhood Family Center program to be included in the outcome study. (5) Compile lessons learned in two major areas: historical lessons learned in the initiative implementation and evaluation lessons learned from the outcome study. (6) Continue the strong involvement of the Neighborhood Family Center Planning and Advisory Committee and the Center Directors in the process of designing and implementing the outcome study. (7) Identify any existing Juvenile Welfare Board internal monitoring processes and tools that could be applied to the outcome study. (8) Recognize that most of the programs, services, and activities of the Neighborhood Family Centers are strong and suitable for rigorous evaluation. (9) Decide what social indicator databases will be used, including the availability, periodicity of the databases for compatibility with the outcome study.

iii

Neighborhood Family Center Initiative Evaluation Design Overview

Results Driven Satisfac tion: For Families, Neighbo rhood, Collaborative Partners, Public

+ + Cause Effects Produce Benefits

Measured by Interim and Advantages, Cost Savings, Quality of life Long Term Outcomes Measured through Process and Monitoring Outcomes .. .. ~ .. ~

Development of Development of Enrichment of Mobilization of the Responsiveness to I Social Capital Human Capital Neighborhood Environment Community (Neighborhood) Neighborhood

I Assets and Needs ~~ A~ ~ .. A~

t Increase Build Skills Develop Range of Build Governance Implement Core Services Family Resources Natural Supports and Critical-Cohesion Stage Development Principles

Increase Academic Build Achievement Contribute to Safety and Attract and Develop Assets and Security Volunteers Develop Linkages, Reduce Collaborative Support Risks Services

Provide Develop Community-Improve Opportunities for Foster Citizenship Driven Activities, Intra/Inter- Group Meaningful Enhance Aesthetics Programs and Events Relations Participation

Juvenile Welfare Board Goals and Operating Principles for Neighborhood Community Development 1. The strengthening and preservation of families. 2. Building stronger individuals, families and neighborhoods. 3. The development of comprehensive strategies that address multiple risk factors and will result in the reduction of crime and substance abuse and the improvement of

maternal/child health. 4. The development of a sense of competence on the part of participants about themselves, their skills and abilities, as well as a sense of responsibility for themselves and

their neighborhoods. 5. The establishment of a network of neighborhood groups of people who have identified their own needs and resources and have demonstrated their ability to mobilize their

neighborhoods to address those needs. 6. The development of positive role models of involved citizens for neighborhood children. 7. Improvement in the overall quality of life in neighborhoods receiving services and in Pinellas County as a whole.

--