PERGAMON

.~_

EVALUATION and PROGRAM PLANNING

Evaluation and Programming Planning 2 1 (199 8) 199-209

Salvaging an evaluation from the swampy lowland

Joan Fitzgerald”,*, Janice Matthews Rasheedb

Abstract

This article discusses methodological issues that emerged in evaluating a three-year paternal involvement demonstration project for poor. inner-city non-custodial fathers. The Paternal Involvement Demonstration Project is a public-private partnership developed to strengthen the financial and emotional relationship of program participants and their children. While the evaluation was considered

formative in many respects, methodological constraints were placed upon the project in order to meet federal and state waiver requirements. This article presents the methodological dilemmas that resulted from these contradictory goals. The tradeoffs that

were made to enable a meaningful evaluation are discussed. The findings of this evaluation project resulted in the creation of a permanent program and are presented in this article. fp 1998 Elsevier Science Ltd. All rights reserved.

Kr~~wortl.\~ Evaluation: Mixed-method evaluation; Paternal involvement programs: Non-custodial fathers: Inner-city: Low-income fathers

1. Introduction

This article discusses several methodological issues that

emerged in evaluating the Paternal Involvement Dem-

onstration Project (PIDP), a three-year, public-private

partnership initiative to demonstrate the value of pro-

viding intensive job training, placement and paternal involvement services to young non-custodial fathers

whose children receive AFDC. While the evaluation was considered formative in many respects, rigid metho-

dological constraints were placed upon it in order to meet

federal and state waiver requirements. These con-

tradictory goals resulted in several methodological dilemmas. This article presents these issues and engages in a discussion of the adjustments that were made so as

to be able to provide meaningful input in a policy arena

in which there is a sense of urgency to create effective programs- --welfare to work.

2. Background

There is a large amount of literature on the relative value of quantitative vs qualitative research meth- odologies in program evaluation. Purists argue that the

*Corresponding author. Tel.: 312.996.8361: fax: 312.996.8933. e-

mail: jfitren ulc.edu

SO149- 7189.98 $19.00 (‘ 1998 Elsevier Science Ltd. All rights reserved

PII:SOl39 7189(98)0001 I- I

two methodologies are based on incompatible assump-

tions and thus cannot be complementary (Mark et al.,

1997). A consensus is emerging. however. that they are

mutually complementary approaches (Reichardt and

Cook, 1979; Sechrest and Sidanl, 1995; Shadish. 1995;

Greene and Caracelli, 1997; Riggin, 1997). Greene and

Caracelli (1997) suggest the underlying premise for

mixed-method evaluation is that each offers a meaningful

and legitimate way of knowing and understanding that

allows for deeper and broader insights. The post-posi-

tivistic paradigm frees the researcher from the underlying

philosophical assumptions of the logical positivists that

assert an inherent superiority of the quantitative

method-rendering the qualitative method a weaker sib-

ling of the social science research family. The employment

of mixed-method approaches offers researchers and their

audience an opportunity to capture the complexity of a

program and provide answers to a wider array of ques-

tions. Indeed, several articles in the recent issue of Ncn

Directions,for Evaluation edited by Greene and Caracelli

(1997) assume compatibility and lay out a theoretical

framework for an emergent realist perspective and use case studies to examine effective strategies for choosing complementary combinations of methodologies.

This literature assumes that the researchers have the liberty of deciding the relative emphasis on each approach prior to starting the evaluation. While this is often the case. the particular dilemma discussed here was how to

200 J. Fitzgerald, J. Matthews RasheedjEcaluation and Program Planning 21 (1998) 199-209

salvage an evaluation employing both quantitative and qualitative analyses once it became increasingly clear that quantitative data were insufficient to conduct statistical

tests of significance due to problems that emerged in

implementation. Our solution was to increase the empha-

sis on the qualitative measures. This required several mid-

evaluation methodological changes. While the solution

may have jeopardized the validity of the quantitative measures, we were faced with the tradeoff that evaluators,

especially of social service demonstration projects, must

often make between traditional notions of social science

validity and usefulness of findings. This dilemma of prac-

tice is best stated by Schiin (1983):

In the varied topography of professional practice, there

is a high hard ground where practitioners can make

effective use of research-based theory and technique

and there is a swampy lowland where situations are confusing messes incapable of technical solution. The

difficulty is that the problems of the high ground, how-

ever great their technical interest, are often relatively

unimportant to clients and the larger society, while in the swamp are the problems of greatest human concern.

Indeed, despite the methodological problems that arose, the funders and implementors of the PIDP still expected results and were not particularly interested in issues of

self-selection bias and diminishing sample size. Evalu-

ation studies of demonstration programs typically have

been saddled with the same fundamental methodological

limitation: the inability to attribute cause and effect in the absence of being able to use tightly controlled exper-

imental designs (Levine, 1993). In the interest of the pro- ject funders and the larger community looking to answers

from this demonstration, we chose to continue our evalu-

ation in the swampy lowlands. Logical positivistic research paradigms, epitomized by

experimental isolation, laboratory control and a hier-

archical preference for use of randomized control groups, can be of limited value in program evaluation efforts and offer little guidance in the face of programmatic changes

made by demonstration site staff as well as other con-

founding methodological dilemmas. Since demonstration or pilot projects seldom place service provision con- straints on project sites it is reasonable to anticipate that

the professional ethos of service providers will lead them to make changes in program interventions (the inde- pendent variables). These changes present formidable methodological challenges in program evaluation efforts and can undermine well conceptualized research.

This article presents an examination and discussion of the unavoidable (inadequate sample size due to dropout) and personally directed (changes in program inter- vention, continual replacement of participants) changes in the program that necessitated methodological change. This evaluation experience, which we view as typical in

many respects, gives support to our position that the use of a post-positivistic research paradigm, with its even

emphasis on the utility of both quantitative and quali- tative approaches, is the best approach for examining

both the process of implementation and the outcomes of

a demonstration program. The ultimate value of an evaluation is to determine the

effectiveness of the program in achieving its goals and

whether and how to replicate it. Using this criterion, we can point to the continuation of the project on a per-

manent basis with funding from several leading foun-

dations and local and national recognition of the program as evidence of the utility of our approach.

3. The paternal involvement demonstration

The Paternal Involvement Demonstration Project

(PIDP) was developed to fill a gap in public policy on welfare-to-work transition-programs focusing on poor

men. Recent welfare reforms, such as the Family Support

Act (FSA) and the Personal Responsibility and Work

Opportunity Reconciliation Act of 1996, focus mainly on single mothers. These reforms attempt to make fathers

more responsible for their children by increasing their child support, but provide no assistance to fathers in increasing their income or in developing relationships

with their children. The problem disproportionately affects African-

American men. Employment is the first step in increasing

the ability of these men to support their children.* The solution to the problem of lack of parental involvement

is more complex than simply finding men jobs, however. Wilson’s (1987) work in particular has focused on the

relationship between poverty and family stability. Due

to the structure of the welfare system, women have no incentive to live with or marry a man who does not have

enough income to bring the household over the poverty level. Inability to find employment further decreases the

black male’s sense of self worth. Clinical research has shown that minority and poor clients are more likely than

their white counterparts to hold a view that their own inadequacies, rather than external events, control their

destiny (Latting and Zundel 1986). The Paternal Involvement Demonstration Project

(PIDP) was created to respond to the need for welfare-

* The high unemployment among many urban African-American

men results from the interaction of three structural factors: spatial

mismatch, skills mismatch, and racial discrimination (Fitzgerald and

Patton, 1994; Moore and Laramore, 1990; Kasarda, 1989). Further

barriers to employment include low self-esteem, ‘streetwise’ attitudes,

low literacy levels, substance abuse, police records, gang activity, low

skill levels and chronic illness (Chicago Institute on Urban Poverty,

1993).

to-work programs that respond to the unique needs of

poor men in Chicago. The program was conceived col-

laboratively in the Fall of 1990 by staff from the Illinois

Department of Public Aid (IDPA) and the Woods Chari- table Fund and was funded by several foundations, with waivers from the federal Department of Health and

Human Services. A steering committee was organized to develop the program objectives and a request for pro-

posals for implementation. evaluation and policy advo- cacy. Since there were few programs of this type that

could serve as models for PIDP, the steering committee sought three sites for service delivery so that there would

be enough variation among them to test a variety of

approaches. It was hoped that the demonstration would serve as a model for welfare reform.

The program had three policy goals:

(1) To assist non-custodial fathers of children on AFDC

to become economically self-sufficient and to be able to provide financial support to their children, through

placement in good jobs that provide adequate salaries

and benefits. (2) To strengthen the ability of disadvantaged fathers to

provide and maintain on-going. positive relationships with their children.

(3) To modify public welfare policy to eliminate existing

disincentives to paternal economic and social self-

sufficiency and to establish the value of on-going federal and state funding for paternal self-sufficiency

programs.

The program served non-custodial fathers who had at

least one child receiving Aid to Families with Dependent Children. Participants had to be between the ages of 18

and 35, willing to stay in the program for two years and to have their paternity adjudicated. The children’s mother

had to sign a consent form for the father to participate with their child.

Program services were provided by three neigh-

borhood-based human and social service agencies in Chicago, chosen through a proposal process. Each

agency was free to construct the services as they saw fit

and to add other services. The purpose of choosing more

than one site was to compare the effectiveness of the different approaches each agency developed for the program. Approximately $200,000 was distributed among the three sites for providing the first year of

services. Each site was expected to serve 50 participants.

Each of the three organizations developed a different approach to providing the human and social services mandated: active case management; employment place- ment and/or job training; and family-focused parenting education. During the third program year one organ- ization dropped out of the demonstration and was restar- ted at another site. The programs designed by the three initial sites are presented separately.

3.1. Site me

The approach of site one was rooted in the belief that

the men’s personal, social. educational and employment problems should be addressed simultaneously. A three-

phase program was implemented. Phase one lasted three months and included a series of assessment andjob readi-

ness workshops. job search and the launching of the group’s weekly discussion meetings. Ideally. all par-

ticipants would find employment by the beginning of

phase two. During this phase, which lasted four months.

workshops were ofered on job survival issues and father

child relationships. This phase also included career. life and educational assessment sessions, and monthly coun-

seling sessions with the site director. Monthly father-

child activities. planned by the group, began in this phase.

In phase three. which lasted nine months. participants

were encouraged to take leadership roles within the

group, to maintain their employment and to continue to develop stronger relationships with their children.

3.2. Site tll.0

The underlying belief of the program at site two was

that the men first had to address personal and self-esteem

issues before they could become good employees or parents. Participants first entered a life skills program,

offered in weekly small-group sessions. The participants

completed self-esteem exercises, discussed strategies for

solving personal problems and ways to enhance relation-

ships with their children. In the job preparation and placement component 01

the program. the men were provided assistance in

developing an employment plan, completing appli- cations, resume writing and in undertaking a job search.

Services for housing, drug treatment, health care and employment training have been arranged with other

organizations.

3.3. Sitt) tllrec

The philosophy of the program at site three was that

once the men were employed. the other aspects of their

life would fall into place more easily. The site director’s belief was that once the men were supporting themselves, significant gains in self-esteem would occur, making it easier to address personal problems and become more

involved with their children. Five activities were included in its program: (I) Case

management including recruitment and retention of par- ticipants and forming a personal development plan aimed at self sufficiency. (2) Parent and family education that included parental skill building workshops. cultural and academic activities with their children and adult group discussions on parental roles and responsibilities. (3) Employment preparation. including job seeking skills.

202 J. Fitzgerald. J. Mutthews Rasheed/Et)aluation and Program Pluming 21 (1998) 199-209

job placement assistance and job retention counseling.

(4) Education and training, as needed, and (5) Personal Skill Building sessions include self-assessment and self-

improvement skill building, self-esteem enhancement,

personal goal setting and values clarification. Since the

year one program focused exclusively on employment training and placement, the parenting aspect of the pro-

gram was not developed in the first year.

4. Evaluation methodology

The proposed evaluation included an impact assess-

ment, a process analysis and a cost analysis. To meet the

requirement of federal government waivers, the original

design was largely quantitative. It called for an impact

assessment using random assignment of eligible men into

a treatment and control group. Using the contingent

frameworks of Reichardt and Cook (1979) and Shadish et al. (1991) a methodology was developed that relied

primarily on quantitative indicators, using limited quali-

tative analysis to verify and elaborate on the findings. The impact assessment would examine changes in the

participants over time and compare their outcomes to

those of control group over a three-year period (Fig. 1).

The original design called for use of analysis of vari-

ance with gain scores to examine changes in performance

on income, employment, parental involvement and self esteem with repeated scores on the same tests (Cook and

Campbell, 1979). The data were obtained from an intake interview and

semi-annual participant and control group surveys and

interviews. Employment, education and income data

were obtained from the participant intake form. Once the

men were enrolled, employment was tracked by the sites directly through the employers. Child support payments

were tracked through IDPA, since the payments were deducted from earnings. Improvement in parental

responsibility was measured with a self-report on the

amount of financial support provided to the child, actual

wage deductions for child support and on the frequency of contact with the child.

Father-child interaction was to be measured through two indicators, self reports and Perosa’s Structured Fam-

SITE ONE SITE TWO SITE THREE COMPARISON GROUP

employment and income educational attainment child support father-child interaction self-esteem locus of control

Time 1

Fig. 1. Impact Evaluation Framework

ily Interaction Scale (Perosa, 198 1). Self esteem was to be measured with the Rosenberg Self-Esteem Scale, a lo- item Guttman scale (Rosenberg, 1979). Locus of control

was to be measured using the Nowicki-Strickland Internal-External Locus of Control and Responsibility

Scale (Nowicki and Strickland, 1973). To complement the quantitative data, we proposed to

conduct semi-annual focus groups to allow the par-

ticipants to report on program impacts that were not captured in the surveys. This would allow a deeper under-

standing of how the program elements were impacting

the participants.

The process analysis determined how effectively the services were provided and how appropriate the inter-

ventions were in achieving the desired goals. It served

three purposes. First, it provided feedback to the service providers so that service changes could be made while

the program was still in progress. Second, it allowed the

evaluation team to examine the degree to which pro- grammatic shortcomings were due to faulty implemen-

tation of the interventions vs the choice of interventions provided. The third purpose of the process analysis was

to provide data for program diffusion. This analysis

focused on identifying implementation problems, pres-

entation of solutions attempted and their effectiveness and characteristics of successful organizations and indi-

viduals providing services.

5. Program elements that necessitated changes in the evaluation methodology

The following discussion elaborates the changes made

by necessity because of unanticipated events or by choice

to improve the services provided.

5.1. Changes in entry cohorts

The evaluation team was selected after site selection

already had taken place. Shortly after being selected, the

evaluation team was informed that the sites had allowed participants to enter their programs. Thus, random assignment into the treatment and control groups was no

longer an option, the only alternative was to move to a

Time 2 -> Time 6

J. Fit:gerald, J. Matthews Rasheed/Eraluation and Program Plan&y 21 (1998) 199 209 203

non-equivalent comparison group design. Since there is

debate over the extent to which findings from exper- imental design studies of social service programs are gen-

eralizable because they do not reflect how people enroll in programs (Humphreys; Braver and Smith; Levine, 1993) we were confident that a comparison group design

could yield generalizable results. Nevertheless, we were

concerned that this change in methodology would intro- duce self-selection bias into the analysis, which would make it difficult to differentiate between selection matu-

ration differences and treatment effects (Bell et al.. 1995; Hollister and Hill, 1995).

During the first six months of the project, 40 non-

custodial fathers were recruited for the comparison group from Illinois Department of Public Aid offices. As a con- dition of their enrollment, the men agreed to be available

on an semi-annual basis for repeat interviews, for which

they would be paid $20. The men left two addresses and telephone numbers where they could be contacted to

arrange future interviews.

xZ tests were used to test for differences between the groups on key demographic traits.* Significant differ-

ences between the groups were found on only one of nine variables. Significantly more comparison group members received food stamps than participants.

Although not ideal, we were satisfied that reasonable comparisons could be made between the outcomes of the two groups.

5.2. Rccruitrne~lt problems

Despite the early entrants into the program, difficulty in recruiting participants became a problem. A key con-

tributing factor was that the Illinois General Assistance

(GA) program was eliminated just as PIDP started. Since over 500 clients went through IDPA offices everyday,

IDPA assumed that recruitment would occur through the

GA Jobs program. Without this source of eligible men, the sites had to assume more responsibility for recruit- ment than anticipated.

The site directors thought that the paternity adjudi-

cation requirement was the main reason men hesitated

to enroll in the program. Another major factor in the

difficulty in recruitment stems from the sociological and psychological characteristics of the men associated with long-term poverty and unemployment. The literature

suggests that poor people are likely to be isolated and have a sparse or non-existent social network-an impedi-

ment to ‘word of mouth’ recruitment efforts (Gadsden and Smith, 1994: Miller. 1994). For many men, the most

*The variables were age, raceiethnicity. number of children,

education. food stamp recipient status, living in a household in which

someone worked, childhood public aid recipient, childhood family

structure and parent’s employment status during childhood.

frequently heard response from friends and family was

that the program ‘is just trying to get child support from

you’. Under the original design, the programs would not

have begun until 50 men had been assigned to each site. As implemented. the sites started their programs with as

few as ten men due to difficulties in attracting higher

numbers and an anxiousness to start once men had been recruited. This meant that the research design would have

to take into account that the men were entering the pro-

gram in different cohorts.

5.3. Program uduptatioi~

Changes in program interventions and initial difhculty in recruiting men were two factors that further muddied

the evaluation. The program changes with each cohort

meant that rather than tracking three groups of 50 men who entered the program at the same time, each of these

cohorts had to be considered independently, since each

received a different set of services. Combined with the high dropout rate. this meant that some cohorts had as

few as three men. Thus the ‘n’ for each comparison was

too small to conduct statistical comparisons of before and after measures. This left before and after measure-

ments on individuals over time as the only way of mea- suring program outcomes.

From an evaluation perspective, one of the advantages of having men participate in cohorts is that the mix of

services could be adapted (without presenting metho-

dological problems) as staff learned which activities and

approaches were most effective. However, the continual infusion of new participants (due to poor initial recruit-

ment efforts and high dropout rates) made it difficult to identify discrete cohort groups with similar prqject

experiences.

5.4. High dropout rutr

The dropout rate was very high among both the par- ticipant and comparison groups. It is well-known that

retention is a problem in programs serving this popu-

lation (Eckland. 1968). The participants dropped out for a number of reasons (discussed later), which mainly

focused on their lack of trust and inability to see immedi- ate improvements in their financial situation as a result of the program.

There was even less motivation for the comparison

group to participate. The approach for maintaining con- tact with the comparison group followed well-established strategies (Coen and Patrick. 1996: Cohen et al., 1993). Comparison group members were assured of the con- fidentiality of their responses. In order to increase their comfort level. the interviewers were young African- American males. The men were paid cash for the inter- views and were assured they would be paid in the future

204 J. Fitzgerald. J. Matthews RashredlEualuation and Program Planning 21 (1998) 199-209

as well. Several attempts were made by both mail and phone to reach the comparison group for the second and further interviews. For those who could not be reached,

IDPA provided updated information when available.

These efforts resulted in reaching only 17 of the original 40 comparison group members. Since the federal waiver

required a comparison group and since the participants

were being recruited in cohorts, a decision was made

to select an additional cohort of 40 comparison group

members. The same problems with dropout was present in this group. This meant that the quantitative measures for father-child interaction, self esteem and locus of con-

trol could no longer be used to assess program impact.

The combined effect of changes in entry cohorts,

recruitment problems and program adaptation was that

much of the planned statistical analysis of differences between the participant and comparison groups and of

the participants over time could not be completed. Sum- mary statistics were calculated for both groups on all

variables, but the number of matched comparisons for

three iterations was as low as three. The only option for salvaging the evaluation was to add more qualitative

indicators of program process and outcomes. After several discussions with the program funders and

the sites, a decision was reached to rely more on quali-

tative analysis. It was at this point, almost a year into the

evaluation, that the move to a case study approach began.

Yin (1994) argues that the case study method, which

encompasses both qualitative and quantitative methods, is the quintessential form of post-positivistic research

methodology. Thus, the incorporation of this design into the overall methodology facilitated the optimal mixing

of quantitative and qualitative data for a more complete, contextual analysis of the data. Introducing more quali-

tative data did not violate our ideological approach, nor did it present a technical dilemma.

In an attempt to contextualize the data and retain the

complexity of participant response we used varied sources of data. Ethnographic interviews of site directors, case

managers, family development specialists and project consultants were conducted. In addition, documents

(social histories taken by licensed mental health clin- icians), archival records (administrative subcommittee meeting minutes, organizational records, personal diaries

and poetry of the participants), direct observation of

group meetings of the participants and participant obser- vation of father-child field trips were used.

This qualitative approach allowed for a considerably deeper and broader understanding of what elements of the program worked, and the extent to which factors external to the program. The qualitative data also were used to gain increased awareness and understanding of the psychosocial themes and patterns that operate in the men’s personal lives, information on the motivations, behaviors and life circumstances of the participants to inform future programming, policy and case man-

agement/counseling efforts. Since we had little access to those who left the program, the remaining men were asked to speculate as to the reasons for high program

dropout. Men from all three sites participated in focus group

interviews held at six-month intervals. There were new and long-term participants in attendance at all three sites

for the focus group discussions. Together, these inter-

views provided the evaluation team with a more in-depth understanding of the program, its successes and identi-

fication of obstacles that prevent the program from achieving its goals.

6. Findings

Several findings emerged from the mixed-method

approach, and are elaborated below.

6. I. Employment and income

The quantitative data revealed that PIDP increased the employability and employment of public-assistance- dependent, non-custodial fathers. The ability of the sites

to place men in employment improved over time. Two of

the three sites placed 63% and 69%, respectively, of

the total number of participants who enrolled in their

programs during the third year. Their increased success was a result of decreased dependency on public-sector

intermediaries to assist in job placements to a strategy of developing contacts and relationships with private-sector

employers. This supports existing knowledge of the importance of networking in obtaining employment

(Moss and Tilly, 1992; Neckerman and Kirschenman, 1989; Mier and Giloth, 1985). The role of the sites was

that of a trusted intermediary to the private sector for

those who lack access to such networks. All PIDP sites had some success in placing participants

in employment. The sites became better at placement in years two and three (Table 1).

The employment statistics made it quite clear that the integrated service provision approach of site one was

the most effective. Site two assisted the men in finding employment, but focused on self esteem and personal

issues. Site three focused on employment as well, but was having difficulty meeting the program criteria since there was no parenting component. Ideological battles within the organization resulted in little being accomplished dur- ing the first year. In the second year, site two emulated the approach of site one and produced significantly higher employment rates for its clients.

Interviews with the site staff revealed many of the difficulties the men had in finding jobs. From the service delivery side, it became apparent that the sites would have to have a full-time employment preparation specialist to

Table I Employment of participants at the three sites

Site one

Year I Year 2

Total enrollment

Tonal job placement\

Placements as 3 ‘!G ofcmployment

Average hotlrl~ wgc

A\;erage length <>I’ rmployment

40

25

63%

S5.41

6 month5

identify leads on possible job openings and to maintain relationships with employers.

A problem in locating jobs for the men was that many

of the job openings in the local economy are being created in distant suburbs and were not easily accessible by public

transportation. The men reported that it was very difficult for them to get to work on time using public trans- portation.

Further, evidence suggests that there simply are not enough jobs available anywhere for which the men could

qualify. A recent study that examined whether there were

enough -jobs available to ‘end welfare as we know it’,

found that in the Chicago metropolitan area there are six job seekers for every available job. If one were to include

only those jobs that paid a living wage, then the job availability would fall to one job for every 18-33 job

seekers, depending on how living wage was defined (Carl-

son and Theodore, 1996). This structural characteristic of urban labor markets could not be overcome by this 01

any other program.

The focus groups and interviews with staff allowed us

to understand why so few men were able to find and maintain their jobs. Four interrelated problems were

identified: low wages. lack of employment skills. lack of coping skills and substance abuse.

Given the low skill levels of most of the men, few of them qualified for jobs paying much over the minimum

wage. Average hourly earnings of employed participants at the three sites were $5.41, $5.50 and $5.75. respectively.

The qualitative analysis allowed us to examine how low wages were a source of discouragement among the par-

ticipants. As a condition of participation, the men agreed to have child support deducted from their wages. This

made their already small paychecks seem even less worth the etiorl.

It was due to lack of employment skills and in many cases lack of basic literacy. that few of the men qualified for jobs paying much over minimum wage, which was discouraging for the men. While the men understood the need for and were willing to pay child support, they were

Site two Site three

Year 1 Year 2 Year I Year 2

6 I

7

1 I ‘! 0

s5.75

n;,

subjected to a formula for paying it (20% of gross income

per child) that would leave them with little left. This was

an issue particularly for men who could make money

from ‘side hustles’, licit or illicit. indeed, the site directors and program staff expressed mixed feelings about asking

participants to work at minimal wage jobs and then turn much of their earnings over to the courts to support

several children. Some staff even pondered if it was ethical

to encourage a man to sweep floors and make minimum wage and forego ‘side hustles’ when the net result was

that the mothers got a minimum increase from the state

and the family did not make any real gain since the men

often provided more cash support before, though this support was unreported to IDPA.

Many of the men were not ready for the responsibility

of holding a full-time job. Several of the men commented that they did not feel they belonged in their employment

settings. Since they were hired because of the program,

they reported being treated differently by their employers and by fellow workers. This feeling of alienation was very

discouraging to the men. The men lacked coping skills in such situations. Whether it was hearing people make

racist comments or other forms of criticism. the men would simply quit if they felt intimidated or ridiculed on

the job.

The inability to maintain jobs was partly because work patterns had not been established and many had little or no experience in relating to others in ;L formal setting outside of their early school experiences. Site statr

observed that some of the men used the same language that they would use on the street in their place of employ-

ment. Many men had to learn different communication skills that are more appropriate for the work place. as

well as appropriate dress.

Several men reverted to substance abuse after receiving their paychecks. Major depression and reliance on drug and alcohol to relieve depression were additional hurdles

206 J. Fitzgrruld, J. Matthews RasheedjEvaluation and Program Planning 21 (19%‘~ 199-209

the men had to mount in order to retain jobs. This prob- lem caused some of the men to miss days at work and in some cases resulted in their being fired while still in the

probationary period. In that many of the men in the PIDP

program had significant alcohol and drug problems, the

effectiveness of nearby drug/alcohol rehabilitation pro- grams proved to be a key element in determining program

success. The sites reported that much of the attention

rate among the participants was attributed to the men’s difficulty in resolving their alcohol and drug addictions

6.5. Father-child interaction

The program seems to have had a noticeable impact on many of the men’s parenting skills. For example, the

men reported making good use of stress management

techniques. They reported an increased ability to talk out

difficulties with their children and to use alternative forms

of child management, as opposed to hitting and yelling. The men also reported being aware of a wider repertoire

of parent-child activities. For instance, with the onset of

the literacy programs, some of the fathers reported that they were reading to and with their children. The program

also taught them that it is okay just to be a good father

and that they do not have to have money or take their children somewhere in order to spend time with them.

Although the data should be interpreted cautiously due

to high dropout rates, there is both quantitative and qualitative evidence that the participants increased the

time spent with their children (Table 2). The focus group sessions confined that the most impor-

tant and noticeable change among the men was their new

found sense of empowerment and entitlement of their

paternal rights, roles and functions. The men pioneered innovative ways of establishing a role for themselves in

their children’s lives. For example, some of the men

would go to the mother’s house and perform maintenance tasks and odd jobs around the house as a way of con-

tributing to their children’s welfare. Another father, after having completed the adult basic education (ABE) and

Table 2

Interaction between fathers and their first child after PIDP intervention

Frequency Participants time

I (86) %

Participants time

3 (9) %

Comparison* time

(65) %

Comparison time

3 (12) %

general equivalency diploma (GED) programs, reported going over to his children’s house each night to read them a bedtime story. Yet other fathers reported that they act as escorts for their children through their dangerous

neighborhood, thus giving the mother a reprieve and

allowing the children more opportunity to participate in evening extra-curricular activities. By focusing broadly

on these contributions rather than simply measuring increased child support, we were able to identify con-

siderable ways the participants became engaged with

their children given their constrained circumstances

(Rasheed and Johns, 1995). The qualitative data collected through the focus groups

allowed us to identify programmatic elements that increased parental involvement. We concluded that par-

ental involvement, measured both by financial support

and interaction with children, can be increased through instruction and support groups like those offered through

PIDP. An interesting finding discovered only through the intake interviews is that many men equated the right to

participate in their children’s lives with their ability to

support them financially.

Parenting programs were developed at all three sites

that focused on building parenting skills. All of the pro- grams took an Ajicentric approach in these classes.

Through structured events, staff were able to observe the men with their children and offer advice for dealing with

situations that arose. In the focus groups the men

reported on a new self confidence they felt and the per-

sonal satisfaction that arose from becoming more active in their children’s lives.

Still, the issue of child support loomed large. During the intake interviews many of the participants reported

providing ‘under the table’ support for their children on

an intermittent basis. They resented having child support

withheld from their earnings as a result of the program,

especially because it reduced the amount of support the mother received. Staff at the PIDP sites report that peer

group support encourages fathers to keep up their child support payments. Still, several men reported that it was

Daily

Weekly Twice monthly

Monthly

Few times a year

Holidays

Never

29. I 33.3 21.5 33

33.7 55.6 38.5 33 10.5 0 21.5 17

4.7 0 4.6 0

11.6 0 9.2 17

2.3 0 3.1 0

8.1 11.1 1.5 0

* Percentages add to less than 100 due to rounding

not very encouraging to have to commute by public trans-

portation for two hours each way to a job that pays $6.00

per hour, and receive very little once child support was

deducted. This reality has to be seen by policy makers as a barrier

to obtaining increased child support. Some states are trying to account for this barrier by reducing the amount

of child support that can be deducted from wages accord-

ing to the income earned (Rabinovitz, 1996). Further, as a result of the qualitative analysis we were

able to make three policy recommendations for programs focusing on this population.

1. Education and job training are more effective when

provided along with employment. Despite frequent encouragement. the dropout rate for

the GED courses provided through the program were

high. Interviews with the men revealed that, despite what they were told, they did not believe there was a connection between the GED and employment, and evidence sug-

gests that they are correct. States arc increasingly placing restrictions on the

amount of time unemployed people can stay in training.

This suggests that education and training for programs like PIDP needs to be more clearly connected to employ- ment. An example of such an approach would be to

establish ‘earn while you learn’ hiring agreements that

would have employers hire the men on a part-time or contingent basis while they are obtaining skills relevant

for their employment.

2. The multi-faceted problems facing young, poor non- custodial fathers requires a comprehensive programmatic

response.

The PIDP program planners are correct in their

assumptions that welfare reform has to do more than demand child support if it is to be effective. Each of the three sites offered some combination of employment

skills building. employment readiness training and

addressing personal and interpersonal relationship prob-

lems. It became clear early on that these issues could not be addressed sequentially, but had to be addressed

together.

Each cohort went through assessment, placement, par- enting classes and other programs together. It became obvious through sitting in on the support groups and

through the focus group interviews that the peer support groups Lvere a key factor in encouraging the men to

persevere through whatever difficulties were preventing them from keeping a job or being a better parent. Through the common experiences of the more formal workshops and sessions, the men were able to build on their newly found self confidence and coping skills. Had each of these program components been provided sep- arately under a voucher system that is being proposed by the move toward one-stop employment centers, this

critical element of support that is a key factor in success

would be lost.

3. Even a program that targets fathers has to include

mothers. One of the goals of the program was to increase the

participants involvement in their childrens lives, regard-

less of child support. A key finding was that the coop- eration of mothers was essential in achieving this goal.

Staff at all three sites reported various tensions in the

men’s relationships with their children’s mother. The ten-

sions were in part based on the mothers’ envy of the PIDP services that were offered to the men. particularly in view of the fact that most of the mothers needed

additional support services themselves.

For the most part. the mothers were supportive of the

fathers’ attempts to become more in\polved with their

children. There was. however. some reporting of mothers being adversarial to the men’s efforts to become better

connected with their children. Sever-al mothers resented

the sites providing Christmas prcscncs for the men to give to their children since they could not aflbrd to buy their

children presents. From their point 01‘ view. ;I long-absent

father who suddenly v,ants to become a presence in their children’s lives is seen as competition for the affection of

the children. A Sew mothers would not grant permission

for the participants to take their children on future events

planned by PIDP.

The core issue for some mothers \vas that of anger

directed at the fathers for taking so long to get their lives together. In many cases the mothers found it difficult to get past their anger at the father for years of unac-

countability. and they wondered why the father had not pulled himself together earlier for the sake of the children.

Effective strategies for ameliorating these tensions

include counseling and encouraging mothers to join par- ticipants in PIDP activities.

In summary. the PIDP project demonstrated that com-

bining job placement assistance. parenting instruction,

peer support and social services provided advantages over offering any of these services separately. The quali-

tative analyses not originally included as part of the

research design clarified the interconnections between skills. employment and the complex factors that separate poor men from involvement in their children’s lives.

7. Conclusion

From an evaluation perspective, it will never be known whether the findings would have been as positive if the data were available for program and comparison group dropouts. We conclude that had we not engaged in the more intensive qualitative analysis. many of the key pro- gram findings would have remained undiscovered. Further, using this knowledge to restructure the program

208 J. Fitzgerald, J. Matthebvs Rasheed/Evaluation and Program Planning 21 (1998) IYY-209

resulted in improved employment outcomes. Based on

these observations, the program has moved from a dem- onstration project to an ongoing program. Most of the original funders of the demonstration have continued

supporting the program.

There has been a recent burgeoning of demonstration-

type pilot programs, similar to the one discussed in this

article, designed to deal with the critical social issue of strengthening the paternal role of poor, inner-city, non- custodial fathers (Rasheed, forthcoming). In our case, the

use of mixed-research methods provided greater potential

for knowledge building. The exclusive reliance on the

dominant positivist method would have obscured many

aspects of the paternal roles and functions of the poor, inner-city, non-custodial father. This population is not

well understood by social science researchers. Current

biases place mixed-method methodologies at a dis- advantage in bidding on these evaluations. It is therefore

imperative that service providers convince funders that

the post-positivist, mixed-method approach will increase this understanding.

Acknowledgements

The authors would like to acknowledge that the data

used in this article are part of their research project report (An Evaluation of the Paternal Involvement Demon-

stration. UIC Center for Urban Economic Development,

project #360) that was funded by the John B. and Cather-

ine T. MacArthur Foundation.

References

Bell, S. H., Orr, L. L., Blomquist, J. D.. &Cain, G. G. (1995). Program

Applicants as a Comparison Group in Evaluating Training Programs.

Kalamazoo, MI: Upjohn Institute for Employment Research.

Carlson, V., & Theodore, N. (1996). Are There Enough Jobs? Chicago:

The Urban League.

Chicago Institute on Urban Poverty. (1993). No Work. No We&xre.

Able-Bodied Men on the Sireels ofChicago. Chicago: author.

Coen, A. S., Patrick, D. C., & Stem, D. (1996). Minimizing Attrition

in Longitudinal Studies of Special Populations: An Integrated Man-

agement Approach. Eraluation and Program Planning, 19,309%3 19.

Cohen, E. H., Mowbray, C. T., Bybee, D., Yeich. S., Ribisl. K.. & Freddolino, P. P. (1993). Tracking and Follow-Up Methods for

Research on Homelessness. Evaluation Revielv. 17, 331-365.

Cook, D., & Campbell, D. (1979). Q uasi-E.uperimentation: Design and

Analysis Issues,for Field Settings. Boston: Houghton Mifflin.

Eckland. B. K. (1968). Retrieving Mobile Cases in Longitudinal

Surveys. Public Opinion Quarterly, 32, 51-54.

Fitzgerald, J., & Patton, W. (1994). Race, Job Training and Economic

Development: Barriers to Racial Equity in Program Planning. The

Reriebv qfB/ack Political EconoqJ,. 23, 93-l 12.

Gadsden, V. L., & Smith, R. R. (1994). African-American Males and

Fatherhood: Issues in Research and Practice. Journal qf’Negro Edu-

cation, 63, 63448.

Greene, J. C., & Caracelli, V. J. (1997). Defining and Describing the

Paradigm Issue in Mixed-Method Evaluation. In J. C. Greene and

V. J. Caracelli (Eds), Advances in Mixed-method Evaluation: The

Challenges and Benefits of Integrating Diverse Paradigms. N~II+

Dircctions,for Evaluation. (1974. summer) (pp. 5-l 8). San Francisco:

Josey-Bass.

Kasarda, J. (1989). Urban Industrial Transition and the Underclass.

Annals of the Amcric~an Acadenl), of Political and Sociul Science, 501,

2647.

Latting, J. E.. & Zundel. C. (1986). World View Differences Between

Clients and Counselors. Soc,ia/ Case Work, 12, 66.-71,

Levine, J. A. (1993). Including Fathers in Head Start: A Framework

for Public Policy and Program Development. Families in Society.

The Journal of Contcmporur~~ Humun SertYces. 28, 420.

Mark, M. M., Feller, I., & Button. S. C. (1997). Integrating Qualitative

Methods in a Predominantly Quantitative Evaluation: A Case Study

and Some Reflections. In J. C. Greene & V. J. Caracelli (Eds).

Adrancc~ in Mised-method Et~uluation: The Challenge.\ cm/ Benefits

of Intc,gra/ing Dicvrse Paradigms. Ne,v Direc/ions for Eruluution. ( 1 974. summer) pp. 47-51. San Francisco: Josey-Bass.

Mier, R.. & Giloth. R. (1985). Hispanic Employment Opportunities: A

Case of Internal Labor Markets and Weak-Tied Social Networks.

Social .%ience Quarter!).. 66. 296~ 309.

Miller. D. B. (1994). Influences on Paternal Involvement of African

American Adolescent Fathers. Child and Adolescent Social Work

Journal, II, 363-376.

Moore, T. S., & Laramore. A. (1990). Industrial Change and Urban

Joblessness: An Assessment of the Mismatch Hypothesis. Urban

Affirirs Quarter/>,. 25. 640-58.

Moss, P.. & Tilly. C. (1992). Why Black Men Are Doing Worse in the

Labor Market: A Rericll, of’ SuppI!,-Side and Demand-Side Espla-

nations. New York: Social Science Research Council.

Neckerman, K.. & Kirschenman, J. (1991). Hiring Strategies. Racial

Bias, and Inner-City Workers. Social Problems, 38, 433-447.

Nowicki. J., & Strickland, B. R. (1973). A Locus of Control Scale for

Children. Journal of Consul/ing and Clinical P.\~chologj,. 40. 148%

154.

Perosa. L, Hansen, J. & Perosa, S. (1981). Development of the Struc-

tural Family Interaction Scale. Famil! Therapy. 8,. 77-90.

Pressman, J. L.. & Widavsky, A. (1973). Inzplenlerttariorl. Berkeley:

University of California Press.

Rasheed, J. M. (forthcoming). Program Implementation Issues of a

Paternal Involvement Project for Poor. Non-custodial African-

American Fathers. Sociul Work (forthcoming).

Rasheed, J. M.. & Johnson. W. E. (1995). Non-Custodial African-

American Fatherhood: A Case Study Research Approach. Journal

of Community Practice, 2. 99-l 16.

Reichardt, C., & Cook, T. (1979). Beyond Qualitative Versus Quan-

titative Methods. In T. Cook and C. Reichardt (Eds). Quulitutire

und Quanti/a/irc Research Merho& in Er~rluarion Research (pp. IO

29). Beverly Hills, CA: Sage.

Riggin. L. J. C. (1997). Advances in Mixed-Method Evaluation: A

Synthesis and Comment. In J. C. Greene and V. J. Caracelli (Eds).

Adcunce.~ in Mised-method Evaluation. The Challenges and Benefits

of Integrating Direr-x, Paradigms. Nelt, Directions fi)r Evaluation.

(1974. summer) (pp. 87-95). San Francisco: Josey-Bass.

Robinson G.. & Hill, J. (1995). Problems in the Evaluation of Com-

munity-wide Initiatives. In J. P. Connell. A. C. Kubisch, L. B.

Schorr and C. H. Weiss (Eds). Approuches to Evaluuting Communit~~

fnitiutircs (pp. 127-172). Washington DC: The Aspen Institute.

Rosenberg. M. (1979). Con(,eiring the Self. New York: Basic Books.

Schiin, D. (1983). The Reflective Praclitioner. New York: Basic Books.

Sechrest, L.. & Sidani, S. (1995). Quantitative and Qualitative Methods:

Is There an Alternative? Evaluation and Program Planning. IH. 77-

87.

Shadish, W. R. (1995). Philosophy of Science and the Quantltativee

Qualitative Debates: Thirteen Common Errors. Eraluafion und Pro- Wilson, W. J. (1987). 7%~ Tru/y Dictr~lr~clntcr,qc,u’. Chlcago: The University

grant Planning. 18. 63-75. of Chicago Press.

Shadish. W.. Cook. T. & Leviton, L. (1991). Foundarions q#‘Program Yin. R. K. (1994). C‘usr .S/UC~I. Rcwnrc~lr. Thousand Oaks. CA: Sage.

G&urion. Newbury Park, CA: Sage.