CRJS 4466PROGRAM & POLICY EVALUATION

LECTURE #6

•Evaluation projects

• Questions?

Assessing the Need for Programs

Figure 6-1: Simplified Performance Management Cycle

Table 6-1: Maslow’s Hierarchy of Needs

• A need is “a social construction negotiated between a set of social agents with responsibility for social programs and policy and a set of claimants and their advocates who assert that a problem exist that warrants intervention” (Rossi, Freeman and Lipsey, 1999)

• Within the context of program evaluation, the “primary focus of needs assessment is not on human needs broadly defined but, rather, on social conditions deemed unsatisfactory through some process of social judgement and presumed remedial by social programs” (Rossi, Freeman and Lipsey, 1999)

• Assessment of need for a specific program or program proposal

OR

• As a planning and decision-making tool for policymakers to determine how resources should be allocated across various service areas

Needs Assessment

• definitions of need:1. normative (expert opinion based on comparison)2. felt need (client perceptions)3. expressed demand (e.g. waiting list clients)4. comparative need (based on service utilization in one population, inferred to another)5. mandated requirements (e.g. legislation)6. humanitarian (moral/ethical requirement)

Need and demand

• “whereas a population at risk includes all those with a high probability of having or acquiring a given condition, a population in need is a group of potential targets who currently manifest the condition”(Rossi, Freeman and Lipsey, 1999)

• Demand – refers to a desire or willingness to obtain or receive a service (only partial overlap with criteria associated with need)

• “Clearly, the distinctions between populations at risk, in need, and at demand are important for estimating the scope of a problem, anticipating the size of the target population, and subsequently designing, implementing, and evaluating the program”(Rossi, Freeman and Lipsey, 1999)

Needs Assessment

• “a decision-aiding tool used for resource allocation, program planning, and program development based on the

assumption that planned programming can alleviate distress and aid growth”

• requirement to conduct needs assessments BEFORE programs are designed, implemented – it is the first type or stage of program evaluation

• not just needs for service – but criteria related to service usage, actual demand, etc.

Planning a Needs Assessment

• purpose of assessment• scope of assessment (population)• stakeholders to include• budget/resources• time frame• identify specific information required• determine availability of information/need to collect • design methodology and data collection instruments • gather and analyze data• prepare draft report• disseminate draft report to stakeholders for comment• formally disseminate results

Position 1999 2000 2001 2002 2003Total # of potential candidates for the position currently in your police service

Chief

Deputy-Chief

Superintendent

PROJECTED HIRINGChief / Deputy Chief / Superintendent Positions

(please record projected # of positions to be hired in each year)

You are requested to distribute Questionnaire Part B to the individuals you believe are potential candidates for chief / deputy-chief / superintendent positions in the 1999 – 2003 period.

SIX LEVEL POLICE TRAINING SYSTEMLevel IPolice service orientation and in-service training prior to attending the Ontario Police College. 1 –2 weeks in length.Level IIBasic recruit training program at the Ontario Police College, 60 days in length. Level IIIIn-service training and coach officer training at police service. Normally three months in duration.Level IV Two – three week program at the Ontario Police College occuring 18 months after completion of basic recruit program.Level VIn-service training.Level VIThree week program at the Ontario Police College.

Basic Constable Level IV Level VI

Year Trained Trained Untrained Trained Untrained APT1988 14631989 9421990 1373 1125 -3381991 620 983 +411992 491 892 -4811993 500 468 -152 36 -1086 5591994 654 0 -491 11 -684 1731995 713 1612 -339 432 -899 2961996 558 -96 180 -262 5671997 2021 -511 911 -282 2721

Total 6756 4389 -2367 361 -3213 1867

TABLE 1Patrol Refresher Training (Level IV, VI and APT)

Actual Figures, Trained and Untrained, 1988 - 1997Notes to Table: 1 figures estimated based on current (1997) conditions. 2 no Toronto

officers since 1994.

I. Communication NotImportant

SomewhatImportant

Very Important

Must be in refresher

training?

1. Verbal Skills 1 2 3 2. Written Skills 1 2 3 3. Interpersonal Skills 1 2 3 4. Sensitivity Skills 1 2 3 5. Listening Skills 1 2 3 6. Interviewing Skills 1 2 3 7. Ability to mediate 1 2 3 8. Knowledge of human behaviour 1 2 3 9. Ability to be patient 1 2 3 10. Ability to feel & show compassion 1 2 3 11. Presentation skills (public speaking, giving evidence)

1 2 3

InstructionsBeside each job element you are asked to circle the number that indicates how important you think it is that an experienced constable be skilled at that job element. In addition, you are asked to put a check-mark beside those job elements which you feel must be included in patrol refresher training.

TABLE 5Need For and Delivery Characteristics - Patrol Refresher Trainingb

Stage Three SampleN = 30

Responses

Need for Refresher Training? Yes 29 (97%)No 1 (3%)

Frequency of Refresher Training? As needed 8 (27%)Once a year 2 (7%)Every 2-3 years 10 (33%)Every 5 years 5 (17%)Don’t know/missing 5 (17%)Don’t know 1 (1%)

Course Delivery? OPC course 5 In-service 9 Modules 10Instructor/scenarios 21CBT 3Video 3Inter/intranet 4Distance ed 1

Recruits versus Level IV & VI TrainedActual and Forecast Values to 2002

(Cumulative Training Numbers)

Course 1994 1995 1996 1997 1998 1999 2000

Level IV, VI and ‘old’ APT

184 500 1305 475

Re-designed

APT

90a 1000 6000 7000

TABLE 6Training Numbers (Actual & Projected)c

Comparison of previous refresher training numbers with re-designed APT numbersc Re-designed APT pilot courses.

Needs Assessment Approaches

• secondary data analysis – including census, ‘patterns of use/client utilization data’, other socialindicators data, provincial/municipal databases, etc.(note validity, reliability, ecological fallacy)

• ‘impressionistic approaches’ – key informants, community forums, ‘town halls’

• nominal groups, Delphi technique and focus groups• surveys• convergent analysis/multi-method approaches • being creative• mandated requirements• public opinion

Forecasting need

• Forecasting future trends can be quite risky, which is not to say that they should not be used, but that they should be reviewed critically

• Forecasting techniques can be qualitative, causal modeling or time-series

Community Readiness

• no awareness• denial• vague awareness• preplanning• preparation• initiation• stabilization• confirmation/expansion• professionalization

Data Sources - Existing Statistics

• public opinion polls (e.g. Environics, Gallup) • census data (e.g. poverty)

• vital statistics (e.g. suicides; infant mortality; cancer)

• social indicators surveys (e.g. unemployment; divorces)

• social surveys (e.g. the GSS)

• organization records (e.g. school records; sales figures)

• statistical reports (e.g. stock prices; crime statistics)

Data Sources - New Information

• questionnaires (e.g. competency inventories)

• structured interviews (e.g. drug investigators)

• focus groups (e.g. police workload)

• observation (e.g. Wapekeka, NBPS)

• simulation (e.g. remand counts)

Describing service needs

• “fleshing out” the nature of needs - where and how do the needs arise?

• the importance of social context: ethnic/cultural factors, age factors, sensitivity, social class factors (e.g. mandatory charging practices; investigative interviewing)

• need, demand, access to services