JARED C. SCHULTZ, PH.D., CRCUTAH STATE UNIVERSITY

Using Quantitative Data for

Effective Program Evaluation

Misleading Title?

The first critical question is not “How do we measure?”

The first critical question is “What do we measure?”

This presentation is not an overview of quantitative data collection procedures.

This presentation is intended to provide a foundation to address the question of “What do we measure?”

Transformational Learning

Adult learners bring the context of their experience to learning

Has a positive and negative impact on learning

“becoming critically aware of one’s own tacit assumptions and expectations and those of others and assessing their relevance for making an interpretation” (Mezirow, 2000,p.4)

Take a few minutes and write down your beliefs, attitudes, and emotions regarding program evaluation. Positive and Negative. Ambitions and Concerns.

What Program Evaluation is Not

Only a requirement for reporting to RSA and other oversight entities.

Program evaluation, quality assurance, and continuous improvement are not events, or things to implemented quickly. They represent shifts in organizational ideology and culture, and require a long-term, systemic view.

What is Program Evaluation?

“The use of social research methods to systematically investigate the effectiveness of social intervention programs…” (Rossi, Lipsey, & Freeman, 2004).

Data Collection and AnalysisIlluminates Effectiveness and Informs

Continuous ImprovementIs useful for all aspects of organizational

functioning.

An issue of organizational culture.A systemic method of addressing quality of

services, outcomes, and personnel.

Addresses processes and systems within the organization.

Role of Data

Informs Program DevelopmentInforms Program EvaluationInforms Program ImprovementInforms Professional HiringInforms Professional EvaluationInforms Professional Improvement /

Development

Bead Experiment

We need to hire: 2 Counselors Supervisor Data Analysis Specialist

Processes and Systems

Process The use of inputs into an action to achieve a desired

outcome. Inputs include people, materials, equipment, methods,

measurement, environment.

System – network of processes within an organization that work together for the aim of the organization.

(Neave, 1990; Deming, 2000)

An Example of Systems Thinking

A State agency implemented an incentive payment program that was directly linked to case closures.

Incentive Program Goals

Improve Rehabilitation Counselor Retention, Decrease Turnover

Increase Rehabilitation Counselor Compensation

Increase the quantity of Successful closures, Productivity

It was originally presented as a Trial Program

Since Implementation

Voluntary terminations have increased from an average of 15 per year to 21 per year

Voluntary resignations have not declined with the existence of the incentive program

Compensation

An average of 25 counselors receive the incentive in a given year ($894 before taxes)

On average 91 counselors receive no incentive

The average increase in compensation across the board is $180 before taxes per counselor for the 3 year period

Increase Productivity

Average 26's Per Counselor

23.5

24

24.5

25

25.5

26

26.5

27

27.5

28

28.5

2002 2003 2004 2005 2006 2007

Asking the Program Evaluation Question

Identify the ProblemDefine the problem in terms of processes and

systems Draw out the process Identify the way that processes influence the system

Models and Theories

Models and theories provide the context for making the questions concrete.

Provide the structure for understanding the variables.

Don’t assume that models and theories do not apply to your unique situation.

References

Deming, W.E. (2000). Out of the crisis. Cambrige, MA: The MIT Press.

Mezirow, J. (2000).Learning to think like an adult: Core concepts of transformation theory. In J.Mezirow and Associates (Eds.). Learning as transformation: Critical perspectives on a theory in progress. San Francisco: Jossey-Bass.

Neave, H.R. (1990). The Deming dimension. Knoxville, TN: SPC Press.

Rossi, P.H., Lipsey, M.W., & Freeman,H.E. (2004). Evaluation: A systematic approach (7th Ed.). Thousand Oaks, CA: Sage.