Why Evaluate?

Before After

www.usablellc.net

Organizations evaluate in order to find the number.

The number of kids who graduate from high school.

The number of people who are able to get and keep a new job for six months.

The number of points by which students’ math scores rise.

The number of points by which a psychiatric patient’s depression score decreases.

These numbers all describe outcomes.

Outcomes are about the results of a program.

Outputs are about the steps you followed to produce those outcomes.

It’s important to understand the difference.

But what is the number for?

Why do we need to know the number?

Well, existing funders want to know if their investments yielded results.

New funders want to investin programs that work.

Board members, staff, clients and community stakeholders want to associate with programs that are effective.

But what really is the number?

To be useful, the number has to represent a comparison against expectations.

Expectations that without the program, people/ institutions/ communities would not have changed.

Expectations that without the program, people/ institutions/ communities would not have changed.

* If you remember back to your statistics course, this was called the null hypothesis.

So essentially, we have an experiment.

Differences between the number before the program and the number after the program have to be due to the program*.

*All other things being equal

That was easy, right?

Pretty reasonable, wouldn’t you say?

Did you ever sign up for an online service where you had to check a box indicating that you had read and agreed to all the fine print?

Did you actually read all the fine print?

Neither did we.

But in this case you have to.

Differences between the number before the program and the number after the program have to be due to the program*.

*All other things being equal

But all other things are equal.NEVER

The boys may do better than the girls.

Counselors may interpret the treatment modality differently.

Participants with different work histories may respond differently to the work readiness program.

Looking at the averagechange in:

Math GradeDepression ScoreMonths on the jobmay not be enough.

But wait, it gets even worse.

What if the boys do better than the girls UNLESS the girls are taught by Ms. Smith in which case they do better?

What if more experienced counselors do better regardless of differences in treatment modality?

What if differences in work histories are only important for workers moving from traditionaloffice work?

Organizations that want to improve their programs need to understand these differences.

Because maybe the girls need more teachers like Ms. Smith.

*Find out what Ms. Smith does that she’s so effective with the girls.

Because maybe the girls need more teachers like Ms. Smith.

And only the less experienced counselors need more training.

And only the less experienced counselors need more training.

* Devote resources to retraining just the less experienced counselors.

And the workforce readiness program needs to treat people with experience doing office work differently than those who held blue collar jobs.

And the workforce readiness program needs to treat people with experience doing office work differently than those who held blue collar jobs.

* Create two different programs, one for former office workers, the other for former blue collar workers.

Maybe what this means is that evaluation is about more than just the number.

Maybe beyond asking the question “did the

program work” it should ask:

Where did it work?For whom?Under what circumstances?

We believe that effective organizations use evaluation to document positive outcomes….

to inform continuous improvement.&

If you found this useful, we’d love it if you would share it with your networks.

www.usablellc.net