Workshop #6: Measuring Access to Justice: Why is it important and how

are we going to do just that?Society of Ontario Adjudicators and Regulators Annual

Conference November 6, 2014

Dr. Keiko Kuji-Shikatani, C.E. Research, Evaluation and Capacity Building Branch

Student Achievement Division, Ministry of Education

You’ve got to know where you are if you want to get to where you want to get to.

Google Maps - ©2012 Google

Keiko.Kuji-Shikatani@ontario.ca SOAR’s Annual Conference Workshop #6 Measuring Access to Justice

For vision-and values-driven social innovators highest form of accountability is internal

Seriousness of inquiry and resulting learning constitutes accountability: Are we walking the talk? Are we being true to our vision? Are we dealing with reality? Are we connecting the dots between here-and-now reality and our vision? And how do we know? What are we observing that’s different, that’s emerging? (Patton 2010)Keiko.Kuji-Shikatani@ontario.ca SOAR’s Annual Conference Workshop #6 Measuring Access to Justice

Logic Models DefinedLogic models are visual methods of presenting an idea.

Logic Models: • Offer a way to describe and share an understanding of

relationships among elements necessary to operate a program or change effort

• Describe a bounded project or initiative: both what is planned (the doing) and what results are expected (the getting)

• Likened to a picture map of how the program theoretically works to achieve benefits for participants - often expressed as brief diagrams or flow charts

Logic Models provide a clear roadmap to a specified end. Knowlton & Phillips (2012) The Logic Model Guidebook – Better Strategies for Great Results, Treasury Board of Canada

(2001) p.11

Keiko.Kuji-Shikatani@ontario.ca SOAR’s Annual Conference Workshop #6 Measuring Access to Justice

Uses of Logic Models

Logic models serve as a tool with multiple uses:• To clarify for managers and staff the linkages between

activities, outputs and the expected outcomes of the policy, program or initiative. In so doing, it will serve to clarify and distinguish the expected immediate, intermediate and ultimate outcomes

• To communicate externally about the rationale, activities and expected results of the policy, program or initiative

• To test whether the policy, program or initiative “makes sense” from a logical perspective

• To provide the fundamental backdrop on which the performance measurement and evaluation strategies are based (i.e., determining what would constitute success)

Treasury Board of Canada, 2001, p.11

Keiko.Kuji-Shikatani@ontario.ca SOAR’s Annual Conference Workshop #6 Measuring Access to Justice

6

Two Types of Logic Models: One LogicTheory of Change Logic Model and Program Logic

Models differ by level of detail and use but represent the same logic:

• A Theory of Change Model is a general representation of how you believe change will occur

• A Program Logic Model details:– Resources– Planned activities– Outputs– Outcomes over time that reflect intended results

Knowlton & Phillips (2008) The Logic Model Guidebook – Better Strategies for Great Results

Keiko.Kuji-Shikatani@ontario.ca SOAR’s Annual Conference Workshop #6 Measuring Access to Justice

We believe that … • Why do you need to do this? • How do you know? • High level descriptions of the

evidence/evaluation (including needs assessment) and research, practice, and theory.

• Define the assumptions that support the specified strategies that inform choices

Keiko.Kuji-Shikatani@ontario.ca SOAR’s Annual Conference Workshop #6 Measuring Access to Justice

… will get us there.

Where do you want to get to? High level descriptions, specifying the intended results

Keiko.Kuji-Shikatani@ontario.ca SOAR’s Annual Conference Workshop #6 Measuring Access to Justice

… doing this …

How would you summarize what you are doing to get there?

High level descriptions of how intended changes will occur

Name the activities and strategies that will deliver your intended results

Keiko.Kuji-Shikatani@ontario.ca SOAR’s Annual Conference Workshop #6 Measuring Access to Justice

Discuss with your elbow partner

Try the elevator conversation test

We believe that…. doing this …. will get us there

Keiko.Kuji-Shikatani@ontario.ca SOAR’s Annual Conference Workshop #6 Measuring Access to Justice

Discuss with your elbow partner: articulate where you want to get to

• Objectives: What do you want to achieve?• What change(s) will occur?Describe stepwise series of outcomes

Keiko.Kuji-Shikatani@ontario.ca SOAR’s Annual Conference Workshop #6 Measuring Access to Justice

Discuss with your elbow partner: articulate the doing

• Name all the activities needed to generate the outcomes.

• Define the resources/inputs that link directly to and will "supply" the activities.

• Identify the outputs that reflect the accomplishment of activities. (Knowlton & Phillips, 2012)

Keiko.Kuji-Shikatani@ontario.ca SOAR’s Annual Conference Workshop #6 Measuring Access to Justice

Discuss with your elbow partner: What would tell you that you are getting there?How do you know?

Logic models shows when, where and how to look for information most needed to manage the program and determine its effectiveness.

Keiko.Kuji-Shikatani@ontario.ca SOAR’s Annual Conference Workshop #6 Measuring Access to Justice

Context Challenges

• Common Pitfalls: blind spots (unintentional omissions & errors out of habits) & myths (e.g. access = use, knowledge = action, activities = outcomes, send = communications)

• Logic, scale (of effort) and specificity (discrete work seeking near-term outcomes)

• Politics, persuasion (grandiose claims?) and perception (shared-view vs. truth/reality)

(Knowlton, L., & Phillips, C. C. (2012). The Logic Model Guidebook - Better Strategies for Great Results. Thousand Oaks, CA: Sage)

Keiko.Kuji-Shikatani@ontario.ca SOAR’s Annual Conference Workshop #6 Measuring Access to Justice

Testing model quality - SMARTSMART tests outcomes• Specific: what to do is clear enough to act-on and is

connected to outcomes• Measurable: the content can be both quantified and

qualified• Action oriented: the content is selected to provoke

change in awareness, knowledge, skill and behavior• Realistic: the content is both plausible and feasible• Timed: the content specifies a duration and illustrates

the time-dependent sequence of outcomes for progress toward results Knowlton & Phillips (2012)

Keiko.Kuji-Shikatani@ontario.ca SOAR’s Annual Conference Workshop #6 Measuring Access to Justice

Testing model quality - FITFIT tests activities• Frequency of occurrence• Intensity of strength of the given effort• Targeted at a specified market or audience

The treatment (or intervention) needs to be both appropriate (the right choice) and adequate (the right dose) for the resolution of problems or generating a change in awareness, knowledge, skill or behaviour.

Knowlton & Phillips (2012)

Keiko.Kuji-Shikatani@ontario.ca SOAR’s Annual Conference Workshop #6 Measuring Access to Justice