SOCHA

Mixed Method for Measuring Social Change (3MSC™): An application of Qualitative Comparative Analysis to

Education

May 15, 2014

Presentation by

Carroll Patterson, PhD Partner

SoCha,llc

SOCHA

The Agenda

• Conceptual Backdrop: Problems facing M&E and Impact Evaluations

• The 3MSC Solution: Define and Demonstrate

• Analyzing the Data with an Iterative and Collaborative Approach

• Conclusion and Implications

SOCHA

Some Current Challenges Facing M&E

• The Log Frame and Implementation Blinders

• Silo effect of M&E and Program Management

• Unidirectional output indicators lead to questions regarding “What do the numbers mean?” but oftentimes provide no answers.

• Targets are arbitrary and not rooted in desired change • Most projects lack formal analytical tools that can tell them what

numbers mean, what the dosage should be, and how we do this.

Result: Performance Management is prioritized over hypothesis testing

SOCHA The Rigorous Impact Evaluation Problem

The Randomized Control Trial (RCT), aka the “Gold Standard,” now ultimately passes the verdict on “what works and what doesn’t.” BUT, although RCTs are incredibly expensive, they do not answer “how” questions There are two reasons for this: • Answering “How” questions requires more gold, i.e. increased sample size • Collecting the information needed to answer “How” questions sits outside the

traditional M&E framework RCTs also: • Are driven by Quantitative concerns; sample size determines knowledge • Require highly specialized skills and principle investigators who lack the

situational awareness of ground implementation. • Are completed at the end of the project.

RCT’s are not designed to empower program learning, and in fact

disempower Implementers who have little say in the verdict

SOCHA The Result: Conceptual Gap bzw. Evidence and

Implementation Today we find a large conceptual gap between the formal analytical concerns of an impact evaluation who generate “evidence” and the day to day implementation concerns of programming. We need a mechanism to bring implementers back into the analytical space in way that can inform programming, and fill some of the gap not touched by the Gold Standard

Yi = Formal Analysis for RCTs

bX = M&E Data for Performance Management

Unchartered ME&L Space

Analytical Space

Implementation Space

SOCHA Our Solution

Solution SoCha has offered: • Rigorously answer how questions through the formal logic of

qualitative comparative analysis, aka the “Silver Standard.”

• Incentivize “outside of the frame,” Enhanced M&E to collect information relevant to successful outcomes

Examples taken from a pre-primary education project in Tanzania, but this solution has wider application to other sectors

SOCHA

What is 3MSC? Background: 3MSC is our trademarking of “Qualitative Comparative Analysis” (QCA), which has spread from academia to policy evaluation. 3MSC refers to the application of QCA to development assistance.

3MSC represents a solution to the qualitative (rich data but limited generalizability, informal) vs. quantitative (generalizability but limited usefulness, formal) dilemma by combining features of both.

It is well suited for small-medium sample sizes (e.g. 15-300) and moves us beyond “single variable” explanations by thinking in terms of configurations and combinations.

3MSC identifies how context, various program components , and external factors combine to explain social change outcomes. 3MSC assumes that:

• Social change is not the result of a single factor, but of a combination of factors

• Different combinations of factors can produce the same outcome

• One factor can have different impacts on the outcome, depending on its combination with other factors and the context

• The lack of a factor can be just as important as the presence of one.

SOCHA

3MSC is the Silver Standard

3MSC is the Silver Standard for exploring “how” and “why” questions of social change, such as how context, external factors and complex programs influence outcomes, because:

Cheaper: Can answer how questions at a much lower price, e.g. with a much lower sample size

Formal: Relies upon a rigorous mathematical approach based upon Boolean Algebra to analyze the qualitative data, aka sets, through the application of mathematical principles.

Qualitative: 3MSC requires that we have an understanding of what we are trying to achieve and clear criteria for identifying when success is reached. This in turns requires familiarity with our cases, implementation awareness and technical expertise

But…..WE HAVE TO KNOW WHAT WE WANT AND WHAT SUCCESS LOOKS LIKE!

The Qualitative Definition of a successful vs. an unsuccessful outcome is CRUCIAL

SOCHA Boolean Algebra and M&E

For 3MSC, meaningful change is divided into sets of membership/nonmembership and analyzed with Boolean algebra, in which the values of the variables are the truth values true and false, usually denoted 1 and 0 respectively. We apply this to M&E by changing Indicators into Sets. Indicators: • aggregate; count the quantity of things, especially outputs directly controlled by

the project • define results in terms of “more is better”….more implementation means more

outputs, and assumes these outputs lead to outcomes, etc. 1+1=2

Sets: • categorical and based upon membership inside or outside of a category of

outcomes, 1 = membership; 0=non membership. • Sets also define results in terms of qualitative definitions of the success (=1) or

unsuccess (=0) 1+1=1

Boolean Algebra is a FORMAL logic and thus reduces individual biases

SOCHA

The Rules: Necessary and Sufficient Conditions

Boolean Algebra can use combinations of sets to identify the Necessary and/or Sufficient conditions/outputs for successful outcomes, e.g. early learning E.g. teacher training is necessary but not sufficient to improve early learning outcomes It also identifies how there are multiple paths to achieving the same outcome. E.g. teacher training, done in combination with deworming OR government education support, can improve early learning outcomes In other words, we can add complexity and still generalize. When conducted in cooperation with rigorous RCT/quasi experimental research designs, we gain a more complete picture of how and why positive social change occurs, i.e. it does not replace statistical analysis, but complements it.

Findings:

Conclusion 1: The program is necessary, but not sufficient, for success

Conclusion 2: Neither DE funding nor Deworming are necessary for success

Conclusion 3: The program MUST be implemented in combination with either District

Government OR External Donor Programs to be successful

3MSC™ Treatment/Control Analysis of how Program, District Education Funding and Deworming Presence relates to Early Learning

HH Identifier Program

Implemented? District Education

Funding? WHO

Deworming ?

Improved Learning Outcomes

Result

Schools in the Treatment Group (i.e. the program) s=150

A (62) 1 1 1 Success All factors present are sufficient, but not necessary

B (32) 1 1 0 Success Combining the program with DG funding is sufficient

C (24) 1 0 1 Success Combing the program with deworming is sufficient

D (34) 1 0 0 Unsuccess The program alone is insufficient but necessary

Schools in the Control Group (i.e. not in the program) s=150

E (49) 0 1 1 Unsuccess Combining Public Health with Deworming is insufficient

F (59) 0 1 0 Unsuccess DG funding is insufficient

G (27) 0 0 1 Unsuccess Deworming is insufficient

H (16) 0 0 0 Unsuccess Doing nothing is insufficient

Necessary and Sufficient Analysis with a Control/Treatment Design

SOCHA Expanding the Analysis: Tanzania Early Pre-Primary

In Tanzania, we recently looked at 5 “endogenous” factors, i.e. program components, and how they interact with 7 “exogenous but relevant” factors we believe are related to our outcome. This is used to test the assumptions of our pilot phase and inform full roll out. Outcome: Improved Preprimary Teacher Pedagogy Components: Teacher Training, Mentoring, Parent Partnering, Learning Kits, Head Teacher Training Exogenous School Factors: School Management Committees, Capitation Grants, Parent Teacher Associations Exogenous Environmental Factors: Local Government Awareness and Support of Pre-Primary, Local Civil Society Organizations that promote ECD/E In terms of operations, we staff a field monitoring team who apply a continuous but “light touch” approach to 300 schools.

SOCHA 3MSC can handle much larger “sets” of complexity to organize relevant data in a meaningful way,

e.g. exogenous school variables, remote and community factors, etc.

Tanzania Case Study

Expanding the Analysis: Tanzania Case Study

School ID Program Components Exogenous School Factors Remote Factors

Quality Teaching? Teacher

Training Mentoring

Parent Partnerships

Learning Kits

HT Training

SMC Capitation

Grant PTA

Districts Aware

Local PPE Advocacy

1 1 1 1 1 1 0 1 1 0 0 0

2 1 0 0 0 0 1 1 0 0 1 1

3 1 1 1 0 0 0 1 1 0 0 0

4 1 0 0 1 1 1 0 0 1 0 1

5 1 0 1 1 1 1 0 0 0 0 1

6 1 1 0 0 0 1 1 0 1 0 1

7 1 0 1 0 0 0 0 0 0 1 1

8 1 0 0 1 1 1 0 0 0 1 1

9 1 1 1 1 0 0 0 1 0 0 0

10 1 0 0 1 0 1 0 0 1 1 1

11 0 0 0 1 1 0 1 1 1 0 0

12 0 0 0 0 0 0 1 0 1 0 1

13 0 0 0 0 1 1 1 1 1 0 0

14 0 0 0 0 1 1 0 0 0 1 1

15 0 0 0 0 0 1 1 1 0 0 0

16 0 0 0 1 0 0 1 0 1 1 1

17 0 0 0 1 1 0 1 1 0 1 0

18 0 1 0 1 0 0 1 1 1 1 0

SOCHA Pilot Results on Improving Teaching Pedagogy

Exogenous Factors: • PPE Advocacy and District Awareness are neither necessary nor sufficient • Functioning SMCs and Capitation Grants form part of a set of sufficient

conditions, but only in combination with each other • The LACK of PTAs is part of a combination of factors associated with success Component Factors: • Head Teacher Training was not relevant to changes in pedagogy • Mentoring is necessary condition • Teaching is sufficient but not necessary • Learning kits plus mentoring is another combination, esp. when capitation

grants are lacking and SMCs stand alone • Parent partnering has no association with improved Teacher Pedagogy, but may

change when we shift the outcome to “school readiness” After the components were costed out, we found that optimal combinations involve combining Mentoring with Learning kits in schools that receive capitation grants and have strong SMCs

SOCHA Wave 2: Collaborative and Iterative

Wave Two will build on the Pilot Wave by collaboratively analyzing the results and raise new questions for 3MSC to analyze, i.e. it is both collaborative and iterative. Collaborative 3MSC only gives back configurations of conditions…i.e. it identifies the HOW, but doesn’t explain the WHY. 3MSC therefore requires feed back from Program Managers, Experts and Stakeholders to define/refine the categories, interpret the results, explain contradictions and identify new factors to be analyzed. Iterative Wave One will raise new questions (e.g. role of HTs). 3MSC requires a dynamic data collection mechanism that is incentivized to collect specific and relevant information that changes as the analysis raises new questions. Note: This does not require massive surveys, and can be answered with only a few additional cases.

SOCHA Conclusions and Implications

3MSC™ is a legitimate, mathematically based approach to formally answering “how” questions around complex program effectiveness. Moves the M&E paradigm: • Away from Results as increasing outputs • Toward Results as a meaningful change from one condition to another Changes our thinking about how we produce social change: • Gives us an analytical strategy for disentangling complexity while retaining the

ability to generalize • Allows us to explore variation in outcomes and avoid eliminating context • Facilitates “Configurational thinking” Insurance Policy for RCTs: • Builds upon the counter factual and sampling design to tell the story in a more

complete way • Explains anomalies and cases where the data doesn’t make sense • Disentangles the successful elements of an intervention from unsuccessful ones.

SOCHA

More information: Carroll Patterson cpatterson@sochaglobal.com www.sochaglobal.com +254 731 000 699 in Kenya +66 838 856 311 in Thailand