Can you teach someone to be financially capable?

Results from meta-analysis of the impact evaluation literature

February 27, 2014

HIGH-LEVEL GLOBAL SYMPOSIUM ON FINANCIAL EDUCATION

Margaret MillerSenior Economist, Financial Inclusion and Infrastructure Practice

The World Bank Group

Organization for Talk

I. Literature review for financial literacy / capabilityII. Introduction to / rationale for meta-analysisIII. Descriptive statistics on the dataIV. Discussion of meta-analysis resultsV. Concluding remarks

Motivation for work on financial literacy / capability

• Significant body of evidence that people are making financial mistakes (Shui & Ausubel 2004; Gross & Souleles 2002; Firestone, Van Order and Zorn 2007)

• Research suggests that less financially literate / knowledgeable consumers are more prone to making financial errors (Lusardi & Tufano 2009; Stango & Zinman 2009)

• Regulations and consumer protection can help address some of these problems but there are limits (Gine and Mazer 2013; Christelis, Jappelli and Padula 2010; Willis 2011)

• This suggests a role for efforts to increase financial knowledge and skills – but can this be done effectively?

Literature reviews focus on different studies and provide inconclusive results

Pearson Correlation Table

Martin 20071.000

(1.000) 0.416 1.000

(0.173) (1.000) -0.067 0.001 1.000(0.004) (0.000) (1.000) 0.118 0.097 -0.017 1.000

(0.014) (0.009) (0.000) (1.000) 0.161 0.104 0.026 0.425 1.000

(0.026) (0.011) (0.001) (0.181) (1.000) 0.171 0.190 -0.088 0.174 0.047 1.000

(0.029) (0.036) (0.008) (0.030) (0.002) (1.000) -0.057 0.009 -0.060 0.075 0.036 0.084 1.000(0.003) (0.000) (0.004) (0.006) (0.001) (0.007) (1.000) -0.077 -0.099 -0.065 -0.088 -0.100 -0.083 -0.175 1.000(0.006) (0.010) (0.004) (0.008) (0.010) (0.007) (0.031) (1.000) -0.104 -0.038 -0.155 -0.105 -0.167 0.039 -0.002 -0.217 1.000(0.011) (0.001) (0.024) (0.011) (0.028) (0.002) (0.000) (0.047) (1.000)

Total Number of References 44 35 45 82 57 61 100 132 187

Xu and Zia 2012

Lusardi and Mitchell 2013

Agarwal et al 2010

Gale and Levine 2010

Hathaway and Khatiwada 2008

Martin 2007

Collins and O’Rourke 2010

McCormick 2009

Hastings, Madrian, Skimmyhorn 2012

Our approach: first focus more narrowly on interventions then cast a wide net

• Papers were included if they discussed the impact of a financial literacy / capability intervention

• Papers were identified through a variety of approaches:• Search of peer-reviewed articles in Econlit (search terms financial

capability, financial literacy and financial education)• Including all papers from leading literature surveys published over the

past 5 years• Impact evaluation papers funded through the Russia Trust Fund for

Financial Education and Literacy

Peer-Reviewed Jo

urnal

Academ

ic W

orkin

g Paper

Non-Aca

demic

Publicatio

ns & B

ooks0%

10%

20%

30%

40%

50%

28%

34%38%

Type of Publication(n = 188)

6%

9%

2%

6%

75%

2%

Region in which Intervention Conducted(n=188)

Africa

Asia

Europe

Latin America

USA

Other

Mortgage

Credit C

ounselin

g

Business

Managem

ent

Savings/

Retirem

ent

Mixed

Other

0%

10%

20%

30%

40%

50%

5%

9% 7%

30%

43%

6%

Topic of Financial Education (n=188)

Course ta

ught in pers

on

Individ

ual Counse

ling

Mass M

edia

PhonePrin

t

Online

MixedOth

er0%

10%

20%

30%

40%

50%

60% 54%

9%5% 3%

6%3%

18%

3%

Intervention Channel (n=188)

Location of Intervention(n = 188)

High School

University

Work

place

Comm

unity/Third

Party

Home-b

ased

Financia

l Insti

tutio

nMixed

Other

0%

5%

10%

15%

20%

25%

30%

14%

4%

19%

24%

8%

11%

14%

5%

≤2 3-6 7-10 11-20 21-50 >50 Varied0%

10%

20%

30%

40%

50%

16%

11% 11%11%

6%2%

43%

Intensity of Intervention (hours)(n = 148)

Duration of Intervention(n = 156)

≤ 1 week

1 week < x ≤ 1 m

onth

1 month

< x ≤ 6 month

s

6 month

s < x ≤ 1 year

> 1 year

Varied

0%

10%

20%

30%

40%

33%

5%

17%

5%7%

32%

Difficulties in using meta-analysis for this research question• Diverse interventions across many dimensions

• Type of intervention• Duration and intensity of exposure to intervention• Topic / objective of intervention• Target population

• Lack of uniformity in defining outcomes

• Lack of statistical rigor / information including many studies with small sample sizes

• Comparability of effect sizes is difficult• Clustering and additional controls vary across studies• Actual effect sizes/impact depend on control group means

Results of the meta-analysis• Comparable information only available for a few outcomes

(all binary variables)• Savings • Retirement savings • Default on loan • Record keeping

• Information also available on financial literacy test scores but not comparable due to diverse tests, scoring scales

NOTE: Weights are from random effects analysis

Overall (I-squared = 54.5%, p = 0.051)

Drexler Fischer Schoar

Authors

Bruhn Leão et al

Bruhn Ibarra McKenzie

Berg & Zia

Cole Sampson Zia

Doi McKenzie Zia

2011

Date

2013

2012

2013

2011

2012

Dominican Rep

Country

Brazil

Mexico

South Africa

Indonesia

Indonesia

15

Intensity

36

4

26

2

13

1

Channel

1

7

3

1

1

0.03 (0.00, 0.06)

0.14 (-0.02, 0.30)

ES (95% CI)

0.05 (0.04, 0.06)

0.03 (-0.01, 0.06)

-0.03 (-0.09, 0.03)

0.01 (-0.07, 0.09)

0.10 (0.00, 0.20)

100.00

3.73

%

Weight

36.13

24.33

15.98

11.54

8.29

0.03 (0.00, 0.06)

0.14 (-0.02, 0.30)

ES (95% CI)

0.05 (0.04, 0.06)

0.03 (-0.01, 0.06)

-0.03 (-0.09, 0.03)

0.01 (-0.07, 0.09)

0.10 (0.00, 0.20)

100.00

3.73

%

Weight

36.13

24.33

15.98

11.54

8.29

0-.297 0 .297

Papers testing savings behavior after financial education intervention

NOTE: Weights are from random effects analysis

Overall (I-squared = 87.7%, p = 0.000)

Hung Yoong

Muller

Authors

Beshears, Choi, Laibson, Madrian, Milkman

Bernheim & Garret

Lusardi and Mitchell - ALP article

2010

2003

Date

2011

2003

2007

USA

USA

Country

USA

USA

USA

Varied

Varied

Intensity

One letter sent

Varied

Seminars

2

1

Channel

5

7

NA

0.08 (0.01, 0.16)

0.06 (0.01, 0.11)

0.09 (-0.06, 0.24)

ES (90% CI)

-0.04 (-0.07, -0.01)

0.12 (0.07, 0.18)

0.23 (0.13, 0.33)

100.00

%

22.98

12.98

Weight

24.21

22.23

17.60

0.08 (0.01, 0.16)

0.06 (0.01, 0.11)

0.09 (-0.06, 0.24)

ES (90% CI)

-0.04 (-0.07, -0.01)

0.12 (0.07, 0.18)

0.23 (0.13, 0.33)

100.00

%

22.98

12.98

Weight

24.21

22.23

17.60

0-.327 0 .327

Papers testing retirement savings behavior after intervention

NOTE: Weights are from random effects analysis

Overall (I-squared = 69.1%, p = 0.012)

Sonobe Suzuki Otsuka Nam

Mano Akoten Otsuka Sonobe

Bruhn Ibarra McKenzie

Gibson McKenzie Zia

Authors

Doi Mckenzie Zia

2011

2010

2012

2012

Date

2012

Vietnam

Ghana / Kenya

Mexico

Australia/NZ

Country

Indonesia

NA

45

4

2

Intensity

13

1

1

7

1

Channel

1

0.04 (-0.00, 0.09)

0.06 (-0.00, 0.12)

0.18 (0.05, 0.31)

0.01 (-0.03, 0.04)

-0.05 (-0.11, 0.02)

ES (90% CI)

0.09 (0.04, 0.14)

100.00

21.11

9.16

27.17

19.07

%

Weight

23.49

0.04 (-0.00, 0.09)

0.06 (-0.00, 0.12)

0.18 (0.05, 0.31)

0.01 (-0.03, 0.04)

-0.05 (-0.11, 0.02)

ES (90% CI)

0.09 (0.04, 0.14)

100.00

21.11

9.16

27.17

19.07

%

Weight

23.49

0-.314 0 .314

Papers testing record keeping behaviors after intervention

NOTE: Weights are from random effects analysis

Overall (I-squared = 87.6%, p = 0.000)

Berg & Zia

Bruhn Ibarra McKenzie

Gine Mansuri

Agarwal et al

Authors

2013

2012

2011

2010

Date

South Africa

Mexico

Pakistan

USA

Country

26

4

46

35

Intensity

3

7

1,2

1

Channel

-0.02 (-0.06, 0.02)

0.04 (-0.04, 0.11)

0.02 (-0.01, 0.05)

-0.02 (-0.04, -0.00)

-0.09 (-0.12, -0.06)

ES (90% CI)

100.00

16.83

26.81

%

28.82

27.54

Weight

-0.02 (-0.06, 0.02)

0.04 (-0.04, 0.11)

0.02 (-0.01, 0.05)

-0.02 (-0.04, -0.00)

-0.09 (-0.12, -0.06)

ES (90% CI)

100.00

16.83

26.81

%

28.82

27.54

Weight

0-.115 0 .115

Papers testing loan defaults after intervention

Focus on Savings and Retirement Interventions

Assessing the findings• The small sample sizes that we ultimately have for the meta-

analysis limit our ability to draw conclusions

• Effect sizes are not comparable as typically data are not available for this estimation from the published works

• Studies using RCTs show limited impact of financial education interventions at best

• With these caveats, some insights do emerge:

Assessing the findings• Meta-analysis suggests that financial education can impact some financial

behaviors:• Increase saving in general and retirement saving in particular• Promote record keeping

• Descriptive analysis of savings and retirement interventions shows impact across a much larger portion of the literature than is included in the meta-analysis

• Results also suggest that it may be more useful in some circumstances than others • Not found to be significant factor for avoiding default – may be due to importance

of circumstances beyond borrower’s control (health, loss of job, etc.)

• Some surprises too – for example the intensity (hours of exposure) of the interventions was found to have statistical significance only for record-keeping • not conclusive evidence but worth further research

For the future• More attention to cost:benefit ratio: Virtually no discussion of

cost of the interventions or alternative methods to achieve the desired outcomes (Cole, Sampson, Zia 2011 is exception)

• Importance of strengthening / expanding use of rigorous evaluation methods

• Potential value from defining outcome variables in common terms (including use of binary dependent variables) to facilitate comparisons

• Importance of reporting more complete statistical information• Value from developing / contributing to a registry for financial

literacy interventions