1

Randomized Control Trials for Business Solutions:

Putting Them to Work for You

Jon ZinmanDartmouth College

October 6, 2006

2

Research Team Members

• Dean Karlan, Yale University

• Sendhil Mullainathan, Harvard University

3

Overview of Talk

1. Examples of product spaces where academics can help CUs improve profits and member satisfaction (using randomized-control trials)

– Focus on mortgages, small business lending• Where $ is!• Biggest opportunity for comparative advantage. Little

academic and consultant attention compared to, e.g., saving, financial planning, credit card use

2. Methodology:– How we do it– Operational requirements

4

Example 1: Mortgage Menus

• Too much choice depresses consumer demand– Psychology studies– Bertrand-Karlan-Mullainathan-Shafir-Zinman (2005)

showed this in an experiment with an actual consumer finance company

• But what’s the “right” menu of choices?– “Right” depends on our partner institutions’ objectives,

but for now let’s say:• Profitable• Sustainable (long-run member satisfaction)

5

Example 1: Mortgage Menus

• Here is a hard choice:– Choice A: 7-1 ARM, no points at 7.5% vs. – Choice B: Fixed rate, no points, at 8%.

• However, if you add choice C,• Choice C: Fixed rate, 2 points, at 8.05%

• B looks much more attractive and becomes easy choice

6

Outline of a Mortgage Menus Experiment

Experiment:

1) Identify small set of promising menus

2) Run horserace in course of actual ops

3) Analyze results and identify best menu(s)

7

Example 2: Refinancing

• Evidence suggests that mortgagees procrastinate or don’t pay attention, therefore don’t finance at right time

• Business implication. Help members refi at right time and: – Profit (new clients, member retention, more

frequent refis)– Increase client satisfaction

8

Outline of a Refi Experiment

Marketing for attention.• Small cues, messages drive clients choices• Our prior work shows this. Showing a female

photo on direct mail increased consumer loan takeup among dormant male clients by about same amount as a 50% drop in the interest rate

• Tap social networks (potential comparative advantage for CUs)

• So test a set of promising cues/messages via direct mail, phone

9

Outline of Another Refi Experiment

Experiment with automatic refis.– Help members choose refi triggers in advance

• comparative advantages here for CUs: trust, personal touch

– Offer binding and non-binding commitments– Test takeup rate, profitability, ex-post member

satisfaction for a few automatic refi options

10

Example 3:Home equity loan/line utilization

• Seems low given:– Simultaneous holdings with relatively

expensive credit card debt– # of homeowners who have substantial card

debt and no HELOC or HEloan

11

Outline of Home Equity Experiment

• Use psychological principles to drive use of equity lines (instead of more expensive forms of borrowing)

• Marketing with:– Salient rate comparisons– Cues (picture of family, home improvements)– Make mental accounting work for HELOCs

(“put your interest savings back into your house, kids’ education, etc.”)

12

Example 4: Pricing Riskon Small Business Loans

• How do you know where to draw the line?• Some lenders leave profitable deals on table.

Example from Karlan-Zinman (2006):– Experiment on high-risk consumer loans– Added step to Lender’s normal underwriting process:

applicants who were rejected but close to the bar got a loan 25% or 50% of the time (randomly assigned)

– These loans ended up being profitable– Also measurably positive impacts on clients

• Another method we use for identifying profitability frontier: credit scoring for small businesses

13

What we Mean by “Experiment”:Randomized-Control Trials (RCTs)

• What they are:– Identical members or prospective clients get

different offers– Offers are randomly assigned (“drawn out of a

hat”) within a reasonable, predefined range• range is guided by business considerations

14

Why RCTs?

• RCTs are “gold standard” method for identifying causality in social, medical science; biz applications

• Highly practical/profitable biz proposition– Our work with smaller firms– Larger firms have integrated RCTs into routine

marketing and merchandizing operations, face-to-face contact

• Credit card companies• Amazon• Food/drug retailers• H&R Block

15

RCTs: Step-by-Step1. Identify product space and key questions

2. Design experiment to answer key questions

3. Implement design into operations• Piloting• Full launch

4. Monitor implementation for compliance with design

5. Analyze data from experiment– Data sharing: consent and disclosure issues

6. Apply results of experiment

16

RCT Step #1:Identify Area(s) for Experimentation • Process: focused brainstorming

• Players: researchers, senior mgmt.

• Outcomes:– choice of product spaces– outlines of experimental design like ones I

gave earlier

17

RCT Step #2.Design experiment

• Process: take design outline, meld it to operational realities of firm.• Players: research team, senior mgmt, field/area managers, front-line

personnel, IT personnel• Examples:

– Ensure data sharing feasible: consent, disclosures required?– Designing direct mailers– Coordinating experimental direct mailers with routine marketing efforts– Training loan officers to offer a new automatic refi product. Say some clients will

get direct mail or face-to-face offer of this new product, other won’t. Then loan officers need to know:

• Content of new product(s)• Presentation• Who gets the offer? (compliance with randomization)

– Usually involves a software add-on to MIS that loan officer would reference for each client, and automatically prompt a new product offer, or regular product offer

• Staffing requirements:– Research team: we bring a project manager– CU: may want dedicated project manager (depends on scale)

18

RCT Step #3:Implement design into operations

3a. Pilot experiment

• Small scale

• Work out kinks:– Design tweaks– Operational tweaks (compliance, clunkiness)– Fix/eliminate backfires (member complaints,

staff burdens)

3b. Full launch

19

RCT Step #4:Monitor Compliance with Design

• Audits/site visits for any face-to-face components

• Statistical tracking

20

RCT Step #5:Analyze Data from Experiment

• Researchers generate a report– Upfront issue: make sure can hand

researchers the requisite data needed for analysis

• What consent, disclosures are needed, if any?

21

Step #6:Apply Results of Experiment

• What have we learned, and what should the CU do next?– Discussions between researchers, mgmt– Lead to recommendations from researchers

about how to apply the results:• Specific product, pricing, marketing decisions• Strategy and positioning• Opportunities for additional experiments

22

Take-Aways and Next Steps

• RCTs work: standard- and best-practice for answering key strategic questions

• Research team has some ideas for high-impact projects

• We are looking for CU partners

• So let me know if you have interest in RCTs & would like to explore opportunities