Insights and Questions: Customer Response to the Home Electricity Reports

Results from SMUD’s Normative Messaging Pilot

Bruce CenicerosSacramento Municipal Utility District

2

Market Research and Evaluation EffortsBaseline behavior and attitude customer survey to pilot & control groups (March, 2008)

Report design customer focus groups (May, 2008)

Impact evaluation by SMUD Bus. Planning (October, 2008)

Participant survey (May, 2009)

Final behavior and attitude customer survey (June, 2009)

First-year evaluation by SMUD contractor (November, 2009)

Analysis of savings data by market segment (January, 2010)

Persistence study (2010)

Program Administrators

DISCLAIMER:Audience Response to this presentation

may vary

Researchers

4

Report Format

5

Experimental Design

Twelve month pilot (April 2008 thru March 2009)Randomly selected from within geographic pilot areas:

Control group: 50,000 residential customers Test group: 35,000 residential customers

Testing multiple report schemesMonthly vs. quarterlyGraphical vs. text-weighted designsDifferent envelope formats

6

Research Questions for 1-Year EvaluationHow much did participants save?What actions did they take?What, specifically, motivated them?How much of savings is from behavior versus equipment?How much of the savings resulted from participation in other SMUD programs?Who saved the most, and why? Who saved the least, and why? How did they feel about the reports? Did they read them? How didthey use them?

7

Impact Evaluation: Design• General Linear Model (GLM) controlling for:

– Selected housing characteristics– Billing time period (staggered)– Weather

• Monthly billing data:– 12 months pretest (April, 2007-March, 2008)– 12 month test period (April, 2008-March, 2009)

8

Impact Evaluation: Results

• 1.9% savings overall (1.866%-1.873%, 95% confidence)• Average savings: 213 kWh/year• 0.5% savings attributable to recipients who also took a

SMUD rebate or loan (represents maximum attributable to those programs).

• Worst case levelized cost 6.9 cents/kWh (assumes zero persistence, 1.4% savings, zero contribution where rebates were taken)

What is going on at Individual Level?

9

What is going on at Individual Level?

10

Long vacation

Major remodel

Unusual weather

Daughter leaves home

New medical equipment

Ancient AC unit dies

Job loss

Exchange student arrives

What is going on at Individual Level?

11

Pre-post Mail Survey

12

• Blind Study – respondents unaware study was affiliated with the reports• Pretest sent by mail to Control and Pilot groups before program launch to

establish baseline in first quarter 2008 • Post-test sent after one year to Control and Pilot groups to assess change• Margin of error ≈ +/- 3% at 95% confidence• 5,000 invitations/group/yr• Response rates: 20%-24% (4455 completed questionnaires)

Control Pilot

2008/Pre n=1155 n=1213

2009/Post n=1079 n=1008

Mail Survey: Topics Addressed

13

14

Pre-Post Results: What Changes Were Made?

MeasureTreatment Effect

Weight (kWh per year)

Annual Impact (kWh)

Recycled 2nd refrigerator/freezer 0.007 1161 8.13

Installed whole‐house fan ‐0.007 22 ‐0.15

Unplugged appliances not in use 0.076 100 7.6

Turned off PC when not in use 0.013 390 5.07

Total 20.6

Statistically significant actions account for only 9% of savings

15

Direct Mail Recall: Energy Conservation

47% 43%49%

73%

0%10%20%30%40%50%60%70%80%90%100%

"Yes"

2008 2009

Recall receiving energy conservation information in mail from SMUD not included in bill in past year.

ControlPilot

28% relative increase in Pilot group

16

Understanding Usage

34%38% 37%

45%

0%10%20%30%40%50%60%70%80%90%100%

"Stron

gly Agree"

2008 2009

SMUD helps me understand my electric usage.

ControlPilot

4% relative increase in Pilot group

17

Paying Attention to Use

60% 57% 57% 61%

0%10%20%30%40%50%60%70%80%90%100%

"Stron

gly Agree"

2008 2009

I pay attention to changes in the amount of electricity my household uses over time.

ControlPilot

7% relative increase in Pilot group

Reports did not Significantly Increase Awareness of EE Opportunities

18

19

“What ways can you think of to reduce energy use in your home?”

60% 58%51% 49%

0%10%

20%30%

40%50%

60%70%

80%90%

100%

2008 2009

Add or Improve Insulation in Attic, Walls, Pipes, or Water Heater

ControlPilot

20

10% 10% 12% 11%

0%10%20%30%40%50%60%70%80%90%100%

2008 2009

Replace Appliance With an Energy Star® or EE Appliance

Control

Pilot

“What ways can you think of to reduce energy use in your home?”

21

3% 4%11% 9%

0%10%20%30%40%50%60%70%80%90%100%

2008 2009

Keep Thermostat Down in Winter, Up in Summer

Control

Pilot

“What ways can you think of to reduce energy use in your home?”

22

Overall Pilot Impact• Reports may not increase customers’ awareness of

EE opportunities• Instead, the reports may remind them to engage

more often in EE behaviors with which they are already familiar.

23

Customer satisfaction did not suffer, but did not improve

64%

26%

7%2% 1%

0%10%20%30%40%50%60%70%80%90%100%

VerySatisfied

SomewhatSatisfied

Indifferent SomewhatDissatisfied

VeryDissatisfied

Overall, how satisfied are you with SMUD?

24

Participant Survey: Methodology• Mailed in May, 2009 to 5,0001 pilot participants• n=1,375 • Margin of error = +2.2% at 95% confidence level• Respondents told survey affiliated with the Home Electricity

Reports • A visual and narrative depiction of the report was included with

the cover letter to assist in recall

25

Participant Survey: Highlights• 98% recall receiving the reports• 90% read all or most of the reports, most carefully• 70% find them easy to understand• 67% find them valuable• Customers are talking about the reports• Customers aren’t clear about how their neighbors are selected and

have mixed feelings about the relevance of the group and the appropriateness of the comparison.

26

Research Questions for 1-Year EvaluationHow much did participants save?What actions did they take?What, specifically, motivated them?How much of savings is from behavior versus equipment?How much of the savings resulted from participation in other SMUD programs?Who saved the most, and why? Who saved the least, and why? How did they feel about the reports? Did they read them? How didthey use them?

27

Research Questions for 1-Year EvaluationHow much did participants save?What actions did they take?What, specifically, motivated them?How much of savings is from behavior versus equipment?How much of the savings resulted from participation in other SMUD programs?Who saved the most, and why? Who saved the least, and why? How did they feel about the reports? Did they read them? How did they use them?

28

Research Questions for 1-Year EvaluationHow much did participants save?What actions did they take?What, specifically, motivated them?How much of savings is from behavior versus equipment?How much of the savings resulted from participation in other SMUD programs?Who saved the most, and why? Who saved the least, and why? How did they feel about the reports? Did they read them? How didthey use them?

29

Research Questions for 1-Year EvaluationHow much did participants save?What actions did they take?What, specifically, motivated them?How much of savings is from behavior versus equipment?How much of the savings resulted from participation in other SMUD programs?Who saved the most? Who saved the least? How did they feel about the reports? Did they read them? How didthey use them?

30

Research Questions for 1-Year EvaluationHow much did participants save?What actions did they take?What, specifically, motivated them?How much of savings is from behavior versus equipment?How much of the savings resulted from participation in other SMUD programs?Who saved the most? Who saved the least? How did they feel about the reports? Did they read them? How didthey use them?How long does savings persist? How well does it persist when you curtail the reports?

31

Recommendations: Study DesignInnovation needed!Deeper understanding may require combination of data collection methods linked to individual customers:

Measure/behavior saturation survey (RAS?)Survey of changes in household that may account for changes in energy useEnd use load surveys (data loggers)In-home observations vs self-reports?

32

Recommendations: Beginning“Begin with the end in mind”Treat baseline data collection as equally important to post-test dataBring in evaluation and market research experts during program design to:

Provide input on program design (“evaluability”)Design an efficient and effective evaluation roadmapCollect the right baseline dataTell you the questions to ask to optimize targeting and marketing

33

Recommendations: Selection ProcessBudget evaluation work wiselyDesign selection process to ensure a highly-qualified contractor gets the job

Meticulous specificationsHigh weighting or threshold criteria for experience and qualificationsConsider multi-stage process (RFQ)

34

Recommendations: CollaborationJob too big for any one organization to tackle?Collaborate, distribute efforts (think Human Genome, IPCC, SETI)Develop consensus protocols for evaluating behavior programs (a la CA Standard Practice Manual)Develop master plan, each take pieces of puzzle to solveConsider complementary project pilots

Test different strategies, formats, combinationsMeasure different things

Things will only get more complicatedNormative messages

Time-based rates

New bill presentment

Smart GridCommitments

Demand response

In-Home Displays

Awareness campaigns

Contact Information

Bruce CenicerosProgram Planner(916) 732-6747

bcenice@smud.org

Wim BosProgram Evaluation Manager

(916) 732-6579wbos@smud.org