2010 Research on RespondentstVox DataMay, 2010

© Harris Interactive

Presentation to MRIA Ottawa Chapter

Doug Anderson, SVP Public AffairsHarris/Decima160 Elgin StreetSuite 160Ottawa, ON K2P 2P7danderson@harrisdecima.com

Richard Leigh-Bennett, CMRP SVPHarris/Decima160 Elgin StreetSuite 160Ottawa, ON K2P 2P7rleighbennett@harrisdecima.com

What do we know – Response Rates

• The PMRS Response Rate Committee measured refusal rates in 1995, 1999, 2002 and again in 2005 and found that refusal rates have increased and response rates have fallen.

• When analyzed on an increment basis year by year, the 2002 survey suggested that for one-time studies, the rate of refusals was accelerating.

• Telephone surveys have been under attack recently on the grounds that “Results are no longer accurate nor representative” and low response rates are cited as the reason.

© Harris Interactive

Non-response Rate

• “The collective body of empirical work suggests no consistent relationship between response rate and non-response bias. RDD telephone surveys with low response rates generally still have excellent demographic representativeness.” (Keeter et al. 2006).

Questions we had….• With declining response rates, are we just getting the same

respondents who do multiple surveys or are we actually getting different people but they are harder to reach?– Number of calls received asking to do a phone survey and

number of times agreed to do the survey in past year. – Likelihood to complete a phone survey in the next year.

• Online coverage is coming close to telephone coverage, but who is doing online surveys?– Number of times participated in any sort of on-line survey.– Membership in an on-line panel.– Likelihood to complete an online survey in the next year.

Methodology

• Data collection for this study was conducted via CATI ….

• Data collection for this study was conducted April 29th to May 11th, 2010.

• Findings are based on a sample of 2008 adult Canadians. A sample of this size yields a margin of error of +/-2.2% nineteen times out of twenty.

© Harris Interactive

Findings

Estimated Requests and Interviews (12 Months)

© Harris Interactive

Reasons for Declining Telephone Survey Request

© Harris Interactive

Base: Those who declined a telephone survey (n=959)

Reasons for Accepting Telephone Survey Request

© Harris Interactive

Base: Those offering a reason (n=1,642)

Number of Calls Received

32%

28%

30%

22%

18%

32%

42%

22%

28%

32%

30%

42%

43%

35%

44%

33%

40%

44%

39%

41%

39%

40%

19%

19%

21%

20%

29%

19%

12%

27%

21%

18%

20%

7%

10%

14%

14%

21%

9%

3%

12%

10%

11%

10%

0% 20% 40% 60% 80% 100%

55+

35-54

18-34

B.C.

Prairies

Ontario

Quebec

Atlantic

Women

Men

Overall

Only This Call 2 to 4 5 to 10 11+

Number of Calls Received (Mean)

4.09

4.80

5.35

5.79

7.21

4.34

2.92

5.56

4.71

4.74

4.73

0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 5.50 6.00 6.50 7.00 7.50 8.00

55+

35-54

18-34

B.C.

Prairies

Ontario

Quebec

Atlantic

Women

Men

Overall

Mean

Number of Telephone Surveys Completed

46%

40%

47%

35%

35%

47%

51%

42%

40%

47%

44%

39%

44%

38%

46%

38%

40%

41%

38%

43%

39%

41%

13%

12%

12%

16%

20%

11%

8%

16%

14%

11%

13%

2%

3%

3%

4%

7%

2%

4%

3%

3%

3%

0% 20% 40% 60% 80% 100%

55+

35-54

18-34

B.C.

Prairies

Ontario

Quebec

Atlantic

Women

Men

Overall

Only This Call 2 to 4 5 to 10 11+

Number of Telephone Surveys Completed (Mean)

2.71

2.94

2.69

3.11

3.93

2.55

2.16

3.17

2.90

2.69

2.80

0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00

55+

35-54

18-34

B.C.

Prairies

Ontario

Quebec

Atlantic

Women

Men

Overall

Mean

Claimed Rate of Conversion

67%

60%

56%

56%

46%

60%

76%

54%

61%

60%

61%

5%

6%

5%

8%

10%

5%

2%

5%

6%

5%

5%

15%

19%

18%

17%

20%

19%

14%

18%

16%

18%

17%

13%

15%

22%

19%

24%

17%

8%

23%

16%

17%

17%

0% 20% 40% 60% 80% 100%

55+

35-54

18-34

B.C.

Prairies

Ontario

Quebec

Atlantic

Women

Men

Overall

All Calls 75%-99% 50%-74% <50%

Claimed Rate of Conversion (Mean)

0.82

0.79

0.74

0.77

0.71

0.78

0.88

0.74

0.80

0.78

0.79

0.00 0.25 0.50 0.75 1.00

55+

35-54

18-34

B.C.

Prairies

Ontario

Quebec

Atlantic

Women

Men

Overall

Mean

Observed Request : Conversion Ratio1.24

1.25

1.12

1.36

1.26

1.25 1.31

1.27

1.15

1.35

1.3

1.19

1.53

1.31

1.16

1.46

1.23

1.21

1.45

1.25

1.17

1.5

1.24

1.15

1.48

1.33

1.16

1.41

1.38

1.1

1

1.25

1.5

1.75

2A

TL

M 1

8-3

4A

TL

M 3

5-5

4A

TL

M 5

5+

AT

L W

18

-34

AT

L W

35

-54

AT

L W

55

+

QU

E M

18

-34

QU

E M

35

-54

QU

E M

55

+Q

UE

W 1

8-3

4Q

UE

W 3

5-5

4Q

UE

W 5

5+

ON

T M

18

-34

ON

T M

35

-54

ON

T M

55

+O

NT

W 1

8-3

4O

NT

W 3

5-5

4O

NT

W 5

5+

PR

M 1

8-3

4P

R M

35

-54

PR

M 5

5+

PR

W 1

8-3

4P

R W

35

-54

PR

W 5

5+

BC

M 1

8-3

4B

C M

35

-54

BC

M 5

5+

BC

W 1

8-3

4B

C W

35

-54

BC

W 5

5+

Observed Request : Conversion Ratio1.53

1.5

1.48

1.46

1.45

1.41

1.38

1.36

1.35

1.33

1.31

1.31

1.3

1.27

1.26

1.25

1.25

1.25

1.24

1.24

1.23

1.21

1.19

1.17

1.16

1.16

1.15

1.15

1.12

1.1

1

1.25

1.5

1.75O

NT

M 1

8-3

4P

R W

18

-34

BC

M 1

8-3

4O

NT

W 1

8-3

4P

R M

18

-34

BC

W 1

8-3

4B

C W

35

-54

AT

L W

18

-34

QU

E W

18

-34

BC

M 3

5-5

4Q

UE

M 1

8-3

4O

NT

M 3

5-5

4Q

UE

W 3

5-5

4Q

UE

M 3

5-5

4A

TL

W 3

5-5

4A

TL

M 3

5-5

4A

TL

W 5

5+

PR

M 3

5-5

4A

TL

M 1

8-3

4P

R W

35

-54

ON

T W

35

-54

ON

T W

55

+Q

UE

W 5

5+

PR

M 5

5+

ON

T M

55

+B

C M

55

+Q

UE

M 5

5+

PR

W 5

5+

AT

L M

55

+B

C W

55

+

Claimed Request : Conversion Ratio1.85

1.51

1.231.27

1.23 1.31

1.19

1.15

1.11.2

1.13

1.08

1.46

1.22

1.23

1.26 1.36

1.19

1.66

1.36

1.36

1.68

1.241.38

1.19 1.251.38

1.31 1.41

1.29

1

1.25

1.5

1.75

2A

TL

M 1

8-3

4A

TL

M 3

5-5

4A

TL

M 5

5+

AT

L W

18

-34

AT

L W

35

-54

AT

L W

55

+

QU

E M

18

-34

QU

E M

35

-54

QU

E M

55

+Q

UE

W 1

8-3

4Q

UE

W 3

5-5

4Q

UE

W 5

5+

ON

T M

18

-34

ON

T M

35

-54

ON

T M

55

+O

NT

W 1

8-3

4O

NT

W 3

5-5

4O

NT

W 5

5+

PR

M 1

8-3

4P

R M

35

-54

PR

M 5

5+

PR

W 1

8-3

4P

R W

35

-54

PR

W 5

5+

BC

M 1

8-3

4B

C M

35

-54

BC

M 5

5+

BC

W 1

8-3

4B

C W

35

-54

BC

W 5

5+

Claimed Request : Conversion Ratio1.85

1.68

1.66

1.51

1.46

1.41

1.38

1.38

1.36

1.36

1.36

1.31

1.31

1.29

1.27

1.26

1.25

1.24

1.23

1.23

1.23

1.22

1.20

1.19

1.19

1.19

1.15

1.13

1.10

1.08

1.00

1.25

1.50

1.75

2.00A

TL

M 1

8-3

4P

R W

18

-34

PR

M 1

8-3

4A

TL

M 3

5-5

4O

NT

M 1

8-3

4B

C W

35

-54

BC

M 5

5+

PR

W 5

5+

PR

M 3

5-5

4O

NT

W 3

5-5

4P

R M

55

+B

C W

18

-34

AT

L W

55

+B

C W

55

+A

TL

W 1

8-3

4O

NT

W 1

8-3

4B

C M

35

-54

PR

W 3

5-5

4A

TL

M 5

5+

ON

T M

55

+A

TL

W 3

5-5

4O

NT

M 3

5-5

4Q

UE

W 1

8-3

4B

C M

18

-34

ON

T W

55

+Q

UE

M 1

8-3

4Q

UE

M 3

5-5

4Q

UE

W 3

5-5

4Q

UE

M 5

5+

QU

E W

55

+

Expected Future Trend

17%

14%

15%

12%

15%

17%

17%

11%

16%

56%

63%

65%

68%

64%

65%

49%

64%

61%

19%

19%

17%

15%

18%

12%

29%

18%

18%

8%

4%

2%

5%

3%

5%

5%

7%

5%

0% 20% 40% 60% 80% 100%

55+

35-54

18-34

B.C.

Prairies

Ontario

Quebec

Atlantic

Overall

Less Same More DK/NR

In the next 12 months, do you think you will be more likely, less likely, or just as likely to complete a phone survey, if asked??

9%

13%

17%

14%

7%

19%

15%

16%

16%

73%

64%

57%

67%

65%

50%

63%

59%

61%

16%

18%

21%

15%

23%

23%

17%

19%

18%

2%

5%

5%

4%

5%

8%

4%

5%

5%

0% 20% 40% 60% 80% 100%

>$100k

$60k-$100k

<$60k

Univ/Coll

Some PS

HS or Less

Female

Male

Overall

Less Same More DK/NR

Who has done any sort of online survey in past year?

25%

40%

45%

41%

44%

37%

29%

33%

36%

43%

51%

51%

46%

44%

49%

50%

53%

48%

31%

10%

5%

13%

12%

14%

21%

14%

15%

0% 20% 40% 60% 80% 100%

55+

35-54

18-34

B.C.

Prairies

Ontario

Quebec

Atlantic

Overall

Did at least 1 Online, but did none Offline

In the past 12 months, that is since May 2009, how many times have you agreed to participate in any sort of online survey conducted over the Internet?

53%

47%

25%

44%

18%

18%

37%

36%

36%

45%

48%

51%

43%

48%

48%

45%

52%

48%

2%

5%

25%

13%

34%

34%

18%

12%

15%

0% 20% 40% 60% 80% 100%

>$100k

$60k-$100k

<$60k

Univ/Coll

Some PS

HS or Less

Female

Male

Overall

Did at least 1 Online, but did none Offline

Who has joined online panels?

8%

16%

18%

17%

15%

14%

12%

14%

14%

62%

74%

77%

71%

73%

72%

67%

72%

71%

31%

9%

5%

12%

12%

14%

21%

14%

15%

0% 20% 40% 60% 80% 100%

55+

35-54

18-34

B.C.

Prairies

Ontario

Quebec

Atlantic

Overall

Joined Online, but Never Joined Offline

Have you ever joined an online group, community or panel for the purpose of completing surveys over the Internet?

18%

16%

12%

17%

22%

8%

16%

12%

14%

79%

79%

64%

76%

66%

58%

66%

76%

71%

2%

5%

24%

8%

12%

34%

18%

12%

15%

0% 20% 40% 60% 80% 100%

>$100k

$60k-$100k

<$60k

Univ/Coll

Some PS

HS or Less

Female

Male

Overall

Joined Online, but Never Joined Offline

Number of Panels Joined

© Harris Interactive

Expected Future Trends

23%

16%

65%

61%

11%

18%

1%

5%

0% 20% 40% 60% 80% 100%

Online panelists

Telephone

Less Same More DK/NR

More questions…• The toughest strata to reach by phone are among the most participatory

online.– Is dove-tailing samples a way to help, or is it more risk than benefit?

• We build/weight both phone and online samples to look like Census profiles.– If we know that either of the two are not truly identical to the Census

profile and vary in identifiable ways, would we design samples to reflect these limitations?

• Phone response rates have continued to decline, but perhaps that decline has slowed. Our findings show as online panels grow, response rates also decline.– Should we read anything into the responses on respondents’

expectations for the future?