Angels and Demons: Is your survey penalizing the right people?

transcript

Festival of NewMR 2013 – Main Stage

Sponsor logos will go here

Angels and Demons: Are you penalizing the right people?

Annie Pettit, Chief Research Officer

Peanut Labs

Canada

Raise your hand if right now you are…

04/11/2023 2

Lazy Bum Larry

Good Faith Gail

Method to our madness

• 8000 people answered a survey

• Every survey measured against each of 30 data quality measures

• Calculated number of errors for each person (from 0 to 30)

• Grouped people into Angels and Demons based on number of “fails”

Are responders angels or demons?

77%Zero errors

One or more errors

1 Red Herring

• 5.8% of angels chose 1 red herring

• 14.5% of demons chose 1 red herring

Why?• Sounds like a brand they know• It IS a local brand they know• Babies biting ankles

2 Red Herrings

• 1.4% of angels chose 2 red herrings

• 40.2% of demons chose 2 red herrings

Why?• They didn’t notice it. Twice.• They aren’t reading it. At all.• They need to finish it. NOW!

Tier 1 Speeding (<50% of the mean)

• 8.4% of angels speed • 40.7% of demons speed

Why?• They’re computer literate.• They read well. Really well.• They know surveys. Really well.

Tier 2 Speeding (Fastest 1.5%)

• 0% of angels speed • 28% of demons speed

Why?• They’re good at clicking.• They aren’t reading. At all.• They know surveys. Really well.

Straightlining

• 14.2% of angels straightline

• 60.7% of demons straightline

Why?• Every item was phrased the same way• It’s their true opinion

Good vs. Bad Grids

• Tastes great• Looks appetizing• Smells nice• Well packaged• Interesting logo• Pretty colours

• Tastes terrible• Looks appetizing• Smells strange• Well packaged• Odd logo• Pretty colors

What is your level of agreement with each of the following statement? This product:

Strongly Agree

Somewhat Agree

Neutral Somewhat Disagree

Strongly Disagree

Straightlining on A Good Grid

• 2.8% of angels straightline

• 58.4% of demons straightline

Why?• They aren’t reading. At all.• They know surveys. Really well.

Rank From 1 through 6

• 7.9% of angels rank incorrectly

• 50% of demons rank incorrectly

– Choose same number twice– Choose numbers outside the

limits– Straightline

Why?• They aren’t reading. At all.• They click really well.

Contradictions

• 28.4% of angels contradict themselves only once

• 39.5% of angels contradict themselves twice

• 7.9% of demons contradict themselves only once

• 73.8% of demons contradict themselves twice

Why?• Because it’s a valid answer• Babies biting ankles

Verbatim Length

• 0.9% of angels share less than 10 characters

• 56.5% of demons share less than 10 characters

Why?• asdf• aaaa• don’t know

Non-traditional behaviours

• 12.5% of angels buy non-grocery items at grocery stores

• 37.4% of demons buy non-grocery items at grocery stores

Why?• Because it’s a valid answer• Babies biting ankles

High Incidence Awareness

• 3.5% of angels knew <3 of 8 big name brands

• 48.1% of demons knew <3 of 8 big name brands

Why?• Immigrants• Non consumers• Non shoppers

Conclusions

Angels don’t exist

Perfection doesn’t exist

It’s not fair to expect perfection from responders

acting in good faith

Conclusions

To Penalize Angels:• Use 1 red herring• Use only positive grids• Measure only one or two

contradictions• Don’t permit 1 or 2 mistakes

To Penalize Demons:• Use 2 red herrings• Use mixed grids• Use a rank question• Measure short verbatims• Focus on multiple mistakes

Thank you!

Contact me with any questions.

Annie Pettit, PhDChief Research OfficerPeanut Labs

annie.pettit@peanutlabs.comwww.peanutlabsmedia.com416.273.9395

twitter.com/LoveStats

facebook.com/AnniePettit

ca.linkedin.com/in/AnniePettit

From sampling to surveys to listening, you need to get your research done quickly and accurately. Peanut Labs tools help you build your own surveys and select targeted sample from our profiled panel. And, our Evolisten social media listening tool helps you gain a broader understanding of your consumers.

Use our DIY, permission based, profiled sampling tool to select and launch targeted sample

Use our DIY survey design tool to craft your surveys

Obtain quick poll results with our DIY targeted polling tool. Answers in mere minutes!

Listen to millions of social media opinions about people, places, brands, and more.

www.peanutlabs.com

Angels and Demons: Is your survey penalizing the right people?

Marketing