531_crowdscience_wp_online_surveys_0911

IT’S NOT WHAT YOU ASK, BUT HOW YOU ASK ITGo Beyond Basic Behavioral And Analytics Data

By Using Surveys To Really Understand The“Who” And The “Why” Of Your Site Visitors

AN ORANGE PAPER BY

ORANGE PAPER RESPONSE SCALES

© 2011 Crowd Science. All Rights [email protected]

2

TABLE Of CONTENTS

PART I: WHAT IS A SCALE? 4

PART II: WHY USE A RESPONSE SCALE? 5

PART III: CHOOSING A RESPONSE SCALE - GRANULARITY 6

PART IV: CHOOSING A RESPONSE SCALE - SCALE POINTS 8

PART V: CHOOSING A RESPONSE SCALE - BAGGAGE & BALANCE 10

CONCLUSION 12

ABOUT CROWD SCIENCE 13



3

Online Surveys. Everyone sees them and some even take them. Far less care about what happens with the results. At Crowd Science, we help create great online surveys. We enjoy taking them, too. And we certainly care about the results – we work with some of the most well known online companies to make sense of these results so companies are able to take action on the information.

The most important aspect of managing and growing an online content site is to consistently ask the right questions to the right people. The right questions provide the power to gain unique insight and then draw accurate conclusions about:

• Website performance• Site satisfaction• Visitor demographics• Audience growth• Monetization opportunities

Themosteffectiveandefficientwaytoinfluencetheseelementsistogainadeepdemographiclevel understanding of a particular website audience.

This position paper has been developed to shed some light on the importance of creating a high quality survey questionnaire. Since the inception of the Web, the Crowd Science team has had extensive online survey experience and learned a lot about survey best practices. One of the most important aspects of a survey questionnaire is the response scale. The response scale has a key role in the type and quality of the analysis that can be drawn from your survey research.

Sandra Marshall, Crowd Science VP of Research, explains why the response scale matters.



4

QUESTIONNAIRE DESIGN: RESPONSE SCALESQuestionnaires, in market research anyway, are somewhat like the eyes – a window to the soul. That may sound a bit lofty, but we rely heavily on questionnaires to access information on people’s attitudes and behavior. A well-designed questionnaire is vital to getting to the truth.

A poorly designed questionnaire will get you answers – just not necessarily the right ones. It’s that old adage of GIGO (garbage in, garbage out)… after the initial stages of research and sampling design have been addressed, it’s the questionnaire that can make – or break – the research. There are many aspects of questionnaire design, so in the interest of starting somewhere, let’s begin with a topic that incites a lot of discourse and debate: Scales – specifically,responsescales.

Discussions about other types of scales will best be addressed in future position papers.

PART I: WHAT IS A SCALE?Similar to scales used to measure things (like weight or height), surveys employ scales to provide a range or degree of information. Using a scale in a survey provides those who take the survey with a range of responses for each question – providing a richer understanding and level of detail.

Veryunlikely

1

Somewhatunlikely

2

Somewhatlikely

4

Verylikely

5

RESPONSE SCALENeither likelynor unlikely

3



5

PART II: WHY USE A RESPONSE SCALE?A key aspect of quantitative, closed-ended questionnaire design involves not only how the question is worded, but also how we design the response categories. Why ask one question to get one response, when you can ask one question to address several items? For example:

• Instead of asking whether a respondent plans to buy a new car in the next 12 months, ask the degree of likelihood to buy within the next 12 months.

• Ratherthanaskingwhetherarespondentismerelysatisfiedordissatisfiedwithaproduct or service, ask their level of satisfaction or dissatisfaction.

Scales that go beyond “this or that” response options open up a new set of possibilities to better understand respondents. The proper use of response scales can inform where risks lie and provide the opportunity to be in a better position to minimize them.

Shades of GreyA response scale offers more detail than a simple ‘black or white’ type of response. It digs into the range of space between the black and the white:

• Yes or No?• Do You or Don’t You?• Will You or Won’t You?

A scale is almost like having a dimmer switch – the respondent can dial up or dial down the intensity of their response. You end up with more detail on the degree of activity, intent, or attitude. You also have more options when it comes time to conduct the statistical analyses.

With this level of detail, smarter decisions can now be made on how or where to allocate resources and efforts to maintain and grow customer bases/audiences/etc.

So next time you catch yourself asking “whether”, see if the question translates into a scale format that now allows you to ask “how much”. It’ll go a long way in providing a full picture, and better answers.



6

PART III: CHOOSING A RESPONSE SCALE - GRANULARITYOnce you’ve decided to use a scale, you are now faced with the decision of what kind of scale to use, and what it should look like. There are many ways to present response scales, and each one has its particular strengths and purpose.

More Points = Greater SensitivityAs with everything in research, always go back to the objectives:

• What is the desired goal (or research hypothesis)?• How do you intend to use the results?

This will drive every aspect of questionnaire design, including the appropriate response scale to use.

For example, let’s say you want to understand customer satisfaction to track changes between specifictimeperiodswhicharefairlyshort(hourly,perhaps).Youmaythereforewanttheresponse scale to be more sensitive in order to pick up on the smallest of changes. This will requiretheuseofamorefinelytunedscale-onewithmorepointsonit.Thiswouldenableyouto act nimbly on any change that was recorded from hour to hour.

Butperhapsyoumaynotwanttobebotheredbyminutefluctuations,incaseswheretheymay be distracting, less meaningful, or in some way not worthwhile to act upon. Perhaps the measurementnormallyfluctuatesagreatdeal,andyouneedtoobtainasenseofthebiggerpicture,wherefluctuationsgobeyondthenorm.Inthesecases,asmallernumberofscalepoints might be in order. Again, there may be other factors affecting your decision when choosing the right scale range.

Even or Odd?When deciding on how many points to include on the scale, another decision to make is whether to have an odd or even number of points.

An odd number provides a midpoint, which may or may not be desirable, depending on your objectives, and how you intend to use the information. You may not want to use an odd number in order to avoid a neutral response. For example, perhaps you are measuring satisfaction, and youreallywanttoknowwhetheryourcustomersaresatisfied–youwanttoforcetheissueandget them to choose a positive or negative rating, without sitting on the fence. You may decide on a scale that includes the following even number of points:



7

Verydissatisfied

1

Somewhatdissatisfied

2

Somewhatsatisfied

3

Verysatisfied

4

EVEN NUMBER OF POINTS

Byofferingnopointbetweendissatisfiedandsatisfied,youforcetherespondenttocommittoone side or the other, presumably in order to know whether/how much you need to improve your offering.

However, let’s say you’re asking about some hypothetical future behavior, such as intent to purchase. It’s possible that the respondent truly does not know whether they will carry out this behavior,becausetheyrequiremoreinformation(costs,benefits,theirfutureneeds/situation,etc.) before making a decision.

Similarly, asking opinions on topics the respondent has never before encountered, means it is possible that they have not yet formed any opinion on the subject at all. In such cases a midpoint will offer a position where they can respond without committing to a positive or negative answer.

Verydissatisfied

1


2

Somewhatsatisfied

4

Verysatisfied

5

ODD NUMBER OF POINTSNeither satisfiednor dissatisfied

3

Alternatively, if an even-numbered scale is offered in addition to a separate “Don’t know” category, this could achieve the same result. Either way, it’s important to be sensitive to situations where forcing an answer one way or the other may be inappropriate.

In the end, when deciding on the number of points in your scale, think ahead to the analysis and how you want to present results, think about the nature of the measurement you’re taking, and think of the respondent experience in answering your question.



8

PART IV: CHOOSING A RESPONSE SCALE - SCALE POINTSAnchors Aweigh!So far, we’ve addressed why to use a response scale, and some things to consider in deciding the number of points in the scale. Now let’s discuss how to label the scale points.

A Picture Is Worth A Thousand WordsSome scales don’t even use verbal descriptions, and rely on diagrams to conveying meaning, such as simple line drawings of faces with various expressions (e.g., big smile, medium smile, straight-line mouth, medium frown, and big frown). These types of scales may be successful with audiences (like children) that arepreliterate,orforwhateverreasonhavedifficultyexpressing themselves with words.

Such situations may require an interviewer to assist in administering the questions if this is the case. However, even with respondents able to read the questionnaire, such scales may make the experience more fun or engaging, depending on the situation.

Too Many Words Spoil The BrothMostscalesusewordstodefinetheirpoints.Youmightconsiderputtingverbaldefinitionsonevery point in the scale (a.k.a. ‘fully anchored’), or to use a partially anchored scale, say, only definingthetwooppositeends.Somepeoplewonderaboutlabelingthetwoends,plusalabelfor the midpoint if there is one. A general rule of thumb is to fully anchor or label a scale if there arefiveorfewerpointsonit,andforscaleswithsixormorepoints,abipolaranchorcandonicely.

SCALE USING DIAGRAMS

Verydissatisfied

1


2

Verysatisfied

5

FULLY ANCHORED SCALENeither satisfiednor dissatisfied

3

Somewhatsatisfied

4

Verydissatisfied

1

3

6

Verysatisfied

7

PARTIALLY ANCHORED SCALE

4

2

5



9

There may be cases where scales using six or more points can still be completely anchored on all points, but in some cases this can become tedious for respondents to read through (or in telephone surveys, to have it read out loud for every question). On self-completed surveys, having so many labels may compromise the visual appeal, and the questionnaire may appear too cluttered, long, or confusing for the respondent.

Two’s Company, Three’s A CrowdYou may wonder if it is helpful, when using bipolar anchors on each end of a scale, to also label the midpoint to provide another point of reference for the respondent. You are arguably introducing risk by calling unnecessary attention to the midpoint. This emphasis on the midpoint may tempt respondents to use it as an ‘out’ from having to think about their response and provide an opinion. Midpoints are also tempting fallback for lazy or fatigued respondents, so the less attention to the halfway point in the scale, the better the results.

Verydissatisfied

1


3

Mostlysatisfied

6

Verysatisfied

7

FULLY ANCHORED 7 POINT SCALENeither satisfiednor dissatisfied

4

Mostlydissatisfied

2

Somewhatsatisfied

5



10

PART V: CHOOSING A RESPONSE SCALE - BAGGAGE & BALANCELose The Baggage, Keep The BalanceWhen constructing a response scale, it’s important to achieve a nice variety of response options across the scale. A scale that tends to consistently elicit responses skewed toward one very small part of the scale doesn’t tell you much. It would be like measuring agreement with the statement “adequate sleep contributes to good health”. Most people would agree this statement is true; there would probably be minimal variability in the results, either across time or across respondents.

On A Scale Of One To Ten…In many cases, the nature of the question causes responses to skew a certain way. But sometimes it is the nature of the scale that skews the data in a particular direction.

For example, consider 10-point scales, e.g. “On a scale of one (or zero) to ten, please rate your agreement/satisfaction/etc. with the following statement.”

The potential problem with a 10-point scale is that it carries a lot of ‘baggage’ from prior experience. This baggage can include familiarity with other questionnaires that have used this scale. It also can stem from the popularization of various commonly used 10-point rating or ranking systems. For example:

• David Letterman’s Top 10• The movie 10 [Hello, Bo!]• School tests scored out of 10• Casual discussions – “On a scale of one to ten, what did you think of that movie?”

You may be saying, “So what?” These examples demonstrate how respondents are already conditioned look at 10-point scales in a predetermined way. For instance, think of those tests or quizzes that everyone took as kids in school, graded out of ten. Here we have been conditioned tofocusontheupperendofthescale.Specifically,mostofthesetestsrequiredatleasta5to“pass”. Anything below 5 was a “fail”. It didn’t really matter by how much you failed, either - after all, if you failed the test, you failed!

There is little importance or meaning attributed to the differentiation of any scale points below 5. But for points 5 and upward, we attribute differences for each of the scores, even corresponding the numbers to different letter grades, “A” through “F”. This is one situation that can be conditioned into respondents from a fairly early age and contribute to our reluctance to use the lower part of the 10-point scale.



11

If You Don’t Have Anything Nice To Say….There is another piece of ‘baggage’ respondents can carry around: the desire to be nice. Don’t get me wrong - we’re not knocking politeness as a principle. But in the case of questionnaires, the desire to comply with socially acceptable practices can be detrimental to survey results.

Negative feedback is very necessary, because without it we don’t have the means to improve! Nevertheless, many people are reluctant to say bad things because it’s not the ‘nice’ thing to do. And if anything lower than the midpoint of a scale constitutes a ‘failing grade’ then that would be a pretty negative piece of feedback.

Lucky SevenAt Crowd Science, when the situation is appropriate, we prefer to use a 7-point scale, which offers granularity without baggage. Instead of the overused 10-point scale, we throw the respondent off the track and get them to think about things with a fresh approach – and without any baggage. Seven point scales offer a midpoint (for cases where we don’t want to force a positive or negative rating) and enough granularity to track movement from one situation or time to another. And best of all, there is less ‘bunching’ of responses that you would expect to see in the 10-point scale.

Verydissatisfied

1

3

6

Verysatisfied

7

7 POINT SCALE

4

2

5



12

CONCLUSIONIt’s one thing to ask a question. It’s equally, if not more, important to have the right scale to get responses that provide actionable data. Responses can be far more meaningful by making a few simple tweaks to the response scale.

The newly exposed ‘grey’ areas in the response set have tremendous value. By knowing how much or little ‘grey’ area your questionnaire responses have will alter the conclusion you can draw for the question. The number of options to choose from within a scale can also be a huge factor in the type of response you’re looking for. An odd number of responses or a ‘don’t know’ option will allow the respondents to take a neutral stand, while an even number of responses will force them to ‘pick a side.’ The number of options in the response set, along with the way the response options are worded, will drive the type of meaningful responses that you’re looking for.

Remember, it’s not what you ask, but how you ask it.

© 2011 Crowd Science. All Rights Reserved.

13

ABOUT CROWD SCIENCECrowd Science Inc. (http://crowdscience.com) is revolutionizing online market research by providing innovative solutions and technologies to online publishers and brands. Companies like Federated Media, Turner Network and Everyday Health use Crowd Science’s combination of web analytics and survey research to improve marketing and advertising precision. Contact us today at [email protected] or 1-650-625-9722.

OUR PRODUCTS

AUDIENCE PROfILERGet accurate intelligence about audiences that can’t be uncovered using standard analytics, behavioral inferences, or low quality surveys.• Built on SmartSample™ technology• Questionnaires pre-built by our researchers• Rich & statistically valid results

AUDIENCE TARGETERAudience Targeter provides publishers and networks with direct paths to segmenting, targeting and monetizing their audiences.• Measurement – collect deep, customized audience data• Segmentation – organize audiences into sellable subgroups• Optimization – bridge segmentation metrics with ad serving• Extension – grow frequency & reach via re-targeting

CAmPAIGN mAxImIzERDelivering real-time results that allow you to understand and report key brand campaign metrics:• Brand Lift – favorability & usage• Awareness – aided & unaided recall• Intent – purchase intent

SITE SATISfACTIONSite Satisfaction research provides publishers with actionable information about their audiences. Services include:• Full service program management• Cost-effective research & actionable deliverables• Innovative technology & real-time dashboards

ONLINE mARKET RESEARCHFull suite of traditional custom online research services, including:• Research consultation & design• Questionnaire design & programming• Datacollection&fielding• Reporting, analysis, & insights

http://crowdscience.com/

mailto:info%40crowdscience.com?subject=Response%20Scales%20Orange%20Paper

Date post:	28-Mar-2016
Category:	Documents
Upload:	interbrand
View:	214 times
Download:	0 times

531_crowdscience_wp_online_surveys_0911

Documents