+ All Categories
Home > Documents > HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction...

HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction...

Date post: 19-Dec-2015
Category:
View: 242 times
Download: 1 times
Share this document with a friend
Popular Tags:
109
HCI 510 : HCI Methods I Context of Use
Transcript
Page 1: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

HCI 510 : HCI Methods I• Context of Use

Page 2: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

HCI 510: HCI Methods I

• Questionnaires

– Introduction

– Sampling Techniques

– Development Process

– Sample Selection

– Survey Goals

– Guidelines

– Asking Questions

Page 3: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

HCI 510: HCI Methods I

• Questionnaires

– Introduction

– Sampling Techniques

– Development Process

– Sample Selection

– Survey Goals

– Guidelines

– Asking Questions

Page 4: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

This is the information age.

More information has been published in the last decade than in all previous history.

Everyone uses information to make decisions about the future.

If our information is accurate, we have a high probability of making a good decision.

If our information is inaccurate, our ability to make a correct decision is diminished.

Better information usually leads to better decisions.

Page 5: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

There are university course that do nothing but teach survey/questionnaire design.

Industry courses in questionnaire design usually last around 3 days and cost anything from $1500 to $4000.

This lecture will be using information from four main sources :

Walonick, D. S., Survival Statistics, StatPac, 2004.

Questionnaire Design : http://www.cc.gatech.edu/classes/cs6751_97_winter/Topics/quest-design/

Questionnaire Design and Survey Sampling :http://home.ubalt.edu/ntsbarsh/stat-data/Surveys.htm

Survey Design :http://www.surveysystem.com/sdesign.htm

Page 6: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

The main idea of statistical inference is to take a random sample from a population and then to use the information from the sample to make inferences about particular population characteristics such as the mean (measure of central tendency), the standard deviation (measure of spread) or the proportion of units in the population that have a certain characteristic.

Sampling saves money, time, and effort.

Additionally, a sample can, in some cases, provide as much information as a corresponding study that would attempt to investigate an entire population-careful collection of data from a sample will often provide better information than a less careful study that tries to look at everything.

Page 7: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

HCI 510: HCI Methods I

• Questionnaires

– Introduction

– Sampling Techniques

– Development Process

– Sample Selection

– Survey Goals

– Guidelines

– Asking Questions

Page 8: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Page 9: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Personal Interviews

An interview is called personal when the Interviewer asks the questions face-to-face with the Interviewee.

Personal interviews can take place in the home, at a shopping mall, on the street, outside a movie theater or polling place, and so on.

Page 10: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Personal Interviews

Advantages

• The ability to let the Interviewee see, feel and/or taste a product.

• The ability to find the target population. For example, you can find people who have seen a film much more easily outside a theater in which it is playing than by calling phone numbers at random.

• Longer interviews are sometimes tolerated. Particularly with in-home interviews that have been arranged in advance. People may be willing to talk longer face-to-face than to someone on the phone.

Page 11: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Personal Interviews

Disadvantages

• Personal interviews usually cost more per interview than other methods. This is particularly true of in-home interviews, where travel time is a major factor.

• Each mall has its own characteristics. It draws its clientele from a specific geographic area surrounding it, and its shop profile also influences the type of client. These characteristics may differ from the target population and create a non-representative sample.

Page 12: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Telephone Surveys

Surveying by telephone is the most popular interviewing method in the USA.

This is made possible by nearly universal coverage (96% of homes have a telephone).

Page 13: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Telephone Surveys

Advantages

• People can usually be contacted faster over the telephone than with other methods. If the Interviewers are using CATI (computer-assisted telephone interviewing), the results can be available minutes after completing the last interview.

• You can dial random telephone numbers when you do not have the actual telephone numbers of potential respondents.

• Skilled interviewers can often elicit longer or more complete answers than people will give on their own to mail, email surveys (though some people will give longer answers to Web page surveys). Interviewers can also ask for clarification of unclear responses.

Page 14: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Telephone Surveys

Disadvantages

• Many telemarketers have given legitimate research a bad name by claiming to be doing research when they start a sales call. Consequently, many people are reluctant to answer phone interviews and use their answering machines to screen calls. Since over half of the homes in the USA have answering machines, this problem is getting worse.

• The growing number of working women often means that no one is home during the day. This limits calling time to a "window" of about 6-9 p.m. (when you can be sure to interrupt dinner or a favorite TV program).

• You cannot show or sample products by phone.

Page 15: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Mail Surveys

Advantages

• Mail surveys are among the least expensive.

• This is the only kind of survey you can do if you have the names and addresses of the target population, but not their telephone numbers.

• The questionnaire can include pictures - something that is not possible over the phone.

• Mail surveys allow the respondent to answer at their leisure, rather than at the often inconvenient moment they are contacted for a phone or personal interview. For this reason, they are not considered as intrusive as other kinds of interviews.

Page 16: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Mail Surveys

Disadvantages

• Time! Mail surveys take longer than other kinds. You will need to wait several weeks after mailing out questionnaires before you can be sure that you have gotten most of the responses.

• In populations of lower educational and literacy levels, response rates to mail surveys are often too small to be useful. Even in well-educated populations, response rates vary from as low as 3% up to 90%. As a rule of thumb, the best response levels are achieved from highly-educated people and people with a particular interest in the subject (which, depending on your target population, could lead to a biased sample).

Page 17: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Mail Surveys

One way of improving response rates to mail surveys is to mail a postcard telling your sample to watch for a questionnaire in the next week or two. Another is to follow up a questionnaire mailing after a couple of weeks with a card asking people to return the questionnaire. The downside is that this doubles or triples your mailing cost. If you have purchased a mailing list from a supplier, you may also have to pay a second (and third) use fee - you often cannot buy the list once and re-use it.

Another way to increase responses to mail surveys is to use an incentive. One possibility is to send a dollar bill (or more) along with the survey. Another possibility is to include the people who return completed surveys in a drawing for a prize. A third is to offer a copy of the (non-confidential) result highlights to those who complete the questionnaire. Any of these techniques will increase the response rates.

Page 18: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Computer Direct Interviews

These are interviews in which the Interviewees enter their own answers directly into a computer.

They can be used at malls, trade shows, offices, and so on.

The Survey System's optional Interviewing Module and Interview Stations can easily create computer-direct interviews.

Some researchers set up a Web page survey for this purpose.

Page 19: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Computer Direct Interviews

Advantages

• The virtual elimination of data entry and editing costs.

• You will get more accurate answers to sensitive questions.

• Recent studies of potential blood donors have shown respondents were more likely to reveal HIV-related risk factors to a computer screen than to either human interviewers or paper questionnaires.

• The National Institute of Justice has also found that computer-aided surveys among drug users get better results than personal interviews.

• Employees are also more often willing to give more honest answers to a computer than to a person or paper questionnaire.

Page 20: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Computer Direct Interviews

Advantages

• The elimination of interviewer bias. Different interviewers can ask questions in different ways, leading to different results. The computer asks the questions the same way every time.

• Ensuring skip patterns are accurately followed. The Survey System can ensure people are not asked questions they should skip based on their earlier answers. These automatic skips are more accurate than relying on an Interviewer reading a paper questionnaire.

• Response rates are usually higher. Computer-aided interviewing is still novel enough that some people will answer a computer interview when they would not have completed another kind of interview.

Page 21: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Computer Direct Interviews

Disadvantages

• The Interviewees must have access to a computer or one must be provided for them.

• As with mail surveys, computer direct interviews may have serious response rate problems in populations of lower educational and literacy levels. This method may grow in importance as computer use increases.

Page 22: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Email Surveys

Email surveys are both very economical and very fast.

More people have email than have full Internet access.

This makes email a better choice than a Web page survey for some populations.

On the other hand, email surveys are limited to simple questionnaires, whereas Web page surveys can include complex logic.

Page 23: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Email Surveys

Advantages

• Speed. An email questionnaire can gather several thousand responses within a day or two.

• There is practically no cost involved once the set up has been completed.

• You can attach pictures and sound files.

• The novelty element of an email survey often stimulates higher response levels than ordinary “snail” mail surveys.

Page 24: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Email Surveys

Disadvantages

• You must possess (or purchase) a list of email addresses.

• Some people will respond several times or pass questionnaires along to friends to answer.

• Many people dislike unsolicited email even more than unsolicited regular mail. You may want to send email questionnaires only to people who expect to get email from you.

Page 25: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Email Surveys

Many email programs are limited to plain ASCII text questionnaires and cannot show pictures.

Although use of email is growing very rapidly, it is not universal - and is even less so outside the USA (three-quarters of the world's email traffic takes place within the USA).

Many “average” citizens still do not possess email facilities, especially older people and those in lower income and education groups.

So email surveys do not reflect the population as a whole.

At this stage they are probably best used in a corporate environment where email is common or when most members of the target population are known to have email.

Page 26: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Internet/Web Surveys

Web surveys are rapidly gaining popularity.

They have major speed, cost, and flexibility advantages, but also significant sampling limitations.

These limitations make sample selection especially important and restrict the groups you can study using this technique.

Page 27: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Internet/Web Surveys

Advantages

• Web page surveys are extremely fast. A questionnaire posted on a popular Web site can gather several thousand responses within a few hours. Many people who will respond to an email invitation to take a Web survey will do so the first day, and most will do so within a few days.

• There is practically no cost involved once the set up has been completed. Large samples do not cost more than smaller ones (except for any cost to acquire the sample).

• You can show pictures. Some Web surveys can also show video and play sound.

• Web page questionnaires can use complex question skipping logic, randomizations and other features not possible with paper questionnaires or most email surveys. These features can assure better data.

Page 28: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Internet/Web Surveys

Advantages

• Web page questionnaires can use colors, fonts and other formatting options not possible in most email surveys.

• A significant number of people will give more honest answers to questions about sensitive topics, such as drug use or sex, when giving their answers to a computer, instead of to a person or on paper.

• On average, people give longer answers to open-ended questions on Web page questionnaires than they do on other kinds of self-administered surveys.

• Some Web survey software can combine the survey answers with pre-existing information you have about individuals taking a survey.

Page 29: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Internet/Web Surveys

Disadvantages

• Current use of the Internet is far from universal. Internet surveys do not reflect the population as a whole. This is true even if a sample of Internet users is selected to match the general population in terms of age, gender and other demographics.

• People can easily quit in the middle of a questionnaire. They are not as likely to complete a long questionnaire on the Web as they would be if talking with a good interviewer.

• If your survey pops up on a web page, you often have no control over who replies - anyone from Antartica to Zanzibar, cruising that web page may answer.

• Depending on your software, there is often no control over people responding multiple times to bias the results.

Page 30: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Internet/Web Surveys

At this stage it is recommended to use the Internet for surveys mainly when your target population consists entirely or almost entirely of Internet users.

Business-to-business research and employee attitude surveys can often meet this requirement. Surveys of the general population usually will not.

Another reason to use a Web page survey is when you want to show video or both sound and graphics. A Web page survey may be the only practical way to have many people view and react to a video.

In any case, be sure your survey software prevents people from completing more than one questionnaire. You may also want to restrict access by requiring a password (good software allows this option) or by putting the survey on a page that can only be accessed directly (i.e., there are no links to it from other pages).

Page 31: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Summary of Survey Methods

Page 32: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

HCI 510: HCI Methods I

• Questionnaires

– Introduction

– Sampling Techniques

– Development Process

– Sample Selection

– Survey Goals

– Guidelines

– Asking Questions

Page 33: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Development Process

Questionnaires are an inexpensive way to gather data from a potentially large number of respondents.

Often they are the only feasible way to reach a number of reviewers large enough to allow statistically analysis of the results.

A well-designed questionnaire that is used effectively can gather information on both the overall performance of the test system as well as information on specific components of the system.

If the questionnaire includes demographic questions on the participants, they can be used to correlate performance and satisfaction with the test system among different groups of users.

Page 34: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Development Process

It is important to remember that a questionnaire should be viewed as a multi-stage process beginning with definition of the aspects to be examined and ending with interpretation of the results.

Every step needs to be designed carefully because the final results are only as good as the weakest link in the questionnaire process.

Although questionnaires may be cheap to administer compared to other data collection methods, they are every bit as expensive in terms of design time and interpretation.

Page 35: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Development Process

The steps required to design and administer a questionnaire include:

1. Defining the Objectives of the survey

2. Determining the Sampling Group

3. Choose Interviewing Method

4. Writing the Questionnaire

5. Pre-test the Questionnaire

6. Administering the Questionnaire

7. Interpretation of the Results

Page 36: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Development Process

Walonick states that:

Questionnaire research design proceeds in an orderly and specific manner.

Each item in the flow chart depends upon the successful completion of all the previous items.

Therefore, it is important not to skip a single step.

Notice that there are two feedback loops in the flow chart to allow revisions to the methodology and instruments.

Page 37: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Development Process

Page 38: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Development Process

Page 39: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

HCI 510: HCI Methods I

• Questionnaires

– Introduction

– Sampling Techniques

– Development Process

– Sample Selection

– Survey Goals

– Guidelines

– Asking Questions

Page 40: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

What Can Questionnaires Measure ?

Questionnaires are quite flexible in what they can measure, however they are not equally suited to measuring all types of data.

We can classify data in two ways,

Subjective vs. Objective

and

Quantitative vs. Qualitative

Page 41: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

What Can Questionnaires Measure ?

When a questionnaire is administered, the researchers control over the environment will be somewhat limited.

This is why questionnaires are inexpensive to administer.

This loss of control means the validity of the results are more reliant on the honesty of the respondent.

Consequently, it is more difficult to claim complete objectivity with questionnaire data then with results of a tightly controlled lab test.

For example, if a group of participants are asked on a questionnaire how long it took them to learn a particular function on a piece of software, it is likely that they will be biased towards themselves and answer, on average, with a lower than actual time.

Page 42: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

What Can Questionnaires Measure ?

A more objective usability test of the same function with a similar group of participants may return a significantly higher learning time.

More elaborate questionnaire design or administration may provide slightly better objective data, but the cost of such a questionnaire can be much higher and offset their economic advantage.

In general, questionnaires are better suited to gathering reliable subjective measures, such as user satisfaction, of the system or interface in question.

Page 43: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

What Can Questionnaires Measure ?

Questions may be designed to gather either qualitative or quantitative data.

By their very nature, quantitative questions are more exact then qualitative.

For example, the word "easy" and "difficult" can mean radically different things to different people.

Any question must be carefully crafted, but in particular questions that assess a qualitative measure must be phrased to avoid ambiguity.

Qualitative questions may also require more thought on the part of the participant and may cause them to become bored with the questionnaire sooner.

In general, we can say that questionnaires can measure both qualitative and quantitative data well, but that qualitative questions require more care in design, administration, and interpretation.

Page 44: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

When to Use a Questionnaires ?

There is no all encompassing rule for when to use a questionnaire.

The choice will be made based on a variety of factors including the type of information to be gathered and the available resources for the experiment.

Page 45: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

When to Use a Questionnaires ?

A questionnaire should be considered in the following circumstances.

1. When resources and money are limited.

A Questionnaire can be quite inexpensive to administer.

Although preparation may be costly, any data collection scheme will have similar preparation expenses. The administration cost per person of a questionnaire can be as low as postage and a few photocopies.

Time is also an important resource that questionnaires can maximize. If a questionnaire is self-administering, such as a e-mail questionnaire, potentially several thousand people could respond in a few days. It would be impossible to get a similar number of usability tests completed in the same short time.

Page 46: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

When to Use a Questionnaires ?

A questionnaire should be considered in the following circumstances.

2. When it is necessary to protect the privacy of the participants.

Questionnaires are easy to administer confidentially.

Often confidentiality is the necessary to ensure participants will respond honestly if at all.

Examples of such cases would include studies that need to ask embarrassing questions about private or personal behavior.

Page 47: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

When to Use a Questionnaires ?

A questionnaire should be considered in the following circumstances.

3. When corroborating other findings.

In studies that have resources to pursue other data collection strategies, questionnaires can be a useful confirmation tools.

More costly schemes may turn up interesting trends, but occasionally there will not be resources to run these other tests on large enough participant groups to make the results statistically significant.

A follow-up large scale questionnaire may be necessary to corroborate these earlier results.

Page 48: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Selecting Sample

There are two main components in determining whom you will interview.

The first is deciding what kind of people to interview.

Researchers often call this group the target population.

If you conduct an employee attitude survey or an association membership survey, the population is obvious.

If you are trying to determine the likely success of a product, the target population may be less obvious.

Correctly determining the target population is critical.

If you do not interview the right kinds of people, you will not successfully meet your goals.

Page 49: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Selecting Sample

There are two main components in determining whom you will interview.

The next thing to decide is how many people you need to interview.

Statisticians know that a small, representative sample will reflect the group from which it is drawn.

The larger the sample, the more precisely it reflects the target group.

However, the rate of improvement in the precision decreases as your sample size increases.

For example, to increase a sample from 250 to 1,000 only doubles the precision.

You must make a decision about your sample size based on factors such as: time available, budget and necessary degree of precision.

Page 50: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Sample Bias

A biased sample will produce biased results.

Totally excluding all bias is almost impossible; however, if you recognize bias exists you can intuitively discount some of the answers.

The consequences of a source of bias depend on the nature of the survey.

For example, a survey for a product aimed at retirees will not be as biased by daytime interviews as will a general public opinion survey.

A survey about Internet products can safely ignore people who do not use the Internet.

Page 51: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Page 52: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Page 53: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Quotas

A Quota is a sample size for a sub-group.

It is sometimes useful to establish quotas to ensure that your sample accurately reflects relevant sub-groups in your target population.

For example, men and women have somewhat different opinions in many areas.

If you want your survey to accurately reflect the general population's opinions, you will want to ensure that the percentage of men and women in your sample reflect their percentages of the general population.

Page 54: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Quotas

If you are interviewing users of a particular type of product, you probably want to ensure that users of the different current brands are represented in proportions that approximate the current market share.

Alternatively, you may want to ensure that you have enough users of each brand to be able to analyze the users of each brand as a separate group.

Page 55: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

HCI 510: HCI Methods I

• Questionnaires

– Introduction

– Sampling Techniques

– Development Process

– Sample Selection

– Survey Goals

– Guidelines

– Asking Questions

Page 56: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Goals

The importance of well-defined objectives can not be over emphasized.

A questionnaire that is written without a clear goal and purpose is inevitably going to overlook important issues and waste participants' time by asking useless questions.

The questionnaire may lack a logical flow and thereby cause the participant to lose interest. Consequential, what useful data you may have collected could be further compromised.

The problems of a poorly defined questionnaire do not end here, but continue on to the analysis stage. It is difficult to imagine identifying a problem and its cause, let alone its solution, from responses to broad and generalizing questions.

In other words, how would it be possible to reach insightful conclusions if one didn't actually know what they had been looking for or planning to observe.

Page 57: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Goals

A objective such as "to identify points of user dissatisfaction with the interface and how these negatively affect the software's performance" may sound clear and to the point, but it is not.

The questionnaire designer must clarify what is meant by user dissatisfaction. Is this dissatisfaction with the learning of the software, the power of the software, of the ease of learning the software? Is it important for the users to learn the software quickly if they learn it well? What is meant by the software's performance? How accurate must the measurements be?

All of these issues must be narrowed and focused before a single question is formulated.

A good rule of thumb is that if you are finding it difficult to write the questions, then you haven't spent enough time defining the questionnaire objectives. Go back and do this step again. The questions should follow quite naturally from the objectives.

Page 58: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Goals

One of the best ways to clarify your study goals is to decide how you intend to use the information. Do this before you begin designing the study.

This sounds obvious, but many researchers neglect this task. Why do research if the results will not be used?

Be sure to commit the study goals to writing.

Whenever you are unsure of a question, refer to the study goals and a solution will become clear.

Ask only questions that directly address the study goals.

Avoid the temptation to ask questions because it would be "interesting to know".

Page 59: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

HCI 510: HCI Methods I

• Questionnaires

– Introduction

– Sampling Techniques

– Development Process

– Sample Selection

– Survey Goals

– Guidelines

– Asking Questions

Page 60: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Questionnaire Guidelines

Keep it Short …

As a general rule, with only a few exceptions, long questionnaires get less response than short questionnaires. Keep your questionnaire short. In fact, the shorter the better.

Response rate is the single most important indicator of how much confidence you can place in the results. A low response rate can be devastating to a study.

Therefore, you must do everything possible to maximize the response rate. One of the most effective methods of maximizing response is to shorten the questionnaire.

If your survey is over a few pages, try to eliminate questions. Many people have difficulty knowing which questions could be eliminated. For the elimination round, read each question and ask, "How am I going to use this?" If the information will be used in a decision-making process, then it's important. If not, throw it out.

Page 61: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Questionnaire Guidelines

Use the Experts…

One important way to assure a successful survey is to include other experts and relevant decision-makers in the questionnaire design process.

Their suggestions will improve the questionnaire and they will subsequently have more confidence in the results.

Prepare the Analysis…

Formulate a plan for doing the statistical analysis during the design stage of the project.

Know how every question will be analyzed and be prepared to handle missing data.

If you cannot specify how you intend to analyze a question or use the information, do not use it in the survey.

Page 62: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Questionnaire Guidelines

Grab the Attention …

We all know how important first impressions are. The same holds true for questionnaires. The respondent's first impression of the study usually comes from the envelope containing the survey – so make it unique.

Provide a well-written cover letter. The respondent's next impression comes from the cover letter. The importance of the cover letter should not be underestimated. It provides your best chance to persuade the respondent to complete the survey.

Give your questionnaire a title that is short and meaningful to the respondent. A questionnaire with a title is generally perceived to be more credible than one without.

Also, ensure it looks professional – desktop published, modern typefaces.

Page 63: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Questionnaire Guidelines

Clear and Concise…

Include clear and concise instructions on how to complete the questionnaire. These must be very easy to understand, so use short sentences and basic vocabulary. Be sure to print the return address on the questionnaire itself.

Use simple and direct language. The questions must be clearly understood by the respondent. The wording of a question should be simple and to the point. Do not use uncommon words or long sentences. Make items as brief as possible.

This will reduce misunderstandings and make the questionnaire appear easier to complete.

One way to eliminate misunderstandings is to emphasize crucial words in each item by using bold, italics or underlining.

Page 64: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Questionnaire Guidelines

Keep Their Interest…

Begin with a few non-threatening and interesting items. If the first items are too threatening or "boring", there is little chance that they will complete the questionnaire. Make them want to continue by putting interesting questions first.

Place the most important items in the first half of the questionnaire. Respondents often send back partially completed questionnaires. By putting the most important items near the beginning, the partially completed questionnaires will still contain important information.

Hold the respondent's interest. We want the respondent to complete our questionnaire. One way to keep a questionnaire interesting is to provide variety in the type of items used. Varying the questioning format will also stop respondents from falling into "response sets".

At the same time, it is important to group items into coherent categories.All items should flow smoothly from one to the next.

Page 65: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Questionnaire Guidelines

Why Should I Do This …

Provide incentives as a motivation for a properly completed questionnaire.

What does the respondent get for completing your questionnaire?

Altruism is rarely an effective motivator.

Attaching a dollar bill to the questionnaire works well.

If the information you are collecting is of interest to the respondent, offering a free summary report is also an excellent motivator.

Whatever you choose, it must make the respondent want to complete the questionnaire.

Page 66: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Questionnaire Guidelines

Does It Work…

The final test of a questionnaire is to try it on representatives of the target audience.

If there are problems with the questionnaire, they almost always show up here.

If possible, be present while a respondent is completing the questionnaire and tell him/her that it is okay to ask you for clarification of any item.

The questions s(he) asks are indicative of problems in the questionnaire (i.e., the questions on the questionnaire must be without any ambiguity because there will be no chance to clarify a question when the survey is sent out).

Page 67: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

HCI 510: HCI Methods I

• Questionnaires

– Introduction

– Sampling Techniques

– Development Process

– Sample Selection

– Survey Goals

– Guidelines

– Asking Questions

Page 68: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Writing the Questionnaire

At this point, we assume that we have already decided what kind of data we are to measure, formulated the objectives of the investigation, and decided on a participant group. Now we must compose our questions.

Most questionnaires, however, also gather demographic data on the participants. This is used to correlate response sets between different groups of people. It is important to see whether responses are consistent across groups.

For example, if one group of participants is noticeably less satisfied with the test interface, it is likely that the interface was designed without fair consideration of this group's specific needs. This may signify the need for fundamental redesign of the interface.

In addition, certain questions simply may only be applicable to certain kinds of users. For example, if one is asking the participants whether they find the new tutorial helpful, we do not want to include in our final tally the responses of experienced users who learned the system with an older tutorial.

Page 69: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Writing the Questionnaire

Typically, demographic data is collected at the beginning of the questionnaire, but such questions could be located anywhere or even scattered throughout the questionnaire.

One obvious argument in favor of the beginning of the questionnaire is that normally background questions are easier to answer and can ease the respondent into the questionnaire. One does not want to put off the participant by jumping in to the most difficult questions. We are all familiar with such kinds of questions.

It is important to ask only those background questions that are necessary. There is often only a fine line between background and personal information. You do not want to cross over in to the personal realm unless absolutely necessary. If you need to solicit personal information, phrase your questions as unobtrusively as possible to avoid ruffling your participants and causing them to answer less than truthfully.

Page 70: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Types of Questions

In general, there are two types of questions one will ask, open format or closed format.

Page 71: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Types of Questions

Open format questions are those that ask for unprompted opinions.

In other words, there are no predetermined set of responses, and the participant is free to answer however he chooses.

Open format questions are good for soliciting subjective data or when the range of responses is not tightly defined.

An obvious advantage is that the variety of responses should be wider and more truly reflect the opinions of the respondents.

This increases the likelihood of you receiving unexpected and insightful suggestions, for it is impossible to predict the full range of opinion.

It is common for a questionnaire to end with and open format question asking the respondent for her unabashed ideas for changes or improvements.

Page 72: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Types of Questions

Open format questions have several disadvantages.

First, their very nature requires them to be read individually.

There is no way to automatically tabulate or perform statistical analysis on them.

This is obviously more costly in both time and money, and may not be practical for lower budget or time sensitive evaluations.

They are also open to the influence of the reader, for no two people will interpret an answer in precisely the same way.

This conflict can be eliminated by using a single reader, but a large number of responses can make this impossible.

Finally, open format questions require more thought and time on the part of the respondent.

Whenever more is asked, the chance of tiring or boring the respondent increases.

Page 73: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Types of Questions

Closed format questions usually take the form of a multiple-choice question.

There is no clear consensus on the number of options that should be given in an closed format question.

Obviously, there needs to be sufficient choices to fully cover the range of answers but not so many that the distinction between them becomes blurred. Usually this translates into five to ten possible answers per questions.

For questions that measure a single variable or opinion, such as ease of use or liability, over a complete range (easy to difficult, like to dislike), conventional wisdom says that there should be an odd number of alternatives. This allows a neutral or no opinion response.

Other schools of thought contend that an even number of choices is best because it forces the respondent to get off the fence. This may induce the some inaccuracies for often the respondent may actually have no opinion. However, it is equally arguable that the neutral answer is over utilized, especially by bored questionnaire takers..

Page 74: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Types of Questions

Closed format questions offer many advantages in time and money.

By restricting the answer set, it is easy to calculate percentages and other hard statistical data over the whole group or over any subgroup of participants.

Modern scanners and computers make it possible to administer, tabulate, and perform preliminary analysis in a matter of days.

Closed format questions also make it easier to track opinion over time by administering the same questionnaire to different but similar participant groups at regular intervals.

Finally closed format questions allow the researcher to filter out useless or extreme answers that might occur in an open format question.

Page 75: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Types of Questions

The following shows three basic types of questions: multiple choice, numeric open end and text open end (sometimes called "verbatims").

Page 76: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Types of Questions

Rating Scales and Agreement Scales are two common types of questions that some researchers treat as multiple choice questions and others treat as numeric open end questions.

Page 77: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Evokes the truth.

Page 78: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Evokes the truth.

Questions must be non-threatening.

When a respondent is concerned about the consequences of answering a question in a particular manner, there is a good possibility that the answer will not be truthful.

Anonymous questionnaires that contain no identifying information are more likely to produce honest responses than those identifying the respondent.

If your questionnaire does contain sensitive items, be sure to clearly state your policy on confidentiality.

Page 79: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Asks for an answer on only one dimension.

Page 80: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Asks for an answer on only one dimension.

The purpose of a survey is to find out information. A question that asks for a response on more than one dimension will not provide the information you are seeking.

For example, a researcher investigating a new food snack asks "Do you like the texture and flavor of the snack?" If a respondent answers "no", then the researcher will not know if the respondent dislikes the texture or the flavor, or both.

Another questionnaire asks, "Were you satisfied with the quality of our food and service?" Again, if the respondent answers "no", there is no way to know whether the quality of the food, service, or both were unsatisfactory.

A good question asks for only one "bit" of information.

Page 81: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Can accommodate all possible answers.

Page 82: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Can accommodate all possible answers.

Multiple choice items are the most popular type of survey questions because they are generally the easiest for a respondent to answer and the easiest to analyze.

Asking a question that does not accommodate all possible responses can confuse and frustrate the respondent.

Page 83: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Can accommodate all possible answers.

What brand of computer do you own? __ A. IBM PC B. Apple

Clearly, there are many problems with this question.

What if the respondent doesn't own a microcomputer?

What if he owns a different brand of computer?

What if he owns both an IBM PC and an Apple?

There are two ways to correct this kind of problem.

Page 84: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Can accommodate all possible answers.

The first way is to make each response a separate dichotomous item on the questionnaire. For example:

Do you own an IBM PC? (circle: Yes or No) Do you own an Apple computer? (circle: Yes or No)

Another way to correct the problem is to add the necessary response categories and allow multiple responses. This is the preferable method because it provides more information than the previous method.

What brand of computer do you own? (Check all that apply) __ Do not own a computer __ IBM PC __ Apple __ Other

Page 85: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Has mutually exclusive options.

Page 86: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Has mutually exclusive options.

A good question leaves no ambiguity in the mind of the respondent. There should be only one correct or appropriate choice for the respondent to make. An obvious example is:

Where did you grow up? __ A. country B. farm C. city

A person who grew up on a farm in the country would not know whether to select choice A or B.

This question would not provide meaningful information.

Worse than that, it could frustrate the respondent and the questionnaire might find its way to the trash.

Page 87: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Produces variability of responses.

Page 88: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Produces variability of responses.

When a question produces no variability in responses, we are left with considerable uncertainty about why we asked the question and what we learned from the information. If a question does not produce variability in responses, it will not be possible to perform any statistical analyses on the item. For example: What do you think about this report? __ A. It's the worst report I've read B. It's somewhere between the worst and best C. It's the best report I've read

Since almost all responses would be choice B, very little information is learned.

Are you against drug abuse? (circle: Yes or No)

Again, there would be very little variability in responses and we'd be left wondering why we asked the question in the first place.

Page 89: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Follows comfortably from the previous question.

Page 90: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Follows comfortably from the previous question.

Writing a questionnaire is similar to writing anything else.

Transitions between questions should be smooth.

Grouping questions that are similar will make the questionnaire easier to complete, and the respondent will feel more comfortable.

Questionnaires that jump from one unrelated topic to another feel disjointed and are not likely to produce high response rates.

Page 91: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Does not presuppose a certain state of affairs.

Page 92: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Does not presuppose a certain state of affairs.

Among the most subtle mistakes in questionnaire design are questions that make an unwarranted assumption. An example of this type of mistake is:

Are you satisfied with your current auto insurance? (Yes or No)

This question will present a problem for someone who does not currently have auto insurance. Write your questions so they apply to everyone.

This often means simply adding an additional response category.

Are you satisfied with your current auto insurance? ___ Yes ___ No ___ Don't have auto insurance

Page 93: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Does not presuppose a certain state of affairs.

One of the most common mistaken assumptions is that the respondent knows the correct answer to the question.

Industry surveys often contain very specific questions that the respondent may not know the answer to. For example:

What percent of your budget do you spend on direct mail advertising? ____

Very few people would know the answer to this question without looking it up, and very few respondents will take the time and effort to look it up.

If you ask a question similar to this, it is important to understand that the responses are rough estimates and there is a strong likelihood of error.

Page 94: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Does not imply a desired answer.

Page 95: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Does not imply a desired answer.

The wording of a question is extremely important.

We are striving for objectivity in our surveys and, therefore, must be careful not to lead the respondent into giving the answer we would like to receive.

Leading questions are usually easily spotted because they use negative phraseology.

As examples:

Wouldn't you like to receive our free brochure?

Don't you think the Congress is spending too much money?

Page 96: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Clarity of Expression

Page 97: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Clarity of Expression

This is probably the area that causes the greatest source of mistakes in questionnaires.

Questions must be clear, succinct, and unambiguous.

The goal is to eliminate the chance that the question will mean different things to different people.

If the designers fails to do this, then essentially participants will be answering different questions.

To this end, it is best to phrase your questions empirically if possible and to avoid the use of necessary adjectives.

Page 98: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Clarity of Expression

For example, it asking about frequency, don’t supplying choices that are open to interpretation

1. Very Often 2. Often 3. Sometimes 4. Rarely 5. Never

It is better to quantify the choices, such as:

1. Every Day or More 2. 2-6 Times a Week 3. About Once a Week 4. About Once a Month 5. Never

There are other more subtle aspects to consider such as language and culture.

Page 99: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Emotional Phrasing

Page 100: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Emotional Phrasing

Most adjectives, verbs, and nouns in English have either a positive or negative connotation.

"Is this the best CAD interface you have every used?"

Clearly in this case "best" has strong overtones that deny the participant an objective environment to consider the interface. The signal sent the reader is that the designers surely think it is the best interface, and so should everyone else. Though this may seem like an extreme example, this kind of superlative question is common practice.

1. Do you agree with the Governor's plan to oppose increased development of wetlands? 2. Do you agree with the Governor's plan to support curtailed development of wetlands?

These both ask the same thing, but will likely produce different data. One asks in a positive way, and the other in a negative. It is impossible to predict how the outcomes will vary, so one method to counter this is to be aware of different ways to word questions and provide a mix in your questionnaire.

Page 101: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Does not use unfamiliar words or abbreviations.

Page 102: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Does not use unfamiliar words or abbreviations.

Remember who your audience is and write your questionnaire for them.

Do not use uncommon words or compound sentences.

Write short sentences.

Abbreviations are okay if you are absolutely certain that every single respondent will understand their meanings.

If there is any doubt at all, do not use the abbreviation.

Page 103: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Do not ask Embarrassing Questions

Page 104: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Do not ask Embarrassing Questions

Embarrassing questions dealing with personal or private matters should be avoided.

Your data is only as good as the trust and care that your respondents give you.

If you make them feel uncomfortable, you will lose their trust.

Do not ask embarrassing questions.

Page 105: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Avoid Hypothetical Questions

Page 106: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Avoid Hypothetical Questions

Hypothetical questions and answers are based, at best, on conjecture and, at worst, on fantasy.

1. If you were governor, what would you do to stop crime?

This forces the respondent to give thought to something he may have never considered.

This does not produce clear and consistent data representing real opinion.

Do not ask hypothetical questions.

Page 107: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Avoid Prestige Bias

Page 108: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires

Qualities of a Good Question

Avoid Prestige Bias

Prestige bias is the tendency for respondents to answer in a way that make them feel better.

People may not lie directly, but may try to put a better light on themselves.

For example, it is not uncommon for people to respond to a political opinion poll by saying they support Samaritan social programs, such as food stamps, but then go on to vote for candidates who oppose those very programs.

Data from other questions, such as those that ask how long it takes to learn an interface, must be viewed with skepticism. People tend to say they are faster learners than they are.

There is little that can be done to prevent prestige bias. The best way to deal with prestige bias is to make the questionnaire as private as possible. The farther away the critical eye of the researcher is, the more honest the answers.

Page 109: HCI 510 : HCI Methods I Context of Use. HCI 510: HCI Methods I Questionnaires –Introduction –Sampling Techniques –Development Process –Sample Selection.

Questionnaires


Recommended