Food Quality and Preference 35 (2014) 32–40

Contents lists available at ScienceDirect

Food Quality and Preference

journal homepage: www.elsevier .com/locate / foodqual

Comparison of check-all-that-apply and forced-choice Yes/No questionformats for sensory characterisation

http://dx.doi.org/10.1016/j.foodqual.2014.02.0040950-3293/� 2014 Elsevier Ltd. All rights reserved.

⇑ Corresponding author. Tel.: +598 29248003; fax: +598 29241906.E-mail address: gares@fq.edu.uy (G. Ares).

Sara R. Jaeger a, Rafael S. Cadena b, Miriam Torres-Moreno c, Lucía Antúnez b, Leticia Vidal b,Ana Giménez b, Denise C. Hunter a, Michelle K. Beresford a, Karrie Kam a, David Yin a, Amy G. Paisley a,Sok L. Chheang a, Gastón Ares b,⇑a The New Zealand Institute for Plant & Food Research Ltd., 120 Mt Albert Road, Private Bag 92169, Auckland, New Zealandb Departamento de Ciencia y Tecnología de Alimentos, Facultad de Química, Universidad de la República, Gral. Flores 2124, C.P. 11800 Montevideo, Uruguayc Food, Health and Welfare Research Group, Universitat de Vic, Sagrada Família 7, 08500 Vic, Barcelona, Spain

a r t i c l e i n f o

Article history:Received 22 December 2013Received in revised form 28 January 2014Accepted 8 February 2014Available online 15 February 2014

Keywords:CATAConsumer researchYes/No questionsResearch methodologyApplicability scoring

a b s t r a c t

The application of check-all-that-apply (CATA) questions for sensory product characterisation is gainingacceptance and popularity. This question format has been reported to be a quick and reliable means ofgathering sensory profiles from consumers, concurrently with hedonic assessment. However, a limitationof CATA questions is that they do not encourage deep processing by respondents. Forced-choice ques-tions, where respondents answer ‘‘yes’’ or ‘‘no’’ for each term, may encourage systematic processingand be useful when consumers undertake sensory profiling tasks. This research compared sensory pro-files elicited by consumers using CATA questions or forced-choice Yes/No questions and contribute toongoing investigations of CATA questions and related question formats with a view to developing guide-lines for best practise. Across seven consumer studies with 600+ consumers and multiple product cate-gories, consistent evidence was obtained that forced-choice Yes/No questions are associated with higherterm citation frequencies. However, this did not consistently translate into greater product discrimina-tion. Conclusions regarding similarities and differences amongst samples and the stability of sampleand term configurations were generally independent of question format (i.e., whether the sensory datawere elicited by CATA or forced-choice Yes/No questions). Overall, the comparison of CATA and forced-choice Yes/No questions for sensory characterisation suggested parity of the two question formats. Thisextended to consumers’ perceived difficulty and tediousness for completing the test. Regardless of ques-tion format, consumers, on average, perceived the tests as easy and not tedious.

� 2014 Elsevier Ltd. All rights reserved.

1. Introduction

Descriptive analysis with trained panels is a powerful and fre-quently used methodology for describing the sensory characteris-tics of products (Lawess & Heymann, 2010). Accurate and reliableproduct information is obtained, but the time and resourcesneeded for implementation is significant (Murray, Delahunty, &Baxter, 2001; Varela & Ares, 2012). For this reason, interest in thedevelopment of novel methodologies for sensory characterisationthat provide reliable results in short time frames has been increas-ing (Valentin, Chollet, Lelièvre, & Abdi, 2012; Varela & Ares, 2012).

Check-all-that-apply (CATA) questions, introduced in sensoryresearch by Adams, Williams, Lancaster, and Foley (2007) are oneof the approaches that has been gaining popularity. A CATA question

is a variant of the multiple choice question format in which respon-dents are presented with a list of words or phrases and are askedto select all the options they consider applicable/appropriate(Driesener & Romaniuk, 2006). The application of CATA questionshas been reported to be a quick and reliable means of gatheringinformation about consumer perception of the sensory characteris-tics of food/beverage products, providing similar information to thatobtained using descriptive analysis with trained assessors (Ares,Barreiro, Deliza, Giménez, & Gámbaro, 2010; Bruzzone, Ares, &Giménez, 2012; Dooley, Lee, & Meullenet, 2010; Jaeger et al.,2013). Additionally, CATA questions can be used concurrently withacceptability measurement, without significant risk of hedonicbias (Jaeger & Ares, 2014; Jaeger et al., 2013), hereby enabling twotypes of product insights being generated in a single study.

Despite several advantages, a disadvantage of the CATA ques-tion format is that it does not encourage deep processing byrespondents (Krosnick, 1999; Sudman & Bradburn, 1982), who

S.R. Jaeger et al. / Food Quality and Preference 35 (2014) 32–40 33

may engage in satisficing response strategies, for example byselecting the first terms from the list without thinking carefullyabout the product’s sensory characteristics (Rasinski, Mingay, &Bradburn, 1994; Sudman & Bradburn, 1982). For this reason, thelayout of the list of CATA terms has been reported to significantlyaffect consumer responses (Ares & Jaeger, 2013; Lee, Findlay, &Meullenet, 2013).

It has been suggested that strategies aimed at encouraging sys-tematic processing may be useful when consumers use CATA ques-tions for sensory characterisation (Ares et al., 2013). One way ofachieving this is by using CATA question variants that discouragesatisficing response strategies and increase respondents’ attentionto the task. One such question format is where respondents areforced to answer ‘‘yes’’ or ‘‘no’’ to each of the attributes includedin the question (Rasinski et al., 1994; Smyth, Dillman, Christian,& Stern, 2006). Smyth et al. (2006) reported that compared to CATAquestions, forced-choice Yes/No question led participants to selecta larger number of options and to spend more time answering websurveys. A further possible advantage of the forced-choice Yes/Noquestion format is that it may improve the interpretation of re-sults. With CATA questions participants can leave a term unse-lected because it does not apply to the product, because theyoverlooked it or because they were neutral or undecided aboutits applicability (Sudman & Bradburn, 1982). On the other hand,in the forced-choice Yes/No format respondents indicate if eachof the terms applies (‘‘yes’’) or does not apply (‘‘no’’) for describingthe product.

Forced-choice questions were proposed for sensory character-isation by Ennis and Ennis (2013). These authors asked consumersto indicate if each term ‘‘applies’’ or ‘‘does not apply’’ to describeeach of the tested products. Considering that consumers focus theirattention on all the terms included in the list, forced-choice ques-tions (‘‘yes/no’’ or ‘‘does apply/does not apply’’) could producemore differences between samples and improved sensory productcharacterisation. However, considering that forced-choice ques-tions can be more tedious and time-consuming, research compar-ing the usual CATA question format for sensory characterisationwith forced-choice ‘‘yes/no’’ or ‘‘applies/does not apply’’ questionformats is warranted.

Hence, the aim of the present work was to compare CATA ques-tions with forced-choice Yes/No questions for sensory character-isation in terms of discriminative ability, conclusions regardingsimilarities and differences amongst samples, as well as consum-ers’ perceived ease and tediousness for completing the test.

2. Materials and methods

A total of seven studies were conducted, in which 602 consum-ers took part. Six product categories were tested (plain crackers,chocolate, beer, flavoured rice crackers, hop infusion, and mussels)with 3–7 samples per study. A between-subjects design was usedin all studies to compare sensory characterisations from CATAand forced-choice Yes/No question formats. In Studies 3–7 datawere collected as part of sessions that featured multiple tasksincluding several product categories and research methods. Onlydata relevant to the aims of this research are included. Table 1 pro-vides an overview of the studies.

2.1. Participants

Five consumer studies were conducted in Auckland (NewZealand) and two in Montevideo (Uruguay). The total number ofconsumers in each study ranged from 112 to 134 (Table 1). Con-sumers who participated in Study 1 also participated in Study 2,and consumers who participated in Study 4 also participated in

Study 5. In New Zealand participants were registered on a databasemaintained by a professional recruitment firm and were screenedin accordance with eligibility criteria for each of the studies.Participants attended research sessions at the Plant & Food Re-search Sensory Facility in Auckland. In Uruguay participants wererecruited from the consumer database of the Food Science andTechnology Department of Universidad de la República (Uruguay),based on their consumption of the focal products. Participants gaveinformed consent and were compensated for their participation.

Participants were aged between 18 and 67 years old and thepercentage of female participants ranged from 50% to 70%. Theconsumer samples comprised varying household compositions, in-come levels, education levels, etc. but were not representative ofthe general populations in Montevideo and Auckland.

2.2. Samples

Six different product categories were tested (Table 1). All sam-ples in Studies 1–5 were commercially available in Uruguay orNew Zealand and had been purchased from local supermarkets.

The samples in Study 6 were infusions made from hops(Humulus lupulus) (two commercial cultivars and two advancedselections) grown under commercial conditions, harvested at com-mercial maturity and processed to pellet format in an industrialplant. The infusions were prepared by weighing 5 g of pellets in200 g of water and boiling for 5 min. Approximately 1 h prior tobeing served to participants, aliquots of 10 ml filtered infusionsolution were placed blue olive oil tasting glasses which were cov-ered with a watch glass.

The samples in Study 7 were mussels (Perna canaliculus) grownin the New Zealand Marlborough Sounds under commercial condi-tions. They were harvested at commercial maturity and trans-ported on ice to a processing facility. Mussels were steamed in acommercial cooker for 10 min, and to a minimum core tempera-ture of 72 day �C. The mussels were refrigerated at 4 �C andair-freighted on ice to the PFR sensory facility in Auckland.30 min before being presented to consumers, samples wereremoved from cool storage (4 �C) and laid on an upturned halfmussel shell that was placed on boat-shaped dish made ofodour-free white pine.

In all studies, samples were presented at room temperature andserving sizes were always sufficient to allow 2–3 bites/sips persample. Mussles and hop infusions were tasted at room tempera-ture to maintain continuity to other research on these productclasses performed at Plant and Food Research.

2.3. Experimental treatments, sensory terms and data collection

The procedure for data collection in Studies 1–7 was similar.Between-subjects experiments were always used, comparing re-sponses from two experimental treatments: CATA questions andforced-choice Yes/No questions. One experimental treatment was‘CATA’, meaning that participants in this group were asked to checkall the terms that they considered appropriate to describe eachsample. In the second treatment participants were asked to indi-cate if each of the terms included on the list were appropriate fordescribing each sample by answering ‘‘yes’’ or ‘‘no’’. In Studies1–5 consumers were presented the forced-choice response optionsas ‘‘yes’’ then ‘‘no’’, whilst in Studies 6 and 7 the order was reversed(i.e., ‘‘no’’ then ‘‘yes’’). In each study, participants were randomlyassigned to one of the two experimental treatments. Table 1 showsthe number of participants who completed the task using CATAquestions for each of the seven studies.

The sensory terms used in each study were based on previousresearch using the same product categories. The list of termscomprised 8–30 terms and with the exception of Study 6 (aroma

Table 1Overview of the seven studies comparing check-all-that-apply (CATA) questions and forced-choice Yes/No questions for sensory characterisation.

Study ID Number ofconsumersin the test

Number of consumers whocompleted the task usingCATA questions

Product category Number ofsamples

Number ofsensory terms

1 120 60 Plain crackers 5 202 120 60 Chocolate 5 213 112 56 Beer 3 124 117 57 Rice crackers 5 145 117 57 Chocolate 4 306 134 68 Hop infusion 4 127 Aromaa 119 60 Mussels 4b 87 Appearanca 119 60 Mussels 4b 117 Tastea 119 60 Mussels 4b 107 Texturea 119 60 Mussels 4b 9

a Consumers answered four different CATA questions in the same session, one for each modality (aroma, appearance, taste and texture).b Two of the samples were identical.

34 S.R. Jaeger et al. / Food Quality and Preference 35 (2014) 32–40

only) covered multiple sensory modalities (appearance, aroma, fla-vour/taste, texture, after taste/mouth feel). In Studies 1–6 consum-ers answered a single question which included all the selectedterms, whilst in Study 7 consumers answered four separate ques-tions, one per sensory modality (aroma, appearance, flavour andtexture) (Table 1).

Specifically, in Study 1 the following terms were used to charac-terise plain crackers: adhesive, aftertaste, big, brittle, crunchy, dry,greasy, hard, heterogeneous colour, homogeneous colour, off-fla-vour, oily flavour, salty, small, soft, sour, tasteless, thick, thin,toasted flavour. In Study 2, for chocolate samples, the terms were:bitter, burnt, cocoa flavour, caramel flavour, chocolate flavour,crunchy, glossy, greasy, gritty, hard, melts easily, milk flavour,off-flavour, opaque, rough, salty, soft, sour, sticky, sweet, vanillaflavour. The terms in Study 3 (beer) were: bitter, bland, complex,flat, flora, fruity, hoppy, lingering aftertaste, savoury, smokey,smooth, sweet. The terms used for describing rice crackers in Study4 were: bland, buttery, crumbly, crunchy, hard, light, lumpy sur-face, off-flavour, salty, savoury, shiny surface, soy sauce, sticks toteeth, toasted. In Study 5 the following terms were used for char-acterising chocolate samples: bitter, brittle/breaks apart, burnt,butterscotch, caramel, chocolate, citrus, cocoa, creamy, crunchy,fruit pieces, fruity, gooey, hard, hokey pokey, kiwifruit, lingeringtaste, liquid centre, melting, mouth coating, rich, rough, sickly,smooth, sticks in teeth, sticky, sweet, tangy, vanilla, waxy. In Study6 (hop infusions) the terms were: beer-like, carrot/pumpkin soup,spicy, citrus, light, cooked cabbage, intense, floral, herbal tea-like,fruity, hay, green. In Study 7 the terms used to describe mussels’aroma were: bland, crab-like, earthy, fishy, meaty, sea air/seawater, seaweed, typical mussel. Mussels’ appearance was de-scribed using the terms big, bright, dry, dull, fresh, glossy, moist,old, plump, shrivelled/wrinkled, small; whilst the terms used fordescribing flavour were fishy, muddy, persisting flavour, salty/briny, savoury, seafood, seaweed, sweet, typical mussel, weak/bland. Finally, the texture terms in Study 7 were: bits left in mouth,chewy, creamy, firm, moist, plump, rubbery/elastic, soft, tender.Based on recommendations by Ares, Etchemendy et al. (2014),the order in which the sensory terms were listed (both experimen-tal treatments) was balanced within and across consumers, follow-ing William’s Latin Square experimental design.

All samples were labelled with 3-digit random codes. Productswere presented sequentially in accordance with designs that werebalanced for presentation order and carry-over effects (Williams’design). Samples could be tasted more than once. In Studies 6and 7 a break of 1–2 min was forced between each sample. Datacollection took place in standard sensory booths, under whitelighting, controlled temperature (23 �C) and airflow conditions.

In Studies 1–2 and 4–6, participants answered two Likert ques-tions immediately following completion of the study: (i) It was

easy to answer the questions about these samples; and (ii) It wastedious to answer the questions about these samples. The labelled7-point scale was anchored at 1 = ‘disagree extremely’ and7 = ’agree extremely’.

For classification purposes participants’ age, gender, and fre-quency of consumption of the focal products were recorded. Inall instances differences between the participant profiles of theexperimental treatment groups were non-significant (p > 0.15).Hence, it is possible to infer that differences between experimentaltreatments may be linked to differences in study protocol, as op-posed to differences in group characteristics.

2.4. Data analysis

The procedure for data analysis in Studies 1–7 was similar. Foreach experimental treatment (i.e., CATA or forced-choice Yes/Noquestions), frequency of use of each sensory attribute was deter-mined by counting the number of consumers that used that termto describe each sample.

Fisher’s exact test (Fisher, 1954) was used to determine signif-icant differences between experimental treatments in the totalnumber of terms mentioned by consumers to describe the wholesample set, and differences in the frequency of use of each term.

Linear regressions were used to study the influence of numberof samples, number of terms and the frequency of use of the termson the increase in the frequency of use of the terms when forced-choice Yes/No questions were implemented.

Cochran’s Q test (Manoukian, 1986) was carried out separatelyon data from each experimental treatment to identify significantdifferences amongst samples for each of the sensory terms.

Correspondence Analysis (CA) was performed on the frequencytable from each experimental treatment. CA was performedconsidering Hellinger’s distances, as recommended by Meyners,Castura, and Carr (2013). Similarity between the sensory spacesobtained with CATA and forced-choice Yes/No questions was eval-uated using the RV coefficient (Robert & Escoufier, 1976) betweensample and term configurations in the first two dimensions of theCA. The significance of the RV coefficient was tested using a permu-tation test (Josse, Husson, & Pages, 2007).

The stability of sample and term configurations from CATA andforced-choice Yes/No questions was evaluated using a bootstrap-ping resampling approach (Ares, Tárrega, Izquierdo, & Jaeger,2014). The bootstrapping process consisted of extracting randomsubsets of different size (m = 5, 10, 20, 30, . . ., 50, . . ., N) from theoriginal data with N consumers, using sampling with replacement.For each m, 1000 random subsets were obtained. For each subsetthe frequency table corresponding to the data of the selectedassessors was computed and CA was performed. The agreementbetween sample and term configurations in the first two

S.R. Jaeger et al. / Food Quality and Preference 35 (2014) 32–40 35

dimensions of the CA and the reference configurations (obtainedwith all the consumers) was independently evaluated for samplesand terms by computing the RV coefficient between their coordi-nates (Abdi, 2010). Average values (and standard deviations) forthe 1000 random subsets of size equal to the total number of con-sumers in each study (N) were calculated and used as an index ofstability. Also, the minimum number of consumers needed to reachan average RV = 0.95 was determined (Blancher, Clavier, Egoroff,Duineveld, & Parcon, 2012), as the benchmark to assess the stabil-ity of sample and descriptor configurations from CA.

Analysis of variance (ANOVA) was carried out for ease of testand tediousness scores considering experimental treatment asfixed source of variation, whilst consumer (within experimentaltreatment) was specified as a random effect. When significant dif-ferences were detected (p < 0.05), Tukey’s test was applied.

All statistical analyses were performed using R language(R Development Core Team, 2007). FactoMineR was used tocalculate RV coefficients (Lê, Josse, & Husson, 2008).

3. Results

3.1. Frequency of use of sensory terms

Across all studies, consumers used a significantly larger numberof sensory terms (p < 0.0001) for describing samples when answer-ing forced-choice Yes/No questions than when using CATA ques-tions. As shown in Tables 2a and 3a, consumers selected anaverage of 11–37% of the terms to describe samples using CATAquestions, whereas the average number of selected terms rangedfrom 31% to 53% when forced-choice Yes/No questions were used.The average increase in the number of terms ranged from 29% forStudy 6, to 336% for the evaluation of flavour in Study 7 (Tables 2band 3b). The average increase in the number of selected terms didnot significantly depend on the number of samples included in thestudy (p = 0.58) or the number of terms on the list (p = 0.58).

At the aggregate level the frequency of use of the terms for bothquestion formats were significantly correlated (p < 0.001,R2 = 0.78). However, for forced-choice Yes/No questions the aver-age increase in the frequency of use of terms ranged from 31% to144% and was for each study higher than the average frequencyof use of terms in CATA questions (Tables 2d and 3d). Whilst

Table 2Summary of results for the comparison of sensory characterizations with consumers obta

(a) Average percentage of terms used to describe samples

(b) Average increase in the number of terms used for describing samples when usingquestions

(c) Percentage of terms which frequency of use significantly increased when using Yequestions (p < 0.05)

(d) Average increase in the frequency of use of the terms(e) Percentage of terms with significant differences amongst samples (Cochran Q test

p < 0.05)

(f) Percentage of terms for which different conclusions were drawn using CATA andquestions

(g) RV between sample configurations obtained from Correspondence Analysis of datCATA and Yes/No questions

(h) RV between term configurations obtained from Correspondence Analysis of dataCATA and Yes/No questions

* Indicates that the RV coefficient is significant at p 6 0.05.** Indicates that the RV coefficient is significant at p 6 0.0001.nsIndicates that the RV coefficient is not significant (p > 0.05).

frequency of use significantly increased for all the terms consid-ered for texture evaluation in Study 7, frequency of use only signif-icantly increased for a subset of terms in the other six studies.Tables 2c and 3c show that with the exception of textureevaluation in Study 7, frequency of use significantly increased for42–88% of the terms. Average increase in the frequency of use ofthe terms was not significantly affected by the number of samplesincluded in the study (p = 0.93), the number of terms on the list(p = 0.88) or the frequency of use of the terms with the CATAquestion format (p = 0.17).

With the forced-choice Yes/No question format, frequency ofuse did not significantly increase for both highly relevant terms(e.g. crunchy in Study 4, frequency of use 89% for CATA questionformat and 91% for forced-choice Yes/No question format) andterms that were rarely used for describing samples (e.g. beer-likein Study 6, frequency of use 9% for CATA question format and11% for forced-choice Yes/No question format). However, the in-crease in the frequency of use of the terms was inversely associ-ated (p < 0.0001) with the frequency of use of the terms fordescribing samples when the CATA question format was used. Asshown in Fig. 1, all the terms that showed an increase in the fre-quency of use higher than 150% were used by less than 20% ofthe consumers at the aggregate level when the CATA question for-mat was considered. The terms with the highest increase in the fre-quency of use corresponded to both simple sensory characteristics(e.g. sweet, bitter, big, small, firm) and to complex characteristicswhich can lack of a unique meaning for consumers (e.g. meaty, sea-weed, vanilla, waxy, chocolate flavour, vanilla) (Fig. 1). Meanwhile,the number of samples in the study and the number of terms in thequestion did not significantly affect the increase in the frequency ofuse of the terms when the forced-choice Yes/No question formatwas used (p = 0.90 and p = 0.43, respectively).

It was interesting to note that average number of terms used fordescribing samples and frequency of use of the terms were signif-icantly higher for the forced-choice Yes/No question format thanfor the CATA question format regardless of the order in whichthe response options were listed in the forced-choice question(i.e., Yes/No or No/Yes). As shown in Tables 2a–c and 3a–c, theinfluence of forced-choice questions in Studies 6 and 7 (No/Yes)was similar to the influence observed in Studies 1–5 (Yes/No),i.e., frequency of term use increased. Furthermore, in Studies 6

ined with CATA and forced-choice Yes/No questions in Studies 1–6.

Study ID

1 2 3 4 5 6

CATA:21%

CATA:23%

CATA:30%

CATA:37%

CATA:36%

CATA:24%

Yes/No:33%

Yes/No:37%

Yes/No:45%

Yes/No:50%

Yes/No:50%

No/Yes:31%

Yes/No 57% 61% 50% 35% 39% 29%

s/No 70% 67% 67% 50% 80% 42%

73% 82% 57% 41% 73% 31%, CATA:

55%CATA:81%

CATA:75%

CATA:86%

CATA:80%

CATA: 8%

Yes/No:75%

Yes/No:67%

Yes/No:83%

Yes/No:86%

Yes/No:93%

No/Yes:25%

Yes/No 20% 14% 8% 14% 20% 33%

a from 0.95** 0.99** 0.99** 0.99** 0.98** 0.85*

from 0.89** 0.92** 0.97** 0.93** 0.92** 0.18ns

Table 3Summary of results for the comparison of sensory characterizations with consumers obtained with CATA and forced-choice Yes/No questions for the evaluation of four sensorymodalities of mussels in Study 7.

Modality

Aroma Appearance Flavour Texture

(a) Average percentage of terms used to describe samples CATA: 26% CATA: 26% CATA: 11% CATA: 30%No/Yes:46%

No/Yes:37%

No/Yes:48%

No/Yes:53%

(b) Average increase in the number of terms used for describing samples when using Yes/No questions 77% 42% 336% 77%(c) Percentage of terms which frequency of use significantly increased when using Yes/No questions (p < 0.05) 88% 64% 80% 100%(d) Average increase in the frequency of use of the terms 106% 51% 144% 88%(e) Percentage of terms with significant differences amongst samples (Cochran Q test, p < 0.05) CATA: 25% CATA: 91% CATA: 40% CATA: 67%

No/Yes: 0% No/Yes:91%

No/Yes:30%

No/Yes:89%

(f) Percentage of terms for which different conclusions were drawn using CATA and Yes/No questions 25% 9% 40% 44%(g) RV between sample configurations obtained from Correspondence Analysis of data from CATA and Yes/No

questions0.82ns 0.92* 0.72ns 0.88*

(h) RV between term configurations obtained from Correspondence Analysis of data from CATA and Yes/Noquestions

0.45ns 0.91* 0.34ns 0.71*

* Indicates that the RV coefficient is significant at p 6 0.05.While nsindicates that the RV coefficient is not significant (p > 0.05).

1 - smokey, 2 - tasteless, 3 - off-flavour, 5 - bland, 6 - fruity, 7 - bland, 8 - hokey pokey, 9 - liquid centre, 10 - green, 11 - beer-like, 12 - small, 13 - dry, 14 - hay, 15 - crunchy, 16 - bitter, 17 - fruity, 18 - greasy, 19 - carrot/pumpkin soup, 20 - cooked cabbage, 21 - hard, 22 - hoppy, 23 - gooey, 24 - soft, 25 - herbal tea-like, 26 - fruit pieces, 27 - sticks in teeth, 28 - sticky, 29 - old, 30 - bland, 31 - buttery, 32 - fruity, 33 - greasy, 34 - caramel, 35 - off-flavour, 36 - lumpy surface, 37 - toasted flavour, 38 - rubbery/ elastic, 39 - creamy, 40 - sticks in teeth, 41 - soft, 42 - homogeneous colour, 43 - soy sauce, 44 - cacao flavour, 45 - thick, 46 - moist, 47 - hard, 48 - salty, 49 - shrivelled/ wrinkled, 50 - brittle, 51 - plump, 52 - typical mussel, 53 - sweet, 54 - floral, 55 - hard, 56 - melts easily, 57 - spicy, 58 - gritty, 59 - typical mussel, 60 - butterscotch, 61 - sickly, 62 - fishy, 63 - soft, 64 - salty, 65 - moist, 66 - rich, 67 - melting, 68 - toasted, 69 - big, 70 - dry, 71 - cocoa, 72 - fresh, 73 - hard, 74 - savoury, 75 - aftertaste, 76 - earthy, 77 - flat, 78 - crumbly, 79 - floral, 80 - tangy, 81 - salty/ briny, 82 - rough, 83 - tender, 84 - sweet, 85 - heterogeneous colour, 86 - sticky, 87 - mouth coating, 88 - oily flavour, 89 - opaque, 90 - complex

light weak/ blandcitrus

1 23

dull56789

crunchy1011 12131415 16 sweet17 18brightness shiny surface chewy19

2021 22light 2324

2526

27 bi�er28

crunchy2930 crunchy31burned32

33 3435 3637muddy 38off-flavour

394041 sea air/ sea water4243 44 chocolatelingering a�ertaste45 46 light47sour

smooth4849 5051 5253

5455 5657 58

5960

61kiwifruit 62 6364 65 66 salty

6768rough69

70

71727374

7576 7778citrus 7980 seafood8182

8384 858687

88 8990glossyvanilla flavouradhesivebright bits le� in mouth

caramel flavouracid creamy plump lingering tastebri�le/breaks apart savoury smoothintensethinwaxy crab-like

seaweed

smallfishy

milky flavour firm

savoury

sweetbig

persis�ng flavourvanilla

meatychocolate flavourseaweed

bi�er

-50%

0%

50%

100%

150%

200%

250%

300%

350%

400%

0% 20% 40% 60% 80% 100%

Incr

ease

in t

he fr

eque

ncy

of u

se (

%)

Frequency of use in CATA ques�ons (%)

Fig. 1. Average increase in the frequency of use of sensory terms when forced-choice Yes/No questions were considered as a function of the aggregate frequency of mention ofthe terms obtained with the check-all-that-apply (CATA) question format.

36 S.R. Jaeger et al. / Food Quality and Preference 35 (2014) 32–40

and 7 none of the terms showed a significant decrease in their fre-quency of use.

3.2. Differences amongst samples

Cochran’s Q test was used to identify significant differencesamongst samples for each of the focal sensory terms. The percent-age of terms for which significant differences amongst sampleswere identified ranged from to 8% to 91% when CATA questionswere used and from 0% to 91% when forced-choice Yes/No ques-tions were used (Tables 2e and 3e). Further, conclusions regardingsimilarities and differences amongst samples differed between thetwo question formats for 8–44% of the terms (Tables 2f and 3f). Thepercentage of terms in which different conclusions were drawn forCATA and forced-choice Yes/No question formats was not signifi-cantly affected by the frequency of use of the terms using CATA

questions (p = 0.59), the number of samples in the studies(p = 0.41) or the number of terms (p = 0.97).

As shown in Tables 2e and 3e, forced-choice Yes/No questionsled to a larger percentage of terms with significant differencesamongst samples than CATA questions in 5 out of 10 evaluations(Studies 1, 3, 5, 6 and texture evaluation in Study 7). However,the opposite result was found for Study 2, and the evaluation of ar-oma and flavour in Study 7. When aroma was evaluated in Study 7no significant differences amongst samples were established usingforced-choice Yes/No questions, whilst CATA questions detecteddifferences between samples in 25% of the terms. Meanwhile, nodifference between the CATA and forced-choice Yes/No questionformats was found in the percentage of terms in which significantdifferences amongst samples were identified for Study 4 andappearance evaluation in Study 7 (Tables 2e and 3e).

Changes in the frequency of use of the terms at the aggregatelevel were not the only explanation for the observed shifts in

(a)

0.7

0.8

0.9

1

mpl

e co

nfig

urat

ions

S.R. Jaeger et al. / Food Quality and Preference 35 (2014) 32–40 37

results from Cochran’s Q test. For example, in Study 2 the fre-quency of use of the terms burnt and soft did not significantly differ(p > 0.17) between the two question formats, but results fromCochran’s Q test led to different conclusions. This was observedfor the term greasy in Study 1 and crunchy in Study 4, suggestingthat the two question formats led to differences in the way inwhich the terms were used for describing samples.

(b)

0.3

0.4

0.5

0.6

0 10 20 30 40 50 60

Ave

rage

RV

of s

a

Number of consumers in the resampled virtual panel

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 10 20 30 40 50 60

Ave

rage

RV

of te

rm c

onfig

urat

ions

Number of consumers in the resampled virtual panel

(c) 1

3.3. Sample and term configurations from Correspondence Analysis

As shown in Tables 2g and 3g, the RV coefficient between sam-ple configurations in the first and second dimensions of the Corre-spondence Analysis was significant for 8 out of 10 comparisons.Sample configurations were highly similar and samples weregrouped following the same patterns considering data from CATAand forced-choice Yes/No questions for Studies 1–6 and appear-ance and texture evaluation in Study 7. However, for the aromaand flavour evaluations in Study 7 the RV coefficients were lowand non-significant (Table 3g). In these two instances, significantdifferences amongst samples were established for less than 50%of the terms, which points to differences amongst samples beingsmaller than for the rest of the evaluations.

The RV coefficients between term configurations in the first andsecond dimensions of the Correspondence Analysis were lowerthan those from sample configurations (Tables 2h and 3h). TheRV coefficient of term configurations from CATA and forced-choiceYes/No questions for Study 6 and aroma and flavour evaluation inStudy 7 were low and not significant, which indicates differencesin the way in which the terms were used to describe samples. Thisis likely to be linked to the fact that samples were more similarthan in the rest of the evaluations.

(d)

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0 10 20 30 40 50 60

Ave

rage

RV

of s

ampl

e co

nfig

urat

ions

Number of consumers in the resampled virtual panel

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 10 20 30 40 50 60

Ave

rage

RV

of te

rm c

onfig

urat

ions

Number of consumers in the resampled virtual panel

Fig. 2. RV coefficient of sample and term configurations with respect to thereference configuration as a function of the number of consumers considered in theresampled virtual panel for CATA (black lines) and forced-choice Yes/No questions(grey lines) for Study 2 ((a) and (b), respectively) and Study 4 ((c) and (d),respectively). Vertical bars correspond to standard deviations.

3.4. Stability of sample and term configurations

For both question formats the RV coefficient of sample and termconfigurations increased with increasing number of consumers inthe virtual panel, as was expected. Using Studies 2 and 5 as exem-plars, Fig. 2 shows the evolution of the average RV coefficients be-tween the configurations of virtual panels of different size and thereference configuration as a function of the number of consumersfor the CATA and forced-choice Yes/No question formats. In Study2 the stability of sample and term configurations were highly sim-ilar. On the other hand, in Study 5 the RV of sample and term con-figurations tended to be higher for forced-choice Yes/No questionsthan for CATA questions (Fig. 2(c) and (d)). Also, standard devia-tions of RV coefficients were markedly lower for forced-choiceYes/No questions than for CATA questions, which indicates ahigher agreement amongst consumers. However, this trend wasnot observed for all studies.

As shown in Table 4, the average RV coefficient of sample andterm configurations, calculated for a sample size equal to the totalnumber of consumers in the test, was similar for CATA and forced-choice Yes/No questions. The average RV coefficients were higherfor CATA than for forced-choice questions for some of the evalua-tions (e.g. flavour evaluation in Study 7), whilst the opposite trendwas found for other evaluations (e.g. texture evaluation in Study 7).

The minimum number of consumers needed to reach stablesample configurations (i.e., an average RV coefficient of 0.95) waslower for forced-choice Yes/No questions than for CATA questionsfor three of the evaluations (Studies 3, 5 and texture evaluation inStudy 7), whilst the opposite trend was found for two evaluation(Study 2 and appearance evaluation in Study 7). A similar trendwas observed for the minimum number of consumers needed toreach stable term configurations.

38 S.R. Jaeger et al. / Food Quality and Preference 35 (2014) 32–40

3.5. Ease and tediousness of the task

CATA and forced-choice Yes/No question formats were alsocompared in terms of perceived ease and tediousness in Studies1–2 and 4–6. In no instance were significant differences in theseperceptual measures established (p > 0.15), suggesting that askingconsumers to indicate if each of the terms included in a forced-choice question applies/does not apply (i.e., Yes/No) was not per-ceived as more/less easy or tedious than selecting all the wordsthat apply for describing samples. On average, participants‘‘agreed’’ or ‘‘agreed strongly’’ that both tasks were easy (aver-age = 5.6, standard deviation = 1.6), whilst they ‘‘disagreed’’ or‘‘disagreed strongly’’ with the fact that the tasks were tedious(average = 2.5, standard deviation = 1.5).

4. Discussion

Forced-choice Yes/No questions have been proposed as an alter-native to discourage the use of satisficing response strategies whenconsumers answer CATA questions (Krosnick, 1999; Sudman &Bradburn, 1982). Although both question formats have been com-pared in survey research (Rasinski et al., 1994; Smyth et al., 2006),sensory product characterisations provided by CATA and forced-choice Yes/No questions have not previously been compared.Across seven consumer studies the present research has shownthat, compared to CATA questions, asking consumers to indicateif each of the terms included on the list applies or does not applyfor describing samples (i.e., forced-choice Yes/No questions) leadsto an increase in the total number of selected terms. This resultis in agreement with Rasinski et al. (1994) and Smyth et al.(2006), who reported that survey respondents selected more op-tions when answering forced-choice question formats comparedto check-all-that-apply formats.

When CATA questions are used, participants may not check allthe sensory characteristics that apply to describe the productsdue to the use of satisficing response strategies. Participants tryto reduce the time they invest in responding a CATA questionand may fail to check relevant response options. Ares, Etchemendyet al. (2014) showed that when consumers answer CATA questionsfor sensory characterisation repeatedly, they reduce the depth withwhich information is visually processed as the test progresses andtherefore they may fail to select relevant terms. Similarly, Smythet al. (2006) observed that respondents took longer answeringforced-choice questions than CATA questions, suggesting that theforced-choice format encourages deeper processing of responseoptions. However, it is important to stress that higher processing

Table 4Number of consumers needed to reach an average RV coefficient of sample and term confisample and term configurations for a sample size equal to the total number of consumers, oforced-choice Yes/No questions.

Study Average RV of sampleconfigurations coefficientacross simulations

Minimum number of consumersnecessary to reach an RV coefficiensample configurations of 0.95

CATA Yes/No CATA Yes/No

1 0.98 0.98 27 272 0.99 0.99 12 173 0.99 0.99 11 94 0.99 0.99 25 255 0.98 0.99 31 76 0.88 0.89 N/A N/A7 Aroma 0.83 0.85 N/A N/A7 Appearance 0.99 0.98 19 297 Flavour 0.93 0.91 N/A N/A7 Texture 0.96 0.98 45 35

N/A indicates that an average RV coefficient equal to 0.95 was not achieved during the

does not necessarily imply greater validity. In their work Rasinskiet al. (1994) stated that they could not conclude if forced-choicequestions led to more accurate results since external validity datawere not available. In the present work it is also not possible toestablish if sensory characterisations obtained with the forced-choice Yes/No question format has greater validity than thoseobtained with the CATA question format. For this purpose, furtherresearch comparing results from CATA and forced-choice Yes/Noquestions with those provided by trained assessors using descrip-tive analysis would be valuable.

The results from this research suggests that forced-choice Yes/No questions may lead to over-reporting of the applicability ofthe terms for describing samples. It is not clear why. Consumersmay select a larger proportion of terms just because they have toindicate ‘‘yes’’ or ‘‘no’’, and not necessarily because the terms applyfor describing samples. In the present work the increase in the fre-quency of use was not related to the nature of the sensory terms, asit increased for both simple and complex sensory characteristics. Itwas observed that increase in frequency of use tended to be largerfor terms that were seldom regarded as applicable for describingsamples when the CATA question format was used (Fig. 1). Tenta-tively, they seemed to have selected ‘‘yes’’ over ‘‘no’’ if they areundecided whether or not the term applied. Forced-choice Yes/No questions may also be associated with increased frequency ofuse by making attributes more salient due to the fact that partici-pants focus their attention for a longer time in each of the terms.The fact that frequency of use of the terms increased regardlessof the order in which the response options (‘‘yes’’ and ‘‘no’’) wereconsidered (Tables 2a–c and 3a–c), as well as the fact that a similartrend has been found when short list of terms are used (Ares et al.,2013), strengthens this explanation. Therefore, asking consumersto indicate ‘‘yes’’ or ‘‘no’’ to each of the terms of the CATA questionmay lead to over-reporting of the relevance of terms for describingsamples.

In this research results from forced-choice Yes/No and CATAquestion formats led to the same conclusions about similaritiesand differences amongst samples for the majority of the sensoryterms (Tables 2f and 3f). However, some differences were identi-fied. In some instances the discriminative ability of forced-choiceYes/No questions was higher than that of CATA questions, whilstthe opposite trend was observed in other instances. Sample config-urations were highly correlated between forced-choice Yes/No andCATA questions for the majority of the studies. Considering thatthe lowest RV coefficients were found for the studies which in-cluded the most similar samples, it can be concluded that the CATAand forced-choice Yes/No question formats provide similar sensory

gurations from Correspondence Analysis equal to 0.95 and average RV coefficient ofbtained via a bootstrapping resampling approach for check-all-that-apply (CATA) and

t ofAverage RV of termconfigurations coefficientacross simulations

Minimum number of consumersnecessary to reach an RV coefficient ofterm configurations of 0.95

CATA Yes/No CATA Yes/No

0.93 0.94 NA N/A0.95 0.92 60 N/A0.94 0.96 NA 470.95 0.94 57 N/A0.96 0.98 47 320.70 0.72 N/A N/A0.68 0.72 N/A N/A0.95 0.97 60 450.78 0.72 N/A N/A0.93 0.94 N/A N/A

bootstrapped resampling.

S.R. Jaeger et al. / Food Quality and Preference 35 (2014) 32–40 39

spaces when clearly different samples are considered. However, forcompleteness we note that the studies where the sensory spaceswere most different were also the instances where sensory assess-ments were performed for aroma-only or flavour-only. In Study 7,similar sensory spaces were elicited using CATA and forced-choiceYes/No questions for mussel appearance and mussel texture, butnot mussel aroma or mussel flavour.

The similarity between term configurations from forced-choiceYes/No and CATA questions was lower than that of sample config-urations, which indicates that the way in which consumers usedsome of the terms to describe samples differed between the meth-odologies. This difference can be related to the type of responseprovided by consumers. When forced-choice questions are used,consumers have to indicate if each of the terms applies or doesnot apply for describing each sample. However, with CATA ques-tions consumers can leave a term unselected because it does notapply to the product, because they overlooked it or because theycould not make up their mind about its applicability. Therefore,when using CATA questions consumers may select the terms thatstrongly catch their attention and that are regarded as most appli-cable for describing samples.

The stability of sample and term configurations from CATA andforced-choice Yes/No questions, calculated using a bootstrappingresampling approach, were similar (Table 4). This suggests thatasking consumers to focus their attention on each of the terms in-cluded in the list of terms did not increase consumers’ agreementand consequently did not stabilize sensory spaces.

Regarding perceptual measures of ease of task and tediousness,no significant differences between the two question formats werefound. This suggests that asking consumers to indicate if each ofthe terms is applicable or not for describing samples may not com-plicate the task or make it more tedious. However, it has to be ta-ken into account that if the number of samples and/or terms is verylarge, consumers could begin to perceive forced-choice questionsas more tedious than CATA questions.

When using forced-choice Yes/No questions concurrently withhedonic tests, asking consumers to answer Yes/No to each of theterms included in a CATA question can be associated with a higherrisk of bias on hedonic scores. However, Jaeger & Ares (2004) haverecently shown that both CATA and forced-choice Yes/No ques-tions do not lead to bias on hedonic liking scores.

Sudman & Bradburn (1982) have argued against the use of thecheck-all-that-apply question format due to the fact that omis-sion to select a term from the list is difficult to interpret. Theseauthors suggested that forced-choice questions may provide moreaccurate results. When samples are clearly different, results fromthe present work suggest that there are no major differences be-tween CATA and forced-choice Yes/No questions for sensory char-acterisation with consumers. Despite differences in the degree ofcognitive processing required by these two approaches of dataelicitation, the sensory product spaces and product profilesshared a high degree of similarity in the majority of the studies.This suggests that as long as consumers cognitively engage en-ough, additional deeper processing may not alter the sensoryproduct characterisations obtained. Thus, results from sensorycharacterisations obtained using CATA or forced-choice Yes/Noquestions (with balanced presentation of the terms) may notstrongly differ. The present work shows that forcing consumersto focus their attention on each of the terms of the CATA questiondoes not necessarily lead to higher discrimination or agreementamongst consumers. On the contrary, CATA questions can leadto more spontaneous responses which better reflect consumers’perception. An alternative to prevent consumers from selectingterms that are not regarded as strongly applicable to the productscan be to include a ‘‘not sure/cannot decide’’ option within theforced-choice question.

5. Conclusions

This research has continued investigations into the use ofconsumers for sensory product characterisation. A comparison ofcheck-all-that-apply (CATA) questions with forced Yes/No ques-tions revealed the latter question format to be associated with in-creased frequency of term use, but few other differences. Despitereducing the reliance on satisficing response strategies, forced-choice Yes/No questions provided similar sensory spaces to thosederived from CATA questions. When differences between sampleswere minor and/or sensory assessments focused on aroma and fla-vour only, the two question formats yielded results that were inless agreement. Which set of product profiles are more similar tothose generated by trained sensory panellists is a question for fu-ture research. Consumers did not perceive forced-choice Yes/Noquestions to be more difficult/tedious than CATA questions tocomplete.

6. Author contributions

S.R.J., G.A., A.G., R.S.C. and M.T.M. conceived and planned thestudy, and wrote the paper. S.R.J., G.A. and R.S.C. wrote the paper.G.A., L.V. and R.S.C. analysed the data. All other authors contributedto data collection.

Acknowledgements

Staff at Plant & Food Research are thanked for help in planningand collection of data, in particular Belinda Timms and Jacqui Day.Financial support was received from CAPES-Brasil, Comisión Secto-rial de Investigación Científica (Universidad de la República, Uru-guay), and The New Zealand Ministry for Business, Innovation &Employment and Plant & Food Research. The authors are gratefulto Universitat de Vic for the research scholarship granted to authorMiriam Torres-Moreno.

References

Abdi, H. (2010). Congruence. Congruence coefficient, RV coefficient, and mantelcoefficient. In N. J. Salkind, D. M. Dougherty, & B. Frey (Eds.), Encyclopedia ofresearch design (pp. 222–229). Thousand Oaks, CA: Sage.

Adams, J., Williams, A., Lancaster, B., & Foley, M. (2007). Advantages and uses ofcheck-all-that-apply response compared to traditional scaling of attributes forsalty snacks. In 7th pangborn sensory science symposium. Minneapolis, USA, 12–16 August, 2007.

Ares, G., Barreiro, C., Deliza, R., Giménez, A., & Gámbaro, A. (2010). Application of acheck-all-that-apply question to the development of chocolate milk desserts.Journal of Sensory Studies, 25, 67–86.

Ares, G., Etchemendy, E., Antúnez, L., Vidal, L., Giménez, A., & Jaeger, S. R. (2014).Visual attention by consumers to check-all-that-apply questions: Insights tosupport methodological development. Food Quality and Preference, 32, 210–220.

Ares, G., & Jaeger, S. R. (2013). Check-all-that-apply questions: Influence of attributeorder on sensory product characterization. Food Quality and Preference, 28,141–153.

Ares, G., Jaeger, S. R., Bava, C. M., Chheang, S. L., Jin, D., Gimenez, A., et al. (2013).CATA questions for sensory product characterization: Raising awareness ofbiases. Food Quality and Preference, 30(2), 114–127.

Ares, G., Tárrega, A., Izquierdo, L., & Jaeger, S. R. (2014). Investigation of the numberof consumers necessary to obtain stable sample and descriptor configurationsfrom check-all-that-apply (CATA) questions. Food Quality and Preference, 31,135–141.

Blancher, G., Clavier, B., Egoroff, C., Duineveld, K., & Parcon, J. (2012). A method toinvestigate the stability of a sorting map. Food Quality and Preference, 23, 36–43.

Bruzzone, F., Ares, G., & Giménez, A. (2012). Consumers’ texture perception of milkdesserts II- Comparison with trained assessors’ data. Journal of Texture Studies,43, 214–226.

R Development Core Team (2007). R: A language and environment for statisticalcomputing. Vienna, Austria: R Foundation for Statistical Computing [ISBN: 3-900051-07-0].

Dooley, L., Lee, Y. S., & Meullenet, J. F. (2010). The application of check-all-that-apply (CATA) consumer profiling to preference mapping of vanilla ice creamand its comparison to classical external preference mapping. Food Quality andPreference, 21, 394–401.

40 S.R. Jaeger et al. / Food Quality and Preference 35 (2014) 32–40

Driesener, C., & Romaniuk, J. (2006). Comparing methods of brand imagemeasurement. International Journal of Marketing Research, 48, 681–698.

Ennis, D. M., & Ennis, J. M. (2013). Analysis and Thurstonian scaling of applicabilityscores. Journal of Sensory Studies, 28, 188–193.

Fisher, R. A. (1954). Statistical methods for research workers. Edinburgh: Oliver andBoyd.

Jaeger, S. R., & Ares, G. (2014). Lack of evidence that concurrent sensory productcharacterisation using CATA questions bias hedonic scores. Food Quality andPreference, 35, 1–5.

Jaeger, S. R., Chheang, S. L., Jin, D., Bava, C. M., Gimenez, A., Vidal, L., et al. (2013).Check-all-that-apply (CATA) responses elicited by consumers: Within-assessorreproducibility and stability of sensory product characterizations. Food Qualityand Preference, 30, 56–67.

Jaeger, S. R., Giacalone, D., Roigard, C. M., Pineau, B., Vidal, L., Gimenez, A., et al.(2013). Investigation of bias of hedonic scores when co-eliciting productattribute information using CATA questions. Food Quality and Preference, 30,242–249.

Josse, J., Husson, F., & Pagès, J. (2007). Testing the significance of the RV coefficient.In IASC 07, August 30th – September 1st (2007), Aveiro. InternationalAssociation for Statistical Computing: Portugal.

Krosnick, J. A. (1999). Survey research. Annual Review of Psychology, 50, 537–567.Lawess, H. T., & Heymann, H. (2010). Sensory evaluation of food. Principles and

practices. New York, NY: Springer.Lê, S., Josse, J., & Husson, F. (2008). FactoMineR: An R package for multivariate

analysis. Journal of Statistical Software, 25(1), 1–18.

Lee, Y., Findlay, C., & Meullenet, J. F. (2013). Experimental consideration for the useof check-all-that-apply questions to describe the sensory properties of orangejuices. International Journal of Food Sciences and Technology, 48, 215–219.

Manoukian, E. B. (1986). Mathematical nonparametric statistics. New York, NY:Gordon & Breach.

Meyners, M., Castura, J. C., & Carr, B. T. (2013). Existing and new approaches for theanalysis of CATA data. Food Quality and Preference, 30, 309–319.

Murray, J. M., Delahunty, C. M., & Baxter, I. A. (2001). Descriptive sensory analysis:Past, present and future. Food Research International, 34, 461–471.

Rasinski, K. A., Mingay, D., & Bradburn, N. M. (1994). Do respondents really ‘Mark AllThat Apply’ on self-administered questions? Public Opinion Quarterly, 58,400–408.

Robert, P., & Escoufier, Y. (1976). A unifying tool for linear multivariate statisticalmethods: The RV coefficient. Applied Statistics, 25, 257–265.

Smyth, J. D., Dillman, D. A., Christian, L. M., & Stern, M. J. (2006). Comparing check-all and forced-choice question formats in web surveys. Public Opinion Quarterly,70, 66–77.

Sudman, S., & Bradburn, N. M. (1982). Asking questions. San Francisco, CA: Jossey-Bass.

Valentin, D., Chollet, S., Lelièvre, M., & Abdi, H. (2012). Quick and dirty but stillpretty good: A review of new descriptive methods in food science. InternationalJournal of Food Science and Technology, 47, 1563–1578.

Varela, P., & Ares, G. (2012). Sensory profiling, the blurred line between sensory andconsumer science. A review of novel methods for product characterization. FoodResearch International, 48, 893–908.