Kenny Smith Ems

7/28/2019 Kenny Smith Ems

1/11

JOURNAL OF EXPERIM ENTAL SOCIAL PSYCHOLOGY 16, 497-507 (1980)

A Note on the Analysis of Designs in Which SubjectsReceive Each Stimulus Only Once

DAVID A. KENNYUniversity of Connecticu t

ANDELIOT R. SMITH

University of California at RiversideReceived December 6. 1979

In social psychological experimen ts, the manipulat ions of interest are oftenpresented to subjects along w ith, or as part of, some st imulus. An example wouldbe a man ipulation of the verbal label asso ciated with a stimu lus photograph , in aperson percep tion stud y, Ordinarily it is not possible for stimuli to be com pletelycrossed with treatments because subjects cannot be exposed to any st imulus morethan once. In such designs i t is wise to counterbalance the assignmen t of st imuli totreatment condit ions, but this gives rise to dif ficult ies in the anaIysis o f theexperimental data. Data from su ch designs have frequently been misanalyzed inthe publ ished l iterature. This paper p resents a method o f construct ing a counter-balancing schem e to simpli fy the analysis, and appropriate method s of analysis forboth the simplif ied and the general cas e. It is em phasized that stimuli are generallybest treated as a random factor in such designs, pe rmitting increased generalizabi l-i ty, but the case where the st imu lus factor is f ixed is also considered.

Researchers in social psychology often must embed a manipulation ofsome kind within a stimulus context. For example, in the field of personperception researchers have recognized the inadequacy of designs thatask subjects to respond to two- or three-word stimuli like a black or ahandicapped person. An increased concern with the realism of experi-mental tasks has led researchers to use more complex stimuli (for exam-ple, a paragraph-long description of a person) as contexts in which the

This research was supported in part by National Science Foundation Grant BNS 7826672and in part by the University of Cal i fornia Academ ic Senate. Requests for reprints should besent to David A. Kenny, Depa rtment of Psycholog y, University of Con nect icut, Box U-26).Storrs, CT 06268.

49 70022-1031/80/050497-11%02.WO

Copyright @ 1980 by Academic Press, inc.Ai l rights of reproduction in any form reserved.


2/11

498 KEN N Y AN D SMITHmanipulation of interest (for example, race or handicap) can be em-bedded. In other areas, attitude-change researchers embed manipulationsof source credibility within a persuasive communication; attribution re-searchers manipulate consensus or distinctiveness information in the con-text of a sentence describing some event; and so on. Besides advantagesin the areas of realism or meaningfulness of experimental tasks for sub-jects, the use of multiple stimuli can provide both increased power andimproved generalizability. Power is gained by averaging over multiplestimuli in the same way that power is enhanced by averaging over moresubjects. Generalizability is obtained by replicating treatment effectsacross different stimuli.In Table 1, we have diagrammed five possible designs from amongwhich researchers might choose to estimate the size and consistency(statistical significance) of treatment (manipulation) effects. For simplicityof presentation, each design in the table has four subjects, four stimuli,and two treatments. The designs involve the collection of different sub-sets of data; thus designs 2 through 5 can be viewed as design 1 withmissing data. A given design may cause the researcher to modify themodel (because of carryover effects, heterogeneity of covariance, orposition effects), but these factors are not considered in this paper.

TABLE 1FIVE DESIGNS FOR SUBJECTS, STIMULI, AND TREATMENTS

Subjects St imuli Subjects St imuli

11 B2 B3 B4 B

11 12 13 24 2

11 12 13 24 2

Design 12 3 4B B BB B BB B BB B BDesign 32 3 41 1 11 1 12 2 2

2 2 2Design 52 3 41 2 21 2 22 1 12 1 1

11 lb234

11 12 13 14 1

Design 22 3 41 2c 2Design 42 3 41 2 21 2 21 2 21 2 2

a Both levels of the treatment.b Level 1 of the treatment.c Level 2 of the treatment.


3/11

RECEIVING STIMULI ONLY ONCE 49 9Design 1, the fully crossed design (subjects by stimuli by treatments),

has no missing data and hence has the highest power. Design 2 conf~~~~sstimulus with subject in that each subject receives a different stimulus.Design 3 has stimuli crossed with treatments, but subjects are nestewithin treatments. It has two potential advantages: First, subjects are notaware of the experimental variable, since it is not varied within subject.Second, this design is called for when the experimental treatment isdifficult to change once it has been manipulated (e.g., subjects level ofarousal). Design 4 crosses subjects with treatments but nests stimuliwithin treatments. Such a design is useful if stimuli cannot be crossed withtreatments (e.g., stimulus person within a level of sex). Design 5 counter-balances the nesting of stimuli within treatments. Thus, unlike designs 3and 4, the treatment effect here is not confounded with either subject orstimulus.

Although design 1 is, in principle, preferable to the other designs, it maynot be usable in a particular situation because, for substantive reasons, itmay be impossible to present each stimulus to a subject more than once.One example is a person memory experiment, in which the presentatioof a stimulus person more than once to a subject would confound thememory-dependent variable. As a second example, consider a studyperson perception, where the stimuli are photographs of persons and ttreatments are manipulations of the description of each person. Obviouslyone cannot expose any subject to the same stimulus photograph with twopossibly conflicting descriptions, and this rules out design 1. Designs 2-5meet this restriction, but they are not all equally advantageous. Design 5provides the most efficient estimate of treatment effects because treat-ments are crossed with both subjects and stimu1i.l For these reasonsdesign 5 is in relatively common use among socialMeArthur (1972) used it to test Kelleys (1947) vetheory. However, we know of no text that discussesdesign, and that is the concern of this paper.

CONSTRUCTING THE DES!We refer to design 5 as the counterbalanced design. One cara view it as

either the counterbalancing across subjects of the stimuli within treat-ments, or the counterbalancing across stimuli of the subjects withestreatments. The construction of a counterbalancing scheme wil l be de-scribed below with several examples and some general principles. How-ever, to begin with, in constructing a counterbalanced design, two choices

1 Crossing treatment by subject and st imulus resu lts generally results in higher power todetect treatment effects since the interactions of treatment with sub jects and st imuli (whichwil l b ecome the denominators for F-tes ts of treatment effects in this case) are generallysmaller than the main e ffects of subjects or st imuli (which would be the F-test denominatorsif subjects or st imuli were nested within treatme nts, as in Designs 2, 3, and 4).


4/11

500 KEN N Y AN D SMITHmust be made. First, for each subject, how many stimuli should be in eachtreatment condition? Second, should stimulus be considered a fixed orrandom factor in the design? The first question refers to the number ofreplications (hence the power) and the second refers to the target ofgeneralization. Most studies that employ the counterbalanced design haveused only a single stimulus for each treatment level. Unfortunately, thisstrategy both gives low power and precludes treating stimulus as a ran-dom factor.

If stimulus is treated as a fixed factor, then one can only generalize theresults to the particular stimuli employed in the study (cf. Santa, Miller, &Shaw, 1979). Thus the results of a study of attitude change would belimited to the specific attitude topics that were chosen; the results of aperson perception study to a particular set of photographs, etc. Becauseof this limitation, it has become traditional within cognitive psychology totreat stimulus as a random factor (Clark, 1973). With very few exceptions,however, social psychologists have chosen to treat stimuli as fixed.Perhaps the major reason for this is that treating the stimulus factor asrandom substantially reduces the probability of obtaining a significanteffect. This reduced power is logically and inevitably associated with theincrease in generalizability. However, it seems to us that social psycholo-gists should be willing to pay the price of larger experiments and fewersignificant effects to win increased generalizability.

Once the decisions as to the number of stimuli per condition and thefixed or random nature of the stimulus factor have been made, one mustconstruct a counterbalancing scheme. Table 2 provides the simplest

TABLE 2HYPOTHETICALEXAMPLEOFTHECOUNTERBALANCEDDESIGN

Stimulus set (B)

Subject group (A)B, Stimuli B, Stimuli

12 . . . k k+l k+2 . . . 2kA, Subjects 12

nA, Subjects n+ln+2

2n

T , T ,

T2 T l

No te. Tj indicates level i of the experimental factor.


5/11

RECEIVING STIMULI ONLY ONCETABLE 3

EXPECT ED MEA N SQUARE S FOR THE DESIGN IN TABLE 2Source df T su St TXSU TxSt Sl lXStA 1 X Xb 9WA 2(n-1) X XbB 1 Xb x XbStiBd 2(k - 1) x XA x B (=T) 1 X X Xb xbA x StlBd 2(k- 1) x xB x SulA 2(n - 1) X X0SuiA x StiBd 4(n-l)(k-1) -A

a Derived under the assum ption that A and B are randomly formed groups of subjects andst imuli , respect ively; hence u.?, = U$ = 0. Su=subjects and St=st imuli . An x in the tableindicates that the source of variation at the head of the column is a compon ent of theexpected mean square of the effect shown in that row.

b Equals zero i f st imulus is a f ixed factor.c The sym bol / refers to nest ing. That is, SulA is subjects nested within levels of A.d No t e st imabie i f only one st imulus per treatment level, k = 1.

example of the counterbalanced design, There is a dichotomous experi-mental factor, and the stimuli and subjects have been randomly sub-divided into two sets. The subsets of subjects have been denoted as A,and A, and those of stimuli as B, and BZ. Subjects in Group Al receivelevel 1 on the experimental factor with stimuli in set B, and level 2 withB,; subjects in group AZ receive the opposite pattern (level 1 with Bz anlevel 2 with B,).

ANALYZING THE DESIGNWith the counterbalanced design, as with any ANQVA design, t

construction of appropriate F or quasi-F ratios rests on the examination ofthe expected mean squares for the design. General rules for derivingexpected mean squares can be found in many ANOVA texts. Table 3indicates the components of the expected mean squares for the design inTable 2. Note that no unusual assumptions have been made in derivinthese values, except that subjects and stimuli have been randomly as-signed to subsets. There are two aspects of the expected mean squares inTable 3 to which we wish to direct the readers attention. First, the maineffect of treatments is equivalent to the A x B interaction. Thus, anycomputer program which handles crossed and nested designs can analyzethis particular design; one does not need a specialized program. As wil lseen below, the counterbalanced design can be constructed to have tconvenient property whenever the treatment(s) have a P structure.

Second, the treatment effect can be tested with MSB x Su/A2 as an2 In this and the following formulas, Su represents the subject fac tor, St the st imuli , and

the slash represents nest ing. Th us, Su/A is subjects nested within level of factor A.


6/11

502 KEN N Y AN D SMITHerror term if stimulus is fixed, or by the following quasi-F ratios ifstimulus is random:

orF= MS, x B + M&U/A x St/B

MSB x SulA +MSA x SUB

MS, x BF= MSB x SulA + MS, x St/B - M&,,A x St/B .

A quasi-F ratio is approximately distributed as F (with adjusted degrees offreedom) and is often necessary to test a factor when the design containstwo or more random factors (Clark, 1973).

Some researchers have mistakenly analyzed the design in Table 2 byignoring the stimulus factor, that is, summing or averaging over thestimuli within each treatment condition. If the counterbalancing schemesuggested in this paper is used, and if the stimulus factor is actually fixedrather than random,3 then the consequence of ignoring stimuli in theanalysis is simply lower power. In other circumstances, and particularlywhen the stimulus factor is actually random, the consequences are muchmore severe. Santa et al. (1979) have noted that ignoring stimuli ortreating stimuli as fixed when in fact it is random can have very undesir-able consequences: an inflated value for F. The inflation can be so largethat the actual probability of obtaining the observed F can be 40 to 50times larger than the nominal alpha level, even with only 10 sub-jects. . . . Moreover, the inflation wil l be more severe as the number ofsubjects is increased (1979, p. 39). Thus, researchers should not ignorestimulus as a factor in the design, lest they obtain inappropriately sig-nificant Fs (commit Type I errors). Moreover, it is just as incorrect totake account of stimuli but ignore subjects in the analysis as McArthur(1972) apparently did.

THE GENERAL CASEThe basic design in Table 2 can be enlarged in three ways. There may be

more than one treatment factor. There may also be factors that definesubgroups of subjects (for example, a between-subjects experimentaltreatment or subject sex). Finally, the stimuli may be divided into groups(for instance, there may be person descriptions of different lengths, orphotographs of males and females). The addition of such factors increasesthe complexity of the design, but the same strategy can be employed.

3 Stimuli are fixed if they are not chosen from a larger s et of potential stimuli to wh ich onewishes to general ize; instead, the small set of st imuli actual ly used in the experiment is thefocus of interest.


7/11

RECEIVING STIMULI ONLY ONCEThe basic strategy is to enumerate the number of treatment conditions

to be measured for each subject. If there are 4 such conditions then kqstimuli are required for the experiment where k is a positive integer(ideally larger than one). The kq stimuli are divided randomly into q setsof k stimuli. One now chooses a q by q latin square for the design plan.For certain latin squares where q is a power of 2, one can create dummyvariables for the stimulus set and subject factors such that their interac-tion tests a given treatment effect, rendering unnecessary a specializecomputer program to analyze the latin square design This dummy van-able approach is illustrated in the next section,

If the dummy variable approach cannot be employed and if a programto analyze a latin square is not available,* the following approach can beused. For simplicity assume the treatment has three levels, which woulrequire a 3 x 3 latin square. Let us denote the subject sets as A (with threelevels) and the stimulus sets as B (also three levels). To perform a quasi-Ftest on the treatment effect T one needs four mean squares: T, T x %/A,T x St/B, and Su/A x St/B. Three different ANOVA runs on the dasuffice to yield these mean squares. Treating the data as simply an A xdesign (ignoring T) yields the SuiA x St/B mean square. Treating theas T x A (ignoring B) yields the T and T x Su/A mean squares. Fitreating the data as T x B (ignoring A) yields T (again) and T x St/B.these four mean squares one can form either the simple P; test for T(treating stimuli as fixed), MST / MS T X suiA or the appropriate quasi-P;ratio (where stimuli are random).An Example

An experiment designed and analyzed by this method is described inSmith and Miller (1979), and wil l be briefly discussed here as an example.The experiment was a replication of the study by McArthur (1972) on theeffects of information on causal attributions, with the addition of responsetime as a new dependent variable. Following the theoretical model ofKelley (1967), McArthur varied three types of information (~ons~~sus~distinctiveness, and consistency) pertaining to events presented as sen-tences, and obtained subjects attributions as to what caused the events.A sample sentence might be Sue is afraid of the dog. The threeinformational items were manipulated by additional sentences: high (1~~)consensus: Almost everybody (nobody) else is afraid of the dog; big(low) distinctiveness: Sue is not afraid of almost any other dog; (isafraid of almost every other dog); high (low) consistency: In the pastSue has almost always (never) been afraid of this dog. So a completestimulus presentation to the subject in the high consensus/high distinc-

* The ANOVA procedure of the SAS stat ist ical package is one widely available programthat will handle latin square s.


8/11

504 KEN N Y AN D SMITHtiveness/low consistency condition might be the following: Sue is afraid ofthe dog. Almost everyone else is afraid of the dog. Sue is not afraid ofalmost any other dog. In the past Sue has almost never been afraid of thisdog.

In this study, 32 sentences are the basic stimuli, while the three infor-mational factors applied to the sentences are manipulated orthogonally.(There are thus 32 x 23 or 256 different stimuli in total.) There is also onegrouping factor on the sentences: some involve verbs classified as man-ifest (i.e., overt actions) and others as latent (opinions and emotions, as inthe example). There are no subgroups of subjects in this study. Theexperimenters wished to estimate the effects on attributions of causalityfor four factors (consensus, distinctiveness, consistency, and verb type)and all of their interactions across the hypothetical population of sen-tences from which the set of sentences in the study was drawn.

The design plan is shown in Table 4. The 32 sentences were randomlyseparated into eight sets, each containing four sentences (two of each verbtype). Also, the 24 subjects were randomly assigned to eight groups.Table 5 shows the experimental condition for sentences in each sentenceset for each group of subjects. For instance, for subject group 3, sentenceset 2, the stimuli are presented with low consensus, high distinctiveness,and high consistency. Note that the design plan is actually a latin square

TABLE 4DESIGN PLAN FOR SMITH AND MILLER (1979) STUDY

Sentence setSubjectgroup 1 2 3 4 5 6 7 8

111 112112 111121 122122 121211 212212 211221 22222 2 22 1

12 112 211 111 222 122221121 2

E:F :G :H :I :J :

122 211 212 221 222121 212 211 222 221112 221 222 211 212111 222 221 212 211222 111 112 121 122221 112 111 122 121212 121 122 111 112211 122 121 112 111Subject group1, 2, 3, 4 vs 5, 6, 7, 81, 2, 5, 6 vs 3, 4, 7, 81, 3, 5, 7 vs 2, 4, 6, 8Sentence set1, 2, 3, 4 vs 5, 6, 7, 81, 2, 5, 6 vs 3, 4, 7, 8

1, 3, 5, 7 vs 2, 4, 6, 8Note. The three numbers in each cel l of the matrix designate the levels of consensus,

dist inct iveness , and consistency respect ively (1 = low, 2 = high).


9/11

RECEIVING STIMULI ONLY ONCEsince each cell of the 2 x 2 x 2 design for the experimental factors is ineach row and each column. One should note also that it is actually acombination of four 4 x 4 latin squares and that within each of the 4 x 4squares are four 2 x 2 squares. (See Appendix A for detailed instructionson the construction of the square.) This special latin square facilitates theestimation of the effects of experimental factors and their interactions.Any other latin square could be chosen, but then the dummy variablestrategy described below could not be employed.

For this counterbalanced design, the experimental factors are equiva-lent to the interaction of the stimulus sets by the subject groups. To aid inthe estimation and testing of the factors, three dummy variables arecreated for subject group and three for stimulus set. Table 4 defines thesedummy variables. They are assigned so that the interaction of the firstsubject group factor (E) and the first stimulus set factor (II) yields theconsensus manipulation, the interaction of the second subject factor bythe second stimulus factor yields distinctiveness, and the interaction ofthe third subject factor by the third stimulus factor yields consistency.

The design, therefore, contains nine factors altogether: the threedummy subject factors (E,F,G), the three dummy stimulus factors (J), verb type (V), subject (Su), and stimulus (St). Subjects are nestwithin cel ls of E x F x G, and stimulus within cel ls of V x II x I xSubjects and stimuli are crossed. Table 5 gives a translation for the effectsof the three experimental factors; for example, the consistency maineffect is the G x J interaction. In Smith and Millers study, neither theeffect of subject group, F(7, 16) = .68, nor that of stimulus set, F(7, 16) =36, was reliable. Thus, there was no evidence of a violation of tmodels assumptions of random assignment of subjects and stimuli togroups.

SUMMARYThese proposed approaches to the design and analysis problems faced

by researchers in this type of situation are not complete. Problems remainTABLE 5

TRANSLATION SCHEM E FOR THE DESIGN IN TABLE 4Term Represents factorExHFXIGxJEXFXI- IXIExGxFIxJFxGxIxJExFxGxHxIxJ

Consensus (Cs)Distinctiveness (D)Consistency (Cy)Cs x Dcs x cyD x cyC s x D x C y


10/11

506 KENNY AND SMITHin the areas of treatment of missing data and of order effects (Smith andMiller randomized the order of the 32 stimuli separately for each subject,but this is not often feasible). However, this approach to the constructionand analysis of the counterbalanced design should help researchers avoidmany of the problems that have plagued them in the past, notably themistakes of ignoring stimuli (or subjects) in the analysis and of treatingstimuli as fixed when in fact it is random. While many of the advantages ofthe counterbalanced design have been appreciated by social psychologistsin the past (as evidenced by its frequent appearance in the literature), itsanalysis has rarely (if ever) been conducted correctly. With the provisionof appropriate analyses, the advantages of this type of design now seemstronger than ever.

APPENDIX AAfter dividing the subjects and stimuli randomly into eight sets each,

this latin square was constructed by treating each of the three manipulatedfactors separately, as follows. The first factor, consensus, has its twolevels assigned to subject-stimulus combinations in accordance withTable 2; that is, the first half of the subjects (sets l-4) receive level one ofthe factor with the first half of the stimuli (sets l-4), and so on. The firstsubject factor(E) and the first stimulus factor(H) define the division of thesubjects and stimuli into halves; their interaction defines the consensusfactor.

Now, to place the levels of the second factor, one simply focuses onone-quarter of the overall design matrix and repeats the procedure above.Considering only subject sets l-4 and stimulus sets 1-4, for the moment:assign level one of the second factor (distinctiveness) to subject sets 1-2with stimulus sets l-2 and also to subject sets 3-4 with stimulus sets 3-4;level 2 of distinctiveness goes to the other combinations. (This is in effecta quarter-size replica of Table 2 fitted into the first quarter of the overalldesign matrix.) Duplicate this pattern in the other three-quarters to obtainthe complete assignment of levels of the distinctiveness factor and notethat the second subject factor (F) and the second stimulus factor (I) definedistinctiveness by their interaction when one is finished.

The third factor is assigned similarly. Focus on the first 1/16th of thedesign matrix, subject sets 1-2 and stimulus sets l-2. A 1/16th size replicaof Table 2 again fits here, so that the third factor (consistency) has its firstlevel assigned to subject set 1 with stimulus set 1 and also to subject set 2with stimulus set 2; its second level goes to the other combinations.Duplicate this pattern in the other 15/16ths of the design matrix, andassign the third subject and stimulus factors G and J as the contrast of oddvs even-numbered subject and stimulus sets. Their interaction wil l nowdefine the third factor, consistency.


11/11

RECEIVING STIMULI ONLY ONCEREFERENCES

Clark, H . H . The language-as-f ixed-effect fal lacy: A cri tique of-language stat ist ics inpsychological research. Journal of Verbal Learning and Verbal Behavior, 1973, 12,335-339.Kel ley, H . H. Attribut ion theory in social psychology. In D. Levine (Ed .), Nebraskasymposium on motivation (Vol. 15). Lincoln: Un iv. of Nebraska Press, 1967.McA rthur, L. A . The how and wha t of wh y: So me determinants and consequence s of causalattributions. Journal of Personality and Social Psychology, 1972, 22, 171-193.

Santa, J. L., Mil ler, J. J. , & Shaw , M. L . Using quasi F to preven t alpha inflation due tost imulus variat ion. Psychological Bulle tin, 1979, 86, 37-46.Sm ith, E. R. , & Mil ler, F. D . Attribut ional information processing: A react ion t ime model

of causal subtract ion. Journal of Personality and Social Psychology, 1979, 37, 1723-1731.

Date post:	03-Apr-2018
Category:	Documents
Upload:	jake987722
View:	220 times
Download:	0 times

Kenny Smith Ems

Documents