Measuring Emotion p. 1

TOWARD AN UNOBTRUSIVE MEASURE OF EMOTION DURING INTERACTION:

THERMAL IMAGING TECHNIQUES1

Dawn T. Robinson2, Jody Clay-Warner2, Christopher D. Moore3

Tiffani Everett4, Alexander Watts5, Traci Tucker5, and Chi Thai2

2University of Georgia, 3Lakeland College, 4Arkansas Tech University, 5Stanford University

11The authors would like to thank Marla Wilks, Laura Johnson, Trent D. Mize, Gary Myers, Jun Zhao, Daniel B. Shank, Margaret Devlin, Lynn Smith-Lovin, Aaron Williams, Long Doan, and a large team of undergraduate coders for their contributions to the data collection and coding for this project. This project was supported by National Science Foundation grant SES-0519969 to Robinson and Clay-Warner and BCS0729396 to Robinson, Clay-Warner and Thai, and with funds from the University of Georgia’s Office of the Vice President for Research.

A revised version of this paper was published in 2012 Advances in Group Processes 29:225-266.

Measuring Emotion p. 2

TOWARD AN UNOBTRUSIVE MEASURE OF EMOTION DURING INTERACTION:

THERMAL IMAGING TECHNIQUES

The rigor of contemporary sociological theories of emotion exceeds our current ability to

empirically test these theories. Sociologists need new assessments of emotion that are ecologically

valid, non-reactive, not self-report based, temporally sensitive, and accurate. This paper describes

our endeavor to develop and refine a new methodology, infrared facial thermography, that has

promise for measuring affective responses in the context of dynamic social interactions. We

describe some methodological challenges and advantages of such an approach and present the

results of an investigation that highlights the potential utility of this technique for the measuring

affective responses in group processes research.

Measuring Emotion p. 3

TOWARD AN UNOBTRUSIVE MEASURE OF EMOTION DURING INTERACTION:

THERMAL IMAGING TECHNIQUES

Group processes research in the past few decades has turned its attention in serious ways to

the foundational role of affective processes in social dynamics. Kemper’s status-power theory

(1978, 1991, 2011) describes, in part, how changes in status and power relations during the course

of interactions have systematic implications for social behavior, emotion, and cognition.

Relational cohesion theory (Lawler & Yoon, 1996, Lawler et al., 2000) and the affect theory of

social exchange (Lawler, 2001) describe how affect results from interpersonal exchanges and

shapes our relationships with other individuals and with groups. Lovaglia and Houser (1996)

describe the link between emotions and status in task groups. Affect control theory (Heise, 1979;

2007; Smith-Lovin & Heise, 1988) and identity control theory (Burke, 1991; Burke & Stets 2009)

both describe the relationship between identity meanings, interactional outcomes, and emotion.

The rigor of contemporary sociological theories of emotion, however, exceeds our current

ability to test these theories empirically. Research in this area has been especially hampered by the

inability to measure emotion during ongoing social interaction. Recent advances in neuroscience,

endocrinology, and molecular biology provide new insights on physiological and environmental

factors that affect human emotions, cognition, behavior, and health. Recent developments in

technology provide a wealth of new physiological measurement instruments and procedures that

are more sensitive, less invasive, and more affordable than previously available technology.

Together, these advances offer new opportunities to pursue substantial scientific advancement in

our knowledge of basic social processes. The study of emotion is one domain of human social

dynamics research that stands to benefit from these new advances.

In this paper we use well-understood manipulations and measures of affect to investigate

the utility of facial skin temperature as a measure of affect-based response. Specifically, we use

infrared thermography to assess skin temperature changes in four regions of the face. We record

changes in temperature in the facial regions near the corrugator, obicularis oculi, zygomatic major,

and peri-ocular muscles and analyze the relationship between these changes and self-reported

emotion, evaluation, potency, activity, and identity deflection. These sociologically relevant

Measuring Emotion p. 4

affective states register somewhat different heat signatures on the face, offering hope that infrared

thermography may yield a method for assessing some emotion-related responses in a temporally

sensitive, non reactive, and socially situated manner. We also investigate a number of

methodological concerns regarding this new approach, and provide advice regarding “best

practices” for thermographic measurements of emotion in interactional settings.

BACKGROUND

Virtually all definitions of emotion cited by sociologists contain some element of

physiological arousal and an interpretation of that arousal using contextual cues and cultural

knowledge (see reviews in Thoits, 1989; Smith-Lovin, 1995; Turner & Stets, 2005). Despite this,

sociologists have been relatively slow to attend to the physiological components of emotion in

their theories or investigations. There are notable exceptions to this pattern (e.g., Turner, 2000;

Kemper, 1990; Franks, 2010) and as this volume outlines, sociologists are beginning to pay much

more serious attention the role of biology, including physiological response, in a variety of social

processes.

The development of more affordable, more reliable, and less invasive measurement

techniques has facilitated this recent attention to the relationship between the social environment

and emotion-related physiological response. Researchers have linked human hormone responses

to important aspects of group interaction such as social status (Sapolsky, 2002, Hellhammer et al.,

1997), emotion (Frijda, 1987; Kemper, 1990), workplace stress (Chandola et al., 2010) and

competition (Mazur et al., 1992; Mazur & Booth, 1998). Other research demonstrates how the

cardiovascular system responds to occupying positions of authority (Steptoe et al., 1989) and to

perceived threats (Blascovich, et al., 1999). Researchers have begun to uncover how skin

conductance responds to social injustice (Markovsky, 1988), inter- and intra-group interactions

(Vrana & Rollock, 2002), and various emotional states (Vrana & Rollock, 2002). New research in

vocal analysis reveals a relationship between the spectral composition of speech and perceived

dominance (Gregory & Gallagher, 2002; Kalkhoff & Gregory, 2008). Immersive virtual realities

Measuring Emotion p. 5

are showing promise as potential means to introduce emotional and cognitive stimuli in social

settings that are difficult to manipulate in interpersonal situations (Blascovich, et al., 2002; Troyer,

2008).

These advances offer new opportunities to test and refine existing theories of social

process, as well as opportunities for pushing theoretical development into entirely new directions

by enabling researchers to answer questions previously unconsidered by existing theory. The first

step in knowing whether new measurement technologies or strategies are useful is to clearly

identify what theoretical concepts require measurement. Among the most developed and

productive sociological theories of emotions are the two control theories in the structural symbolic

interactionist tradition – affect control theory and identity control theory. These theories make

dynamic predictions about the experience of emotion in ongoing social interactions and the

relationship between identity process and affective experience. A few years ago, Robinson et al.

(2004) reviewed these (and some related) theories and summarized the key concepts that would be

necessary to test these theories more effectively. They concluded that, at minimum, we would need

to measure affective impressions along the dimensions of evaluation, potency, and activity; we

would need to measure positive/negative emotion; and we would need some measure of “the

generalized stress/surprise/disequilibrium that corresponds to deflection and/or discrepancy”

(Robinson et al., 2004:8). We would like to add that the ability to assess these constructs over time,

and in the context of social interactions, is critical to pushing these theories ahead. Below, we

briefly discuss the role that these constructs play in contemporary sociological theories of emotion.

SOME CORE AFFECTIVE CONCEPTS IN THE SOCIOLOGY OF EMOTION

It is not uncommon to see the words feeling, emotion, affect, and mood used almost

interchangeably. Following previous authors in the sociology of emotion (Gordon, 1981; Thoits,

1989; Smith-Lovin, 1995), we use the term affect to refer to a broad category of experiences that

includes both feeling states as well as sentiments and impressions. We use the term emotion to

refer to transient affective feeling states that involve (a) a situational appraisal, (b) physiological

Measuring Emotion p. 6

changes, and (c) a cultural label.2 Moods, in contrast, are affective feeling states that are more

diffuse, less situated, longer in duration, and generally milder in experience than emotions. The

next two affective concepts, sentiments, and impressions, refer to affective appraisals, or

orientations, rather than feeling states. Sentiments, are relatively enduring patterns of affect

directed toward particular concepts, people, events, rituals, etc. Impressions are more transient

directed affective orientations. In order to observe reliably the relatively more stable sentiments,

one must either (1) measure sentiments toward the same object (e.g., sentiments toward

government) over a number of cultural representatives to derive a cultural sentiment, or (2)

measure sentiments toward the same object by the same respondent a number of different times in

order to derive a stable personal sentiment. Impressions, in contrast can be observed in a single

measure – since they are more situational in nature. Our research design is set up to examine

emotion rather than moods and impressions rather than sentiments. In addition, we explore a

third potential type of affective response, deflection, that is unique to the control theory

approaches. We describe this concept in more detail below.

Evaluation, Potency, and Activity

Identity meanings, along with other social meanings, are indexed in affect control theory

along three universal dimensions of affect – evaluation, potency, and activity (EPA: Osgood,

1962; Osgood, et al., 1975; Osgood, et al.,1957). Recent brain imaging research finds that we

process EPA information about words distinctively and early, far before we process other

information about the meaning of words (Skrandies et al., 2004). Researchers conducted studies in

both Germany (Skrandies, 1998) and in China (Skrandies & Chui 2003) looking at evoked

potentials in response to seeing words that varied in systematically E, P, and A meanings, but that

were similar in number of letters and frequency of occurrence within the language. These

researchers found that even though most studies of brain activation find us processing meaning at

2Thoits (1989) uses a fourth element in her definition: display of expressive gestures. We omit this from our definition because we our research is particularly focused on measuring emotional response even in the absence of emotion displays.

Measuring Emotion p. 7

around 400ms latencies, we see much of the information contributed by EPA by around 270 ms.

Results of these studies are consistent with idea that EPA information is fundamental and

cross-culturally universal.

Evaluation is a good-bad dimension that explains the lion’s share of variance in concept

meanings across 21 different language communities worldwide (Osgood, et al., 1975). Children’s

ability to discriminate along a good-bad dimension develops even before our ability to measure it

(Divesta & Dick, 1966). This dimension corresponds fairly closely to the concept of status in

power-status theory (Kemper & Collins, 1990). Potency is a powerful-weak dimension. It is the

second most explanatory dimension of meaning in language studies (Osgood et al. 1975) and is

also the second dimension along which children learn to discriminate (Divesta & Dick, 1966). It

corresponds closely to the concept of power in status-power theory and across a myriad of

sociological theories. The third dimension, activity, corresponding to a lively-quiet distinction. It

is closely related to what Kemper and Collins (1990) refer to as expressivity and similar to the

construct known as emotional energy (ee) in ritual interaction theory.

Deflection

Both affect control theory and identity control theory are theories of social behavior that

integrate a predictive model of emotions. The control assumption in both theories holds that once

an identity is activated, an actor will strive to maintain the meanings associated with that identity

and will produce behavior consistent with those meanings. Affective meanings associated with

identity labels become points of reference to be confirmed. When social events create new

transient meanings within a social situation, the discrepancy between the fundamental reference

meanings and these transient meanings motivates new behaviors that bring transient meanings

back closer to fundamental meanings. This difference between fundamental meanings and

situation-induced transient meanings is the motivating state. This is called deflection in affect

control theory (and discrepancy in identity control theory). Deflection represents the amount of

disruption in meanings produced by the current interaction. It is experienced by actors as a sense of

unlikelihood (Heise & MacKinnon, 1987; Wiggins & Heise, 1987). In identity control theory

Measuring Emotion p. 8

deflection is presumed to be experienced negatively as stress (Burke,1991) or to inspire discrete

negative emotions such as guilt, fear, and jealousy (Ellestad & Stets, 1998; Stets, 2003). Yet,

researchers in these traditions rarely, if ever, attempt to measure directly the experience of

deflection.

Emotion

There is considerable theoretical controversy about the relative utility of conceptualizing

emotions as discrete states that can vary in intensity (e.g., sadness, anger, joy, fear), or as

continuously experienced affective feeling states that vary along two (evaluation, activity) or three

(evaluation, potency, activity) dimensions. Affect control theory, identity control theory,

status-power theory, the affect control theory of social exchange all make predictions about

discrete emotions, but they also make distinct predictions about the valence or positivity/negativity

of emotion in important cases. Virtually all theories of emotion, whether focusing on discrete or

dimensionally indexed emotion, make predictions about the experience of positive/negative

emotion. As mentioned above, identity control theory predicts a direct, negative relationship

between emotion valence and deflection – identity confirmation leads to more positive emotion,

deflection leads to more negative emotion. In contrast, affect control theory predicts the valence

of emotion based on the valence of the social interaction and the direction of the discrepancy –

positive events and positive deflections lead to positive emotion, negative events and negative

deflections lead to negative emotions. For a more detailed review of this contrast see

Smith-Lovin and Robinson (2006). In order to test almost any of the current theories of emotion

in sociology it would be necessary to measure positive/negative emotion at a minimum.

Temporality

Several of the contemporary sociological theories of emotion, including affect control

theory, identity control theory, relational cohesion theory, and the affect theory of social exchange,

have the capacity to make temporally specific predictions about behaviors and emotions in the

course of ongoing interactions. These dynamics are rarely studied. The research that does test

Measuring Emotion p. 9

dynamic predictions from these theories either focuses on behavioral predictions (e.g.,

Smith-Lovin & Robinson, 1992; Riley & Burke, 1995; Schroder & Scholl, 2009) or interrupts

interaction in order to measure emotion (e.g., Lawler & Yoon 1996). Our ability to exploit fully the

power of these dynamic theories of social interaction would be greatly enhanced by the ability to

measure their core constructs in ways that did not alter or interfere with ongoing social interaction.

MEASURING IMPRESSIONS, EMOTION, AND DEFLECTION

An explosion of recent research on the neurophysiology of emotion has focused our

attention on the central nervous system through the use of positron emission tomography (PET),

near infrared spectoscropy (NIRs) and functional magnetic resonance imaging (fMRI) techniques.

These studies are giving us insight into processes such as the lateralization of brain activity that

will undoubtedly lead to more refined theories about the relationship between the brain and social

experience. These insights are welcome, but in their current form, brain imaging techniques either

remove research participants entirely from social interaction or alter the nature of that interaction

by requiring individuals to be enclosed within, or to wear, noticeable instrumentation. In their

current state, these techniques do not help us in our quest for unobtrusive measures of affect.

Research on peripheral nervous system activity has experienced a similar dramatic

increase. Many of the processes generated by the peripheral nervous system do lead to outcomes

that can be measured in ongoing social interactions. Unfortunately, these techniques have

drawbacks as well. Recent advances in instrumentation for the recording of peripheral nervous

system activity may set the stage for socially unobtrusive measures of emotion – using belts that

can be worn under clothing and transmit wirelessly to nearby computers, for example. The

relationship between autonomic measures and experienced emotion, however, is less than

straightforward (see review in Robinson et al., 2004). And, at present, most of this research uses

less discreet technology requiring thermistors and transducers that attach to the skin on the face

and hands, with visible wires that constrain – and likely alter in other ways – social interaction.

Measuring Emotion p. 10

A variety of physiological measures appear useful both for measuring specific dimensions

of emotion and discrete emotions. The most robust responses seem to be those of the facial

muscles. There are two main approaches to studying facial muscular activity. The more traditional

approach uses videotapes of actors and hand coding of emotion displays. The most developed

approach in this tradition is the Facial Action Coding System (FACS) developed by Ekman and

Friesen (1978). This approach uses frame-by-frame visual coding of muscle movements in order

to look for both intended and unintended (leaked) emotion displays. This approach is extremely

labor intensive and has not been widely used in the context of social interaction research. The

second approach, facial electromyography, (fEMG), measures the electrical activation of specific

muscles in the face. These methods require attaching electrodes to the skin in order to assess

electrical activation of the specific muscles at a specific location. The downside of most of the

currently used physiological measures of emotion lies in the constraints imposed by the

measurement techniques themselves. Since these techniques, and especially, the fEMG research,

give us the clearest information about what to expect when measuring emotion using

thermographic techniques, we briefly turn to this literature.

Measuring Emotion in the Face

When we emote we move our facial muscles in ways that correspond to recognizable facial

expressions. Using facial electromyography (fEMG), researchers can measure undisplayed

emotions in the face as well. Cacioppo and colleagues (1986) proposed that visually undetectable

micro-expressions, corresponding to the discrete emotions could be detected by looking at the

activation of various facial muscles using electrodes attached to the skin. For example, when we

smile we make use of a muscle in our cheeks, the zygomatic muscle. The zygomatic major muscle

draws the corners of the lips upward (Hietanen et al., 1998). Researchers frequently use activation

of the zygomatic major muscle to measure happiness responses (e.g., Dimberg & Petterson, 2000).

The empirical literature on this relationship is somewhat mixed. While Larsen et al. (1992) found

Measuring Emotion p. 11

that positive affect3 increased zygomatic major activity, several other studies report no association

between zygomatic major activity and positive or negative affect (Cacioppo et al.,1992; Wexler et

al., 1992). Bradley and Lang (2000) found no association with the zygomatic major when

participants listened to pleasant versus unpleasant sounds. As suggested by Hietanen et al. (1998),

perhaps this lack of association is due to the fact that the muscles in the lower face are under more

voluntary control of the individual, the zygomatic major being one of those muscles.

The orbicularis oculi, the muscle around the eye area, may also be involved in the

expression of positive emotion. The orbicularis oculi is the facial muscle that causes the corners of

the eye to wrinkle and bag under the eye when smiling. This muscle is thought be under less

voluntary control of the individual than the zygomatic major. Hearing content voices increases

responses in the orbicularis oculi muscle compared to angry voices (Hietanen et al., 1998).

The major facial muscle involved with the expression of negative affect is the corrugator

supercili muscle (Larsen et al., 2003). This is the muscle we use to pull together our brow. When

we do so in an upward direction, we appear sad. When we do so in a downward direction, we

appear angry. Most of the research on the corrugator muscle seems to focus on anger. When

Dimberg and Petterson (2000) presented angry stimuli to participants, they observed an increase in

corrugator supercili activity. Heitanen et al. (1998) observed increased corrugator supercili

activity increased after participants heard angry voices compared to content voices. When Jackson

et al. (2000) asked participants to enhance their negative affect, they observed increased corrugator

supercili activity. In contrast, when they asked participants to suppress negative affect, they

observed decreased corrugator supercili activity. Other researchers report increased corrugator

supercili activity when participants are presented with unpleasant stimuli (scenes, sounds,

3While we assume that what Larsen et al. (1992) actually manipulated was positive emotion, we use the more general term, affect, here and throughout this section, to be consistent with the researchers’ own descriptions. In addition, since the paradigms in the studies described in this section span a wide variety of manipulations and measurements, it may well be that feeling states and impressions are conflated and so the more general term affect may be more appropriate, regardless.

Measuring Emotion p. 12

pictures), compared to pleasant stimuli (Bradley & Lang, 2000; Cacioppo, et al., 1986; Codispoti

et al., 2001; Lang et al.,1990).

In sum, the movements of the corrugator supercili muscle distinguish between the

negative, powerful emotion of anger and the negative, weak emotion of sadness. This is the muscle

that we use to scrunch the brow together when we are angry or sad. When we are angry, we bring

the center part of our brow together and down (Ekman & Friesen, 1975). When we are sad, the

center part of our brow comes together in an upward direction (Ekman & Friesen, 1975). Thus,

when controlling for negative affect, activation of the corrugator supercili may well be useful in

distinguishing between more and less potent affect.

Individuals who experience more active emotions or more expressive transient impressions

move more and with larger motions. It is possible that larger movements in the face are associated

with the activity dimension. Beyond classical approaches to coding body motions from videotape,

there maybe more systematic physiological responses that could help us distinguish activity from

other dimensions of affect. For example, general activity and arousal are associated with increases

in blood pressure and heart rate (see review in Robinson et al., 2004). Unfortunately, most current

measures of affect – both questionnaire based and technology based – constrain movement in

some way, if only by occupying the hands and head orientation. Finally, aroused individuals (in a

positive or negative direction) have a more intense eye blink startle response than individuals with

lower levels of arousal (Cuthbert et al., 1996), suggesting that eyes may be a place to look for

responses to the activity dimension.

The advantage of fEMG over FACS is that it is able to measure activation in muscles

whose movements are not visible to the eye (e.g., a suppressed smile). Another potential advantage

is that it is substantially less labor-intensive than the FACS approach. One key disadvantage of this

method for our current purposes is that researchers must choose specific muscles to investigate

prior to collecting data. Consequently, the method is not well-suited for collecting information

about multiple emotions over the course of an interaction. More importantly, this method is poorly

suited for collecting data during interaction, as electrodes hang from the face and connect an

individual to a data collection unit. As a result, fEMG is socially intrusive and sensitive to motion.

Measuring Emotion p. 13

Recent advances in infrared (IR) techniques for measuring skin temperature may well

enable researchers to assess some aspects of emotional experiences directly, without reliance on

either self-report or transducer based methods, and without requiring an observable facial

behavior. Below we describe the theory behind such a measurement technique.

Thermographic Techniques for Assessing Emotion

All objects emit electromagnetic radiation. The electromagnetic spectrum is divided into a

number of wavelength regions– including x-ray, ultraviolet, visible, infrared, microwaves, and

radiowaves, distinguishable only by differences in wavelength. The infrared band (0.075 -3 μm)

lies just between the visible band and the microwave band. All objects with a temperature above

absolute zero (-273̊C) emit infrared radiation. The radiation emitted by an object depends on two

factors, the temperature of the object (T) and the object’s ability to radiate, or emissivity (ε).

Humans cannot see heat at temperatures below about 500̊C. Traditional cameras measure

the electromagnetic energy in the visible spectrum range (0.4-0.7μm). Infrared cameras measure

thermal radiation in the infrared spectrum range at (0.7-14.0μm). The IR spectrum includes the

reflected IR (near infrared and shortwave infrared) and the thermal IR wave bands (mid-wave

infrared, long-wave infrared and far infrared). The reflected IR wave bands contain no information

about thermal properties of objects or people. But the mid-wave and far-wave thermal bands

enable measurement of heat based radiation emitted from human objects – including human faces.

FIGURE 1 ABOUT HERE

Figure 1 depicts the process of measuring facial temperature using infrared themography. The total

amount of radiation received by the camera is expressed in equation 1.

The three components of radiation collected by the camera include

W W W l Wtot obj ref atm= + − + −ετ ε τ τ( ) ( )1 1 (1)

Measuring Emotion p. 14

1. The emission from the face, ετWobj. This is a product of the temperature of the object, Tobj, the emissivity of the face, ε, and the transmittance of the atmosphere, τ. 2. The reflected emission from any ambient sources, (1- ε) τWrelf, where (1- ε) is the reflectance of the face and Trefl is the temperature of the ambient sources. 3. The emission from the atmosphere, (1-τ)Watm, where (1-τ) describes the emittance of the atmosphere and Tatm is the temperature of the atmosphere.

More simply, there are three sources of heat being registered by the camera – heat from the

face, heat from the air in the room, and heat bouncing off of other sources in the room (e.g.,

computers, other people, etc.). These are the factors we need to attend to in order to interpret a

thermographic image. An object’s emissivity determines its ability to reflect radiation from

ambient objects. Fortunately, human skin has an emissivity of near one (.98; Wolfe & Zissis,

1978), and so it reflects very little radiation in the infrared band. Humidity in the atmosphere can

interfere with the ability of infrared radiation to travel much like fog and rain hamper the ability of

visible light waves to travel. With known values for atmospheric temperature, the transmittance of

the atmosphere (humidity), the emissivity of the object (.98 for human skin), and distance between

the object and the detector, detected infrared radiation can be converted into temperature values.

We will discuss these inputs again later – with an eye to their relevance to measuring human

emotion.

The logic behind measuring emotions in this way is highly related to the logic behind both

FACS and the fEMG approaches. Facial muscle activation should be associated with regional

changes in blood flow patterns, which in turn should be associated with regional temperature

changes in the face. Thermal imaging is widely used in the medical field to assess skin temperature

variation that may indicate increased blood flow due to inflammation or lesions (e.g., Anbar,

2002). While much of the recent work in infrared measurement of human faces centers on facial

recognition (c.f., Kong et al., 2005; Socolinsky et al., 2003), research by Pavlidis and colleagues

suggests that infrared imaging may also be used to detect human emotions (Pavlidis et al., 2002;

Puri et al., 2005). Pavlidis and his research team have used thermography to measure anxiety

brought about by lying, as well as frustration induced by a Stroop test. Pavlidis et al. argued that

lying has a specific thermal signature detectable in the eye region. To measure this "signature"

Measuring Emotion p. 15

Pavlidis et al. (2002) examined blood flow patterns around the eyes. They argued that a fight/flight

response accompanies deceitful behavior, which causes blood temperature around the eyes to

increase, which is revealed through a warming pattern in thermal imaging. Pavlidis et al. (2002)

showed that experimental participants who committed a mock crime and then testified to their

innocence were correctly identified using thermal imaging techniques at a rate significantly higher

than that of a traditional polygraph technique (83% vs. 70%).

Puri and colleagues (2002) used infrared technology to detect frustration. They utilized a

within-subjects design in which they exposed 12 test participants to a Stroop Color Word Contrast

test, which is widely used to evoke stress. While particiapants took the Stroop test, they used an

infrared camera to detect warming in the frontal vasculature (forehead). Utilizing a frame capture

rate of 31 frames per second, they measured the mean temperature of the hottest 10% of the pixels

for each participant in the frontal vasculature and then used these measures to calculate blood flow,

using an algorithm previously developed by the research team (Pavlidis & Levine, 2002). They

also took measures of Energy Expenditure (EE), which are known to vary with stress, in order to

validate the infrared measures. The researchers reported that the correlation between the blood

flow measures and the EE measure was r=.91 when one outlier was removed from the data

(resulting in an n of 11).

Infrared imaging has the advantage of being robust across visible lighting conditions, and

infrared cameras can measure emotions remotely, without the need for reactive measures. Infrared

techniques can also generate fast, sensitive measures that can be analyzed dynamically (Anbar

2002). More importantly, Pavlidis et al. (2002) confirmed that increased blood flow to facial

muscles can be detected using infrared technology. Because researchers have already

demonstrated that emotions are associated with the activation of particular facial muscles, the next

tasks are to validate the use of thermography for differentiating between positive and negative

affect, for differentiating between various discrete emotions, and for measuring micro-expressions

that accompany unexpressed emotions.

Assessing changes in blood flow to different regions of the face is not likely to provide a

meaningful measure of all of what sociologists have in mind when we theorize about affect and

Measuring Emotion p. 16

emotion. Our theories recognize the critical role of language, and make assumptions about the role

of symbolic processing in social interactions. A robust, temporally sensitive, and socially

unobtrusive measure of affective response that could be utilized during the context of ongoing

social interactions would, nonetheless, be invaluable for future theory-driven research in this area.

Consequently, we report below some pilot research and a first investigation about the potential

utility of this approach to measuring constructs of theoretical interest to sociological theories of

emotion in interaction.

PRELIMINARY INVESTIGATIONS

Pilot Research on the Physical Setting

We began our investigation by conducting a series of experiments to determine how the

infrared images are affected by ambient heat and backdrop material. Because the recorded

thermographic images are visual representations of heat, it is vital that the heat being measured is

coming from the intended source (i.e., a research participant). Because the image is a

two-dimensional rendition of three-dimensional space, objects behind the intended target object

may influence the thermographic map of that target object. Of additional concern is the residual

heat left by an object once it is removed from the camera. This heat may be retained by certain

substances that could be used as backdrops.

To determine how ambient heat and backdrop color affects the thermographic images, we

measured the temperature of different areas on the thermographic image using a variety of

backgrounds and in proximity to a computer screen using a number of color palettes. We began by

using a cardboard backdrop, which appeared to retain a certain amount of heat from a face once

that face was removed from the camera's view.

We then checked the effects of various background colors on the infrared detection of heat.

We also measured the temperature of one spot on the image to see if any changes were evident

when a computer panel was situated in close proximity to the object of interest. The computer

screen did not affect the temperature of the object as measured by the infrared camera. We

determined that research participants should be situated in front of the camera with several feet of

Measuring Emotion p. 17

blank space behind them. This would prevent any heat from a backdrop from being detected or

retained once an object is removed. We also found that computer monitors could be placed at a

comfortable distance in front of research participants without substantially influencing

temperature readings (based on the responses of the cardboard).

Makeup and Skin Tone

Given the importance of gender and race in sociological research, we also considered some

potential confounds to the interpretation of gender and self-reported race effects on observed skin

temperature using these techniques. Infrared thermography does not register lightwaves in the

visible part of the spectrum, so visible differences in skin pigment should not make any difference

to the thermographic measurement of skin temperature. Different colors of paint, however, have

been observed to have different emissivity (Bramson, 1968), due to differences in material

content. It is possible (though unlikely) that skin pigment chemically alters the skin’s emissivity,

or that it alters skin temperature, or is by other means correlated with differences in skin

temperature. In our queries to physiologists and engineers, we could find no existing answers to

these questions. So, in a separate study in our lab, Mize and Myers (2010) analyzed data from

respondents viewing evocative images and found consistent differences in facial temperatures

between individuals who self-reported as “Black” (n=21) and those who self-reported as “White”

(n=64). These main effect differences did not vary by self-reported emotion or by the type of

image being viewed. They further found that the temperature difference was even more strongly

related to coded skin tone of the respondents (coded from conventional video, alpha=.98) than it

was to self-reported race. As with self-reported race, the measure of skin tone was not otherwise

related to differences in emotion (as measured by self-report or by other physiological

instruments). This analysis suggests that we cannot rule out some systematic measurement

confound associated with skin tone when using infrared thermography. This question calls

strongly for additional research. In the meanwhile, we recommend the interpretation of

temperature change, rather than unconditional measures of temperature, for sociologists interested

in facial temperature.

Measuring Emotion p. 18

We also investigated a potential gender confound via the wearing of cosmetics, especially

foundation makeup (Baker et al., 2010). Four lab assistants took measurements of on regions of

their face and arms before and after applying makeup to one of their arms and to one half of their

faces. Their results showed that (1) applying makeup cooled the skin, including regions of the

skin without makeup but near the areas with makeup, and (2) regions of the skin with makeup

cooled more intensely than the adjacent regions, and (3) the effects of makeup application

disappeared quickly, with no discernable temperature difference between the two arms or the two

halves of the face remaining after about five minutes. Thus, we concluded that makeup, as long

as it is not applied during the thermographic measurement, should not confound temperature

readings using infrared thermography.

Pilot Research on the Experimental Method

We then turned to a preliminary investigation of the sensitivity of our thermal imaging to

positive emotion. Our initial investigation utilized a small within-subjects design (n=8) to examine

infrared images in four regions of the face: (brow – corrugator, eyes – ocular/periocular, cheeks –

zygomatic major and minor, and mouth – orbicularis oris) both before the initiation of a known

positive emotion manipulation and after the manipulation. Following a design adapted from

Robinson and Smith-Lovin (1992) and described more thoroughly below, participants delivered a

prepared speech to an unseen rater and then received feedback on their (purported) social skills.

This study serves as a useful paradigm for this investigation because it allows us to separately track

emotion and deflection. Previous research using this design (Robinson & Smith-Lovin, 1992;

Swann et al., 1987) demonstrated that, as predicted by affect control theory, the valence of

emotional response were strongly predicted by the positive/negative nature of the feedback, while

measures related to deflection were strongly predicted by the interaction between feedback and

identity meanings.

In this pilot study, all participants received extremely positive feedback. Based on previous

research (Robinson & Smith-Lovin, 1990; Swann et al., 1987), we know that this manipulation

produces strong positive emotion. We measured temperatures in the four facial areas (a) just after

Measuring Emotion p. 19

delivering the speech, and again (b) just after participants received the positive feedback.

Encouragingly, t-tests revealed that (as predicted) mouth and cheek temperature rose, while

forehead and eye temperature did not. Even in this very small study, the results were statistically

significant for the mouth (p < .01) and marginally statistically significant for cheeks (p=.06).

Due to the within-subjects nature of the first pilot study design, we naturally cannot trust

that the observed effects were due to positive affect, rather than to the experience of being

evaluated or even to the act of giving a speech. Consequently, we conducted an additional pilot

study along with a larger study in which we independently manipulated positive feedback and

negative feedback. We also included a control condition in which we measured responses just

prior to receipt of feedback which was delayed (by one minute in the pilot study; two minutes in

the focal study). This delay allowed us to compare responses of those receiving positive or

negative feedback to a control group whose members experienced giving a speech and anticipating

the feedback in the same way as the treatment group. This allowed us to control on the stress of

speech-giving, and of anticipating social feedback in our comparisons across groups. The pilot

study allowed us to develop and refine our experimental procedures and our data collection,

coding, and analysis protocols. In particular, we used data from the larger pilot study (N=42) to (1)

refine our procedures for isolating the theoretically relevant pixel data from the stream of thermal

data, (2) develop a computer program for extracting and analyzing the relevant pixel data from the

proprietary thermography software in order to link it to our other sources of data, and (3) improve

our ability to synchronize the data streams coming from various other forms of instrumentation.

We also used various observations from this study to inform decisions made in the focal study and

will refer to these as we describe the full experiment.

PRIMARY INVESTIGATION

In order to generate variation in affect, evaluation-, potency-, and activity- self-sentiments,

and identity deflection, we modified the design used by Robinson and Smith-Lovin (1992) which,

in turn, was adapted from Swann et al. (1987). In this paradigm, individuals are provided with

Measuring Emotion p. 20

either very positive or negative feedback about their social skills. A measure of social self-concept

is used to generate theoretical predictions about the deflection experienced by participants in the

two conditions. Self-reported measures of evaluation, potency, activity, and emotions were

collected after the feedback and while measuring facial temperature. Based on past research, we

expected identity deflection to correspond to an interaction between social self-concept and

feedback condition and valence of emotion to correspond to the positive/negative nature of the

feedback. Procedures closely followed Robinson and Smith-Lovin (1992) with a few key

exceptions.4 A brief description of the methods follows. For more detailed information see

Robinson and Smith-Lovin (1992).

Participants and Setting

Experimental participants were 114 undergraduate students at a large southern university

who volunteered to participate in the study and who received $15 as compensation for their time.

The setting was a medium size (approximately 10' x11') laboratory room equipped with a small

table with a computer mouse, a document stand, and a wall mounted monitor. Mounted just below

the monitor, with the detector in front of the screen, was a FLIR Thermacam SC3000 with a

Sterling-cooled quantum well infrared photodetector (QWIP) with focal-plane-array (FPA)

technology operating in the 8-9 micrometer wavelength. The mid-wave IR sensitive chip consists

of 320 x 240 pixels having a high sensitivity and a thermal sensitivity of 20 mK at 30 C̊ and a scan

rate of 50/60 Hz.

Procedures

Participants were recruited via courses and flyers to sign up for a study in interpersonal

communication. Upon arrival, participants were escorted from a waiting area into the laboratory

4 These exceptions include asking participants to give a speech to a camera rather than to two one-way mirrors, the monitoring of physiological measures using electrodes for assessing skin conductance, and use of a thermistor for recording skin temperature. These other physiological measures were for purposes beyond the scope of the investigation in this paper, but are important to note since they might influence participants’ emotional responses.

Measuring Emotion p. 21

and seated in the room described above. The experimenter then described the purpose of the study

in the following way:

In this study, we are validating two procedures that we have developed that assess communication skills using only non-verbal cues in one of these procedures, and only verbal cues in the other procedure. By comparing the results from these procedures with your feedback about the evaluation process and some physical measures, we hope to determine how good our procedure actually is.

Participants read and signed a consent form, which informed them that they would be

videotaped and that we would collect additional physiological measures. The nature of the infrared

camera was not explained at this time. The experimenter asked the participant to remove a hat, if

wearing one, to spit out any gum, and to turn off his or her cell phone. Participants were also asked

to remove eyeglasses if it was possible to read what was on the computer screen without them.

After consenting, the experimenter left the room and the participant completed a short

“Background Questionnaire” which contained the short form of the Texas Social Behavior

Inventory (Helmreich et al., 1974). The experimenter then returned to the room and provided the

following instructions and information:

In the first part of this study, we would like to see how accurately two different trained observers can judge social skills based only on non-verbal behavior using two new rating systems. I am going to ask you to read a short passage to this rater. The observers are across the hall, viewing you via this camera (point to camera). These observers have each been trained in a different system of judging social skills based only on non-verbal cues. In the second part of this study, you will be paired with a participant who is scheduled to arrive in about 20 minutes. The two of you will then have a face-to face conversation in a different room that we will record and review the audio-only portion of it at a later time using another new system of judging social skills based only on verbal cues. When we reach the second part of the study, I will explain it a little more. When the door to this room is closed, this room is sufficiently soundproof so that the raters will not be able to actually hear anything you say. They will, however, be able to monitor your non-verbal behavior during the presentation. Because what you say makes little difference, and we want to make sure that all of our speakers are equally prepared, I have some material prepared for you to read for your presentation.

Measuring Emotion p. 22

After you finish your presentation, you will be able to see and evaluate the ratings from the observers. Your feedback on their ratings will help us to improve our assessment technique. The way we intend to test our new ratings system of non-verbal communication is to compare the ratings from the scale with physical measures collected from participants like yourself, as well as with the ratings we collect in the second, verbal-only, part of the experiment. But now, to collect the physiological measures, we will attach these devices to three of your fingers and your hand (point to items on table). This should not be in any way painful or uncomfortable. If at any point you are uncomfortable, please let me know.

Instruments to be placed on the participant’s hand were swabbed with a sterilizing pad and

attached to the non-dominant hand. The experimenter then asked the participant to read from a

screen and made adjustments in display size, if necessary, for those who had removed eyeglasses.

The experimenter than left the room. Next, participants read the speech to the camera. The speech

was a passage from Jonathan Livingston Seagull that took approximately three minutes to read.

After the speech, the experimenter told the participant that the observers would need two or three

minutes to transfer their ratings onto summary sheets.

After the three-minute interval, the experimenter left the experimental room and returned

with two rater feedback summary sheets. In actuality, the feedback was prepared ahead of time and

was identical for all participants (except that the gender of the pronouns was matched to the

participant's gender). The feedback summary sheets consisted of a series of seven-point Likert

scale questions at the top, with a space for general comments at the bottom. On the positive

feedback sheet, Rater A characterized the participant as a 7 ("considerably higher than the average

student"-the highest category) on social expertise, communication skills, and confidence scales.

The rater gave the participant a 5 ("slightly higher") on interpersonal sensitivity and a 6

("somewhat higher") on leadership ability. The participant was not rated at or below the midpoint

(4) of any scale. In addition, the positive feedback sheet contained these remarks in longhand

under a section labeled "General Comments":

The presenter is clearly a poised and graceful speaker. Her posture during the reading gave me the impression that she is at ease in social situations. The way

Measuring Emotion p. 23

she communicated through movements of her head made me really want to hear what she was saying! I imagine the presenter is very outgoing, with many close friends. That's about all I could say about her.

On the negative feedback sheet, Rater B gave the participant no rating higher than the

midpoint of the scale. On leadership ability, communication skills, and confidence, the rating was

the next-to-lowest category (2). In the "General Comments" section, rater B wrote:

From her gestures, it appeared that the presenter was nervous and uncomfortable. This was especially evident by the way she looked down when she was speaking. It is hard to tell after such a brief observation, but I think that this presenter is probably awkward around groups of people and does not do well speaking to strangers.

Observers were identified as "Rater A" and "Rater B" and were always presented as

the same gender as the participant. After the participant read the feedback, the experimenter asked

him or her to complete a questionnaire assessing first affective and then cognitive reactions to the

feedback. In the control (delay) condition, the order of the positive and the negative feedback was

counterbalanced such that one half of the participants read the negative feedback first and the other

half read the positive feedback first. When they had completed the requested measures, the

participants were thanked and thoroughly debriefed using a process debriefing approach (Aronson

et al., 1990) in which participants were encouraged to think aloud about the ways in which they

might be affected by the feedback (including throughout the day ahead). This method has been

demonstrated to protect against lingering effects of false feedback (Aronson et al., 1990). In

addition, the experimenter used the debriefing interview to probe for suspicion. Reactions during

post-experimental interviews showed that most participants found the cover story fully

convincing. Four participants reported significant suspicion during debriefing. Four were in the

control condition. Two were in the negative feedback condition. All were in the medium social

self-concept category. On the assumption that suspicion would be likely to dampen affective

responses to our feedback manipulation, we took the conservative approach of not deleting any

cases based on suspicion. One of the individuals in the treatment condition, however, had long

Measuring Emotion p. 24

bangs and wore glasses during the experiment and so was not included in the analyses for separate

reasons described below. Data from the four suspicious participants in the control condition were

not included in any analyses containing deflection (which only used the positive and negative

feedback conditions). So, it is unlikely that suspicion was a factor in the reported analyses.

Emotional Response to Feedback

We administered the long form “state” version of Zuckerman and Lubin's (1985) Multiple

Affect Adjective Check List-Revised (MAACL-R) to assess emotional responses to the feedback.

We used the Positive Affect5 sub-scale, which corresponds most closely (only in reverse direction)

to the Negative Affect sub-scale from the original MAACL used by Robinson and Smith-Lovin

(1992) and by Swann et al. (1987).6 The Positive Affect sub-scale of the MAACL-R is a reliable

and valid measure of low arousal positive emotion based on an index of 21 positive emotion labels

(e.g., calm, joyful, pleasant; Lubin et al., 1986). The pattern of statistical significance reported in

our regressions below was identical when using the other positive emotion sub-scale, positive

affect plus sensation-seeking, which is a composite measure of high and low arousal positive

emotion. The sensation-seeking sub-scale on its own has relatively lower internal reliability

(Lubin et al., 1986), so we decided to report our results using the more interpretable positive

emotion sub-scale.

Evaluation, Potency, Activity

We also assessed identity impressions along evaluation, potency, and activity using

self-report survey measures. We measured each of these identity impressions using two bi-polar 5We use the MAACL-R Positive Affect sub-scale to measure positive emotion, but we retain the copyrighted name of the scale (Positive Affect) for consistency with other research literature using this measure.

6The Negative Affect sub-scale used in the previous studies was a bi-polar measure of emotion valence. The new MAACL-R separates Positive Affect from three types of negative emotion (Anxiety, Depression, Hostility). Since we are conceptualizing emotion valence as a bi-polar dimension in this study, any of these should do for our purpose. The three negative sub-scales have been shown to be less reliable in at least one study (Hunsley, 1990). Consequently, we decided to use to use the Positive Affect sub-scale.

Measuring Emotion p. 25

scales with 36 positions. Participants were asked to think about “how characteristic each adjective

pair is of you.” Each semantic differential item included “Myself” in the center of the rating scale

and included the following adjectives as anchors: (1) evaluation – nice/awful, good/bad; (2)

potency – powerful/powerless, strong/weak; and (3) activity – active/passive, noisy/quiet. For

each dimension of identity meanings, we averaged the two ratings into a composite scale.

Deflection

We constructed a measure of deflection that was based on the experimentally manipulated

feedback, non-experimental variation in participants’ self-reported social self concept, and affect

control theoretical predictions about deflection generated from computer simulations using

INTERACT. Following the earlier studies (Swann et al., 1987; Robinson & Smith-Lovin, 1992),

we used the short form of the Texas Social Behavior Inventory (TSBI; Helmreich et al., 1974) as

the measure of social self-concept. The scale measures social self-concept with items like "I would

describe myself as socially unskilled," "I usually expect to succeed in the things I do," and "I feel

that I can confidently approach and deal with anyone I meet." The TSBI is a 16-item test scored on

a six-point Likert scale with a total range of 16 to 80. We chose the TSBI because of its focus on

"social self-concept." This is a domain-specific self-concept measure that is not related to global

self-esteem and not related to depression, has good variation in the general population, and is

relatively stable within person. So, it serves as an appropriate identity measure for a study of

college students. As stated above, we administered it as part of a "background information"

questionnaire at the beginning of the study, but did not score it until after the experiment. The

experimenter, therefore, was blind to the participant's score during the experimental procedure.

The designation of an individual as a person with high or low self-concept is not

trivial in this research. Out of a possible 16-80 range of scores on the TSBI, our sample range was

30-77, with a mean of 60.3. These scores are considerably higher than the normative scores for the

general population reported by Helmreich et al. (1974), but in line with other studies of college

students (Baumeister et al.,1989; Robinson & Smith-Lovin, 1992; Swann et al., 1987). For

designation of low self-concept, Swann (1987) recommended the use of cutoffs at the lower 10 to

30 percent of undergraduate samples. Following Swann (1987) and Robinson and Smith-Lovin

Measuring Emotion p. 26

(1992), we used a score of 51 as the cutoff for low social self-concept. Simulations using the 2003

U.S. sentiment dictionary in Java INTERACT (Heise, 2011) enabled us to calculate deflection

scores.

The most relevant actor-identity in the INTERACT dictionary seemed to be that of

Instructor. Consequently we used Instructor to represent the rater, the behavior Praise to

characterize positive feedback, and the behavior Criticize to characterize negative feedback. We

identified a number of potentially relevant object-identity profiles: HIGH SOCIAL SELF

CONCEPT IDENTITIES: Popular Student, Confident Student, Outgoing Student, Popular

University Student, Confident University Student, Outgoing University student, Popular

Undergraduate, Outgoing Undergraduate, Confident Undergraduate; LOW SOCIAL SELF

CONCEPT IDENTITIES: Unpopular Student, Insecure Student, Introverted Student, Unpopular

University Student, Insecure University Student, Introverted University Student, Unpopular

Undergraduate, Insecure Undergraduate, Introverted Undergraduate. We averaged the

amalgamated EPA profiles of those trait-object identities in order to create a single EPA profile

that best captured the identity in question.7

We then conducted a search in INTERACT to find the object-person identity that was the

closest match to this profile. Using this approach, Client was the identity that provided the best

profile match for a high social self-concept participant, and Defendant was the best match for a low

social self concept participant. We subsequently conducted simulations for corresponding to each

combination of high and low social self-concept and the two feedback conditions: (a) Instructor -

Praises - Client; (b) Instructor - Criticizes - Client; (c) Instructor - Praises - Defendant; (d)

Instructor - Criticizes - Defendant. Simulations yielded deflections ranging from 0 for “instructor

praises client” to 5 for “instructor criticizes client,” with the other two events falling in between.

These simulation results were used as the values for deflection in our analysis of facial temperature

below. 7It should be noted that since INTERACT processes only the affective meanings of words and not their semantic meanings, no value is lost in reducing the EPA value of the trait and the EPA value of the object into a single EPA value that combines the two. This reduction merely simplifies the A-B-M-O equation into an A-B-O equation (Robinson & Smith-Lovin, 1992).

Measuring Emotion p. 27

Facial Temperature

The FLIR Thermacam SC3000 is internally cooled to 7k (-267.15 C̊). It takes

approximately six minutes to reach temperature and is noisy during the cooling period. So, the

experimenter set up the camera, software, and other equipment approximately 15 minutes before

the participant’s scheduled arrival. Because we knew based on previous pilot work that the

temperature responses we intended to measure were relatively slow, we recorded pixel data for this

study at three frames per second. A wall thermometer/hydrometer located in the participant room

recorded temperature and humidity and remotely displayed that information in the control room.

At the beginning of each experimental session, the experimenter recorded the object parameters

necessary to convert the detected radiation into temperature values – emissivity (.98), distance

(1m), atmospheric temperature (observed) and humidity (observed) – into the Thermacam

Researcher Pro software that controlled the camera and processed the pixel data. The experimenter

set the camera to begin recording just after participants consented and before beginning the

instructions for the session described above. The thermal data was recorded until the end of the

study.

To process the data, a team of coders extracted pixel data from the Researcher Pro

software. This was accomplished in multiple stages.8 First, pre-coders identified the specific

frames for analysis, corresponding to predetermined points in the experiment and meeting certain

other criteria. In the treatment condition, these times corresponded to (1) beginning of session, just

before completing the pre-session questionnaire, (2) just after completing the pre-session

questionnaire, (3) beginning of speech, 4) one minute into the speech, (4) end of speech, (6)

beginning of wait (for feedback), (7) two minutes into the wait, (8) during receipt of the feedback,

just before completing the post-feedback questionnaires, (9) just after completing the

post-feedback emotion questionnaire, (10) just before completing the post-feedback rater

questionnaire, and (11) just afer completing the post-feedback rater questionnaire and just before

the experimenter re-entered the room. In the control (delay) condition, these times correspond to

8A more detailed coding and analysis protocol is available from the first author.

Measuring Emotion p. 28

(1) beginning of session, just before completing the pre-session questionnaire, (2) just after

completing the pre-session questionnaire, (3) beginning of speech, (4) one minute into the speech,

(4) end of speech, (6) beginning of wait (for feedback), (7) two minutes into the wait, (8) three

minutes into the wait, (9) beginning of emotions questionnaire (delivered prior to the feedback in

this condition), (10) just after completing the emotion questionnaire, (11) during receipt of

feedback (both positive and negative), (12) just afer reading feedback, (13) just before completing

the post-feedback rater questionnaire, and (14) upon completion of the post-feedback rater

questionnaire.

Pre-coders identified frames at each of these points in the experiment, trying to avoid

frames where the participant was out of focus, out of range, had a head turned, eyes closed, hands

on face, or mouth open. They used a 3 second window for each predesignated period to allow some

flexibility in avoiding problematic frames. This allowed up to 9 frames from which to choose for

each time point. The actual times were recorded with the pixel data as well. In our earlier pilot

studies, we coded two frames (adjacent, or a maximum of 2 frames apart) for each time period to

see whether extra measurement precision was afforded by this additional data. The difference

between using 1 frame or 2 were indiscernible, so we substantially reduced the labor by switching

to single frame coding.

A second team of trained coders then went to the identified frame and extracted the

relevant pixel data into Excel sheets, by drawing areas corresponding to the four regions of interest

on the face, and then extracting the pixel data from those drawn regions into a spreadsheet. We

developed a batch program to convert all of these Excel files into summarized temperature data by

id, time, and facial region. Following Pavlidis (2002), we calculated the average temperature of the

warmest 10% of the pixels in each extracted region. Most errant data extracted by these kinds of

procedures would be data that is artificially too cool – due to capturing facial hair, eyelashes, open

mouths, a bit of air beyond the face, etc. So, measuring variation in only the top 10% warmest

pixels dampens variation considerably but creates more reliable data. In analyses of the earlier

study, we explored using cutoffs of 5, 10, 20, 30, 40, and 50%. We found that restricting to 5%

noticeably reduced variation and statistical power. We found very little difference between the

Measuring Emotion p. 29

results relying on the top 10, 20, or 30% warmest pixels. Inclusion of the top 40% and 50%

introduced considerable noise into the data, resulting in low reliability So, following previous

research, we opted to use the warmest 10% for the analyses presented here.

FIGURE 2 ABOUT HERE

Panels A and B in Figure 2 show relatively typical images, with the pallet representing

different temperatures – lighter regions representing warmer areas and darker regions representing

cooler areas. Panels C through F illustrate some of the complications of coding and interpreting

thermographic data. Motion, moving out of camera range (Figure 2c), and fingers obstructing the

face (Figure 2d) are the same kinds of challenges that might be faced by a researcher coding facial

information from conventional video. Facial hair poses particular problems for thermographic

analysis of skin temperature, since hair is considerably cooler than skin and provides no

information about facial warming. Heavy bangs and facial hair like that seen in Figure 2e would

downwardly bias temperature readings, even when using the 10% warmest pixels. Glass allows

visible light waves to pass through, but blocks radiation in the infrared spectrum. Consequently,

eyeglass make it impossible to measure temperature in the eye region (Figure 2f).

RESULTS

Experimental Findings

Although the experimental results and the replication of Robinson and Smith-Lovin

findings are not the focus of this paper, some reference to those results will be helpful for

understanding the utility of thermographic techniques for measuring emotion. Results from a 3

(feedback: positive, negative, delay) x 2 (social identity: high, low) MANOVA indicated that

experimental findings replicated those of Robinson and Smith-Lovin (1992) and Swann et al.

(1987). Specifically, positive feedback was associated with more positive emotion and less

negative emotion, and social self-concept interacted with feedback condition to predict

perceptions of identity confirmation. These results were all statistically significant in the predicted

Measuring Emotion p. 30

directions and comparable in magnitude to the earlier studies, suggesting that the study

conditionsproduced systematic variation in affect and deflection.

Facial Temperature

Over the course of the experimental session, recorded brow temperatures ranged across

individuals from 31.69 C̊ to 36.72 C̊, with an average of 34.88 C̊in the time periods before the

feedback and 35.11 C̊ in the time periods after receiving the feedback. Eye temperatures varied the

least, ranging from 33.53 C̊to 36.81 C̊ with an average of 35.32 C̊ in the time periods leading up to

the feedback and 35.53 C̊ in the time periods after. Cheek temperatures were the coolest and

exhibited the most temperature variation, with readings ranging from 30.75 C̊ to 36.2 C̊ and

averaging 33.76 C̊before the feedback and 34.04 C̊afterwards. Mouth temperatures also ranged

widely, from 32.49 C̊ to 37.10 C̊ and averaging 34.96 C̊ before the feedback and 35.50 C̊in the

periods after the feedback.

FIGURE 3 ABOUT HERE

We reported above that in our small within-subjects pilot study of responses to positive

feedback, we found the largest effects for change in mouth temperature. Figure 3 displays the

temperature trajectories of the mouth for each individual by feedback condition. Notice the

substantial heterogeneity. In general, the variations between people were larger than the variations

within person over time, making it imperative to use within-person comparisons for any

inferences. This pattern was even more true for the other regions of the face, for reasons that will

become clear below. Consequently, our primary analyses are of within-person temperature change

rather than of simple mean temperature.

FIGURE 4 ABOUT HERE

A comparison by condition of the mean mouth temperature from just before the feedback

(after the speech) and just after the feedback reveals dramatic warming. Figure 4 shows the means

across these conditions. The change in all three conditions was approximately half a degree C, a

very large change in temperature by standards of previous research. Nonetheless, this change was

Measuring Emotion p. 31

virtually identical across the three conditions. Looking at the timeline for the whole experiment

reveals the reason.

FIGURE 5 ABOUT HERE

Figure 5 summarizes the changes in mouth temperature over time across the three feedback

conditions. Note that the trajectory was remarkably similar in all three conditions. When the

experimenter left the room, the participant’s mouth began to warm. When the participant began to

speak, the participant’s mouth began to cool and did not begin warming again until the speech was

over. Speaking carries moist air over the lips, which cools down the temperature of the skin on the

lips. Even though we only extracted temperature data from frames where participants had a closed

mouth, these frames were necessarily near others where the mouth was open during the speech

itself. The rate of post-speech warming did slow in the positive condition relative to the other two

conditions based on some analyses.9 It is possible that this is a result of greater rates of lips parting

more frequently after positive feedback – perhaps smiling? Even if that is so, this indirect way of

capturing a visible facial movement (lips parting, laughing, smiling) would not be an efficient

measurement strategy. So, we elected to omit all measures of temperature in the mouth region

from our analyses reported below.

Figure 6 displays the trajectory over the experiment of measured facial temperature across

the other three regions by feedback condition. From this display we can see that the eyes are the

warmest, the cheeks are the coolest, and that all three regions of the face generally warmed over

the course of the experiment. Vertical reference lines indicate the first recording taken after

receiving rater feedback. The first line shows the feedback marker for the positive and negative

feedback conditions. The second line indicates when participants in the control condition received

their feedback. These trajectories are smoothed across significant heterogeneity as depicted in

Figure 3. Consequently, the changes in response to feedback appear subtle. However, the brow

temperature appears to warm after feedback in all three conditions (see increase at first line in the

9These and other results of the experimental conditions on facial temperature are available from the first author.

Measuring Emotion p. 32

two feedback conditions and at the second reference line in the delay condition). The cheeks

appear to warm with negative feedback and with the combined feedback at the end of the delay

condition. Eye temperature appears to warm with positive feedback and cool with negative

feedback , but since it also appears to be warming at that same time in the control condition, the

pattern is far from compelling. The purpose of this experiment, however, was to produce variation

in identity impressions, deflection, and emotion and investigate the utility of infrared facial

thermography for assessing these responses. So, now we turn to a discussion of the relationship

between these affective responses and regional changes in facial temperature.

Affective Responses and Changes in Facial Temperature

As indicated by the large individual differences and the possibility of a confound with skin

tone described above, we used models of within-subject temperature change to investigate the

relationship between affective responses and changes in blood flow in the face. Within

sociology, there is considerable debate about the appropriate way to investigate the effects of

independent variables on within-person change. Allison (1990) notes that there are two

competing preferred methods (1) what he calls the regressor approach – regressing the

independent variables on the dependent measure at T2, while controlling for dependent measure at

T1, and (2) what he calls the change score approach – regressing the independent variables on the

difference in the dependent measure at the two time periods (T2-T1). Allison and others (1990;

Stoolmiller & Bank, 1995; Morgan & Winship, 2007) argue that when the dependent measure at

T1 is uncorrelated with the independent variables (or the selection into the treatment), the change

score method is more reliable. We believe that to true in the present case10, but we cannot be

absolutely certain. In the case of a true experiment, this would be clear. Our present research,

however, combines a non-experimental covariate (social self-concept) as part of one the primary

independent variables (deflection). In addition, our other independent variables (evaluation,

potency, activity, emotion) are all forms of affect that we understand to be related to previous

affect. Consequently, we cannot be entirely certain that temperature readings earlier in the study

10Indeed, we did not observe any statistically significant bivariate correlations between any of the independent variables and the “before” measures of temperature in any of the facial regions.

Measuring Emotion p. 33

are not related to earlier differences in affect that end up affecting later affective responses. So,

we conducted our regressions using both approaches. The results were nearly identical. The best

fitting model was the same for both approaches. The parameter estimates were highly similar

across the board. The R2 values for the regressor approach are considerably higher and harder to

interpret, since they include the stability coefficients. There were a few terms that increased in

statistical significance when using the regressor approach, compared to the change score approach,

but no change in the substantive interpretation. Therefore, since some have argued that the

regressor approach may downwardly bias standard errors (Bertrand et al., 2002), and since the

primary consequence of using that approach was to increase the statistical significance a few

terms, we elected to report the analyses below using the change score method.

TABLE 1 ABOUT HERE

Table 1 contains the results of a series of regressions examining the relationship between

warming in the various facial regions and self-report measures of positive emotion, self sentiments

evaluation, potency, and activity. The dependent measure was temperature change in response to

the feedback (temperature just after reading the feedback minus temperature after the

speech/before reading the feedback). The “after” measurements occurred just after the feedback

was read and while respondents were completing the emotion instruments. For each of the three

facial regions, we regressed temperature change first on positive emotion, then on EPA

self-sentiments, then on deflection and EPA, next on emotion and EPA, and finally on all five

variables. Self-reported positive emotion alone predicted brow temperature, with less positive

emotion predicting higher rates of warming (model 1). Model 2 examined the ability of evaluation,

potency, and activity related self-sentiments to predict brow warming. Potency ratings statically

significantly predicted brow warming, with lower potency ratings related to greater warming.

Model 3 added deflection to the EPA ratings to predict brow warming. The negative adjusted R2

makes it clear that deflection was not helpful for explaining brow warming. Model 4 combined

Measuring Emotion p. 34

positive emotion with EPA ratings as predictors.11 The coefficient for positive emotion now drops

to marginal statistical significance, the potency effect remains unchanged from model 2, and the

activity approaches significance. The variance explained in the sample climbs to over 17% and

degree-of-freedom adjusted R2 rises to .101. A comparison of the two nested models (2 and 4)

yields F(1,50)=4.46, p<.05, suggesting that there is a significant benefit of self-reported emotion in

predicting brow warming.

The results for eye warming look notably different. Positive emotion alone (model 6)

approaches statistical significance and EPA self-sentiments alone (model 7) do not predict eye

warming at all. Adding deflection to EPA produces a significant improvement in fit, with an

adjusted R2 of .112. Even though deflection is only marginally statistically significant, a

comparison with model 7, F(1,50)=14.05, indicates that adding deflection benefits the model

substantially. The full model fits even better, explaining more than 35% of the variance in the

sample and yielding an adjusted R2 of .202. This model suggests that both deflection and positive

self-sentiments results in eye warming, and there is some indication that positive emotion may also

increase eye warmth.

Lastly, we examined the relationship between these same reports and change in cheek

temperature in models 11-15. These results look similar to the models of brow warming. As with

the brow analyses, deflection does not contribute to cheek warming, producing models with a

negative adjusted R2 when added (see models 13 and 15) and the best fitting model combines EPA

and positive emotion (model 14). Model 14 looks very similar to 4, with slightly stronger fit and

with the positive emotion variable climbing to statistical significance. The cheek analyses have

more cases (N=60 vs N=50), because no cases were dropped due to hats, bangs, or glasses. So the

minor improvements in fit and statistical significance may well be due to differences in power.

11. Even though these measures were not very correlated at the bivariate level, we anticipated that positive emotion might be correlated with EPA ratings at the multivariate level. None of the variance inflation factors in any of the reported reached 2, however, and none of the tolerance levels dipped below .6.

Measuring Emotion p. 35

CONCLUSIONS

Emotions play a key role in cognitive processes, identity processes, group processes,

motivation, and task performance. Unfortunately, efforts to develop and test robust theory on the

social sources and consequences of emotion have been impeded by a shortage of procedures that

reliably and validly measure emotional states. In an attempt to develop new, non-reactive

measures of affect during identity relevant interactions, we used thermographic imaging to

measure the changes in temperature in different regions of the face as indicators of affective

response. While our results foster optimism about the great potential for thermographic imagery as

a non-reactive, non-intrusive measure of emotion, considerable work must be done before the

technology can be used to detect emotions in social situations.

We found brow temperatures and cheek temperatures to respond in highly similar ways,

warming with negative emotion and with reductions in potency. If there was any difference in the

patterns of these two facial regions, it may be that (lack of) potency was more somewhat related to

brow warming and (lack of) positive emotion was somewhat more related to cheek warming.

Future research should attempt to address this question. If these facial regions do, however,

respond in similar ways that could be a happy outcome for emotion researchers. To date, the early

studies of facial thermography have focused on the eye region and above. Most of this research is

conducted in highly constraining laboratory situations, with participants wearing markers on their

face and straps on their heads. As we noted in our findings, the upper part of the face is plagued

with potential interferences (hats, bangs, eyeglasses) when individuals are freed from such

constraints. So, if we want a useful measure that really does not interfere with social interaction,

the cheek would be a much more convenient place to look.

This research provides the first empirical evidence that facial temperature may be able to

discriminate between positive and negative emotion. The facial temperature signature differed

for positive and negative feedback and varied with amount of positive emotion, with lower levels

of positive emotion leading to higher spikes in temperature in the cheek and brow. This already

would make facial thermography more useful than existing physiological measures (e.g., skin

conductance, cortisol) that only register a non-valenced “reaction.”

Measuring Emotion p. 36

The thermal signature of the eye region in response to our experiment was unique from that

of the brow and cheek. Moreover, the theoretical variables of interest (evaluation, potency,

activity, deflection, emotion) explained more than 35% of the variance in eye temperature change

in our study, with eye temperatures warming with positive self-impressions and with deflection.

Recall that Puri et al. (2002) found eye temperature increases in response to frustration and

Pavlidis et al. (2002) found eye temperature increases in response to lying. These could both be

examples of identity deflection. Our results do support the idea of continued attention to the eye

area in thermographic research. Clearly, eye warming should be the focus of future efforts to

develop indirect measures of deflection/discrepancy/stress as it is used in our theories.

In addition to examining the utility of facial thermography for distinguishing

positive/negative emotion and for assessing deflection, future research also should investigate

whether this approach has the potential for distinguishing between discrete emotions. A

physiological approach to distinguishing between discrete emotion has been somewhat of a holy

grail for emotion researchers – and has led to a pretty complex story (Davidson, 1994). It hardly

seems likely that facial temperature changes will provide a complete and simple answer to that

puzzle. Given the foundational role of facial muscles in the experience and expression of

emotion, however, it is entirely possible that facial thermography could go a long way toward

solving this problem.

As promising as this technology appears to be, there will be challenges to developing this

technique for measuring emotion in social interaction research. Many of these challenges are those

faced by researchers using conventional video imaging – accounting for motion, distance, angle.

The temperature calculations that depend on the determinacy of those issues, however, may be

more sensitive to this sort of variation than a human coder making determinations about visible

behaviors. Other sources of interference – hats, eyeglasses, bangs – provide equal difficulty for

those making facial judgments using the human eye. On the more positive side, this research

shows that infrared thermography can “see” facial changes not visible to the human observer.

The technology for infrared facial thermography has already improved considerably

beyond the technology employed in this study. For this research, we used an older model,

Measuring Emotion p. 37

reconditioned infrared camera with much lower levels of temperature sensitivity than are available

today. Likewise, the prices of more advanced infrared cameras are lowering steadily. Further, it

remains to be seen whether the kinds of findings we discovered could be detected on even lower

resolution (non-cooled, less expensive) infrared cameras. Either way, the development of a valid

set of thermal imaging techniques for assessing emotion would offer the opportunity to study

emotions in contexts not previously possible and could potentially contribute to a wealth of

theoretical development in understanding the dynamics of human behavior.

Measuring Emotion p. 38

REFERENCES Allison, P. D. (1990). Change scores as dependent variables in regression analysis. Sociological methodology, 20(1), 93. Anbar, M. (2002). Assessment of Physiologic and Pathologic Radiative Heat Dissipation Using Dynamic Infrared Imaging. Annals of the New York Academy of Sciences, 972(1), 111-118. Aronson, E., Ellsworth, P., Carlsmith, J. M., & Gonzales, M. H. (1990). Methods of Research in Social Psychology: McGraw-Hill. Baker, E. G., Tucker, S., Friedrich, S., & Jann, P. (2010). Gender, Makeup and the Measurement of Emotion During Social Interaction. Paper presented at the Southern Sociological Society, Atlanta, GA. Baumeister, R. F., Tice, D. M., & Hutton, D. G. (1989). Self-Presentational Motivations and Personality Differences in Self-Esteem. Journal of Personality, 57(2), 547-579. Blascovich, J., Loomis, J., Beall, A. C., Swinth, K. R., Hoyt, C. L., & Bailenson, J. N. (2002). TARGET ARTICLE: Immersive Virtual Environment Technology as a Methodological Tool for Social Psychology. Psychological Inquiry, 13(2), 103-124. Blascovich, J., Mendes, W. B., Hunter, S. B., & Salomon, K. (1999). Social 'facilitation' as challenge and threat. Journal of Personality and Social Psychology, 77(1), 68-77. Bradley, M. M., & Lang, P. J. (2000). Affective reactions to acoustic stimuli. Psychophysiology, 37(2), 204-215. Bramson, M. A. (1968). Infrared Radiation: A Handbook for Applications. New York: Plenum Press. Burke, P. J. (1991). Identity Processes and Social Stress. American Sociological Review, 56(6), 836-849. Burke, P. J., & Stets, J. E. (2009). Identity Theory: Oxford University Press. Cacioppo, J. T., Bush, L. K., & Tassinary, L. G. (1992). Microexpressive Facial Actions as a Function of Affective Stimuli: Replication and Extension. Personality and Social Psychology Bulletin, 18(5), 515-526. Cacioppo, J. T., Petty, R. E., Losch, M. E., & Kim, H. S. (1986). Electromyographic activity over facial muscle regions can differentiate the valence and intensity of affective reactions. Journal of Personality and Social Psychology, 50(2), 260-268. Chandola, T., Heraclides, A., & Kumari, M. (2010). Psychophysiological biomarkers of workplace stressors. Neuroscience & Biobehavioral Reviews, 35(1), 51-57. Codispoti, M., Bradley, M. M., & Lang, P. J. (2001). Affective reactions to briefly presented pictures. Psychophysiology, 38(3), 474-478. Cuthbert, B. N., Bradleym, M. M., & Lang, P. J. (1996). Probing picture perception: Activation and emotion. Psychophysiology, 33(2), 103-111.

Measuring Emotion p. 39

Davidson, R.J. (1994). Complexities in the search for emotion-specific physiology. In P. Ekman & R.J. Davidson (Eds.), The Nature of Emotion: Fundamental Questions. Oxford: Oxford University Press. Di Vesta, F. J., & Dick, W. (1966). The Test-Retest Reliability of Children's Ratings on the Semantic Differential. Educational and Psychological Measurement, 26(3), 605-616. Dimberg, U., & Petterson, M. (2000). Facial reactions to happy and angry facial expressions: Evidence for right hemisphere dominance. Psychophysiology, 37(5), 693-696. Ekman, P., & Friesen, W. V. (1975). Unmasking the Face. Englewood Cliffs, NJ: Prentice Hall. Ekman, P., & Friesen, W. V. (1978). The facial action coding system: A technique for measurement of facial movement. Palo Alto, CA.: Consulting Psychologists’ Press. Ekman, P., Friesen, W. V., & Simons, R. C. (1985). Is the startle reaction an emotion? Journal of Personality and Social Psychology, 49(5), 1416-1426. Ellestad, J., & Stets, J. E. (1998). Jealousy and Parenting: Predicting Emotions from Identity Theory. Sociological Perspectives, 41(3), 639-668. Franks, D. (2010). Neurosciology. New York: Springer. Frijda, N. (1986). The Emotions. Cambridge, UK: Cambridge University Press. Gordon, S. L. (1981). The Sociology of Sentiments and Emotion. In M. P. Rosenberg & R. H. Turner (Eds.), Social Psychology: Sociological Perspectives (pp. 562-592). New York: Basic Books. Gregory, S. W., Jr., & Gallagher, T. J. (2002). Spectral Analysis of Candidates' Nonverbal Vocal Communication: Predicting U.S. Presidential Election Outcomes. Social Psychology Quarterly, 65(3), 298-308. Heise, D. R. (1979). Understanding events : affect and the construction of social action. Cambridge ; New York: Cambridge University Press. Heise, D. R. (2007). Expressive Order: Confirming Sentiments in Social Actions. New York: Springer. Heise, D. R. (2011, April 25, 2011). Java INTERACT, from www.indiana.edu/~socpsy/ACT/index.htm Heise, D. R., & Mackinnon, N. J. (1987). Affective bases of likelihood judgments. The Journal of Mathematical Sociology, 13(1-2), 133-151. Hellhammer, D. H., Buchtal, J., Gutberlet, I., & Kirschbaum, C. (1997). Social hierarchy and adrenocortical stress reactivity in men. Psychoneuroendocrinology, 22(8), 643-650. Hellhammer, D. H., Wüst, S., & Kudielka, B. M. (2009). Salivary cortisol as a biomarker in stress research. Psychoneuroendocrinology, 34(2), 163-171.

Measuring Emotion p. 40

Helmreich, R., Spence, J. T., & Stapp, J. (1974). Short forms of the Texas Social Behavior Inventory (TSBI), an objective measure of self-esteem. Bulletin of the Psychonomic Society, 4(5-A), 473-475. Hietanen, J. K., Surakka, V., & Linnankoski, I. (1998). Facial electromyographic responses to vocal affect expressions. Psychophysiology, 35(5), 530-536. Hunsley, J. (1990). Dimensionality of the Multiple Affect Adjective Check List-Revised: A comparison of factor analytic procedures. Journal of Psychopathology and Behavioral Assessment, 12(1), 81-91. Jackson, D. C., Malmstadt, J. R., Larson, C. L., & Davidson, R. J. (2000). Suppression and enhancement of emotional responses to unpleasant pictures. Psychophysiology, 37(4), 515-522. Kalkhoff, W., & Gregory, S. W., Jr. (2008). Beyond the Issues: Nonverbal Vocal Communication, Power Rituals, and “Rope-a-dopes” in the 2008 Presidential Debates. Current Research in Social Psychology, 1(3). Kemper, T. D. (1978). A Social Interactional Theory of Emotions. New York: Wiley. Kemper, T. D. (1990). Social Structure and Testosterone: Explorations of the Socio-Bio-Social Chain. New Brunswick, NJ: Rutgers University Press Kemper, T. D. (1991). Predicting Emotions from Social Relations. Social Psychology Quarterly, 54(4), 330-342. Kemper, T. D. (2011). Status, Power and Ritual Interaction: A Relational Reading of Durkheim, Goffman, and Collins. Burlington, VT: Ashgate Publishing. Kemper, T. D., & Collins, R. (1990). Dimensions of Microinteraction. American Journal of Sociology, 96(1), 32-68. Kong, S. G., Heo, J., Abidi, B. R., Paik, J., & Abidi, M. A. (2005). Recent advances in visual and infrared face recognition—a review. Computer Vision and Image Understanding, 97(1), 103-135. Lang, P. J., Bradley, M. M., & Cuthbert, B. N. (1990). Emotion, attention, and the startle reflex. Psychological Review, 97(3), 377-395. Larsen, J. T., Norris, C. J., & Cacioppo, J. T. (2003). Effects of positive and negative affect on electromyographic activity over zygomaticus major and corrugator supercilii. Psychophysiology, 40(5), 776-785. Lawler, E. J. (2001). An Affect Theory of Social Exchange American Journal of Sociology, 107, 321-352. Lawler, E. J., Thye, S. R., & Yoon, J. (2000). Emotion and Group Cohesion in Productive Exchange. American Journal of Sociology, 106(3), 616-657. Lawler, E. J., & Yoon, J. (1996). Commitment in Exchange Relations: Test of a Theory of Relational Cohesion. American Sociological Review, 61(1), 89-108.

Measuring Emotion p. 41

Lovaglia, M. J., & Houser, J. A. (1996). Emotional Reactions and Status in Groups. American Sociological Review, 61(5), 867-883. Lubin, B., & Zuckerman, M. (1985). Manual for the MAACL-R: Multiple Affect Adjective Check List-revised. Educational and Industrial Testing Service. Lubin, B., Zuckerman, M., Hanson, P. G., Armstrong, T., Rinck, C. M., & Seever, M. (1986). Reliability and validity of the Multiple Affect Adjective Check List - Revised. Journal of Psychopathology and Behavioral Assessment, 8(2), 103-117. Mazur, A., & Booth, A. (1998). Testosterone and dominance in men. Behavioral and Brain Sciences, 21(03), 353-363. Mazur, A., Booth, A., & Dabbs, J. M., Jr. (1992). Testosterone and Chess Competition. Social Psychology Quarterly, 55(1), 70-77. Mize, T. D., & Myers, G. L. (2010). Exploring Racial and Skin Tone Differences on the Measurement and Experience of Emotion. JURO: Journal of Undergraduate Research Opportunities. http://juro.uga.edu/boc2010.html Morgan, S. L., & Winship, C. (2007). Counterfactuals and Causal Inference: Methods and Principles for Social Research. Cambridge, UK: Cambridge University Press. Osgood, C. E. (1962). Studies on the generality of affective meaning systems. The American psychologist, 17(1), 10. Osgood, C. E., May, W. H., & Miron, M. S. (1975). Cross-cultural Universals of Affective Meaning. Urbana IL: University of Illinois Press. Osgood, C. E., Suci, G. J., & Tannenbaum, P. H. (1957). The Measurement of Meaning. Urbana IL: University of Illinois Press. Pavlidis, I., Eberhardt, N. L., & Levine, J. A. (2002). Seeing through the face of deception. Nature, 415(3), 35-35. Pavlidis, I., & Levine, J. (2002). Thermal image analysis for polygraph testing. Engineering in Medicine and Biology Magazine, IEEE, 21(6), 56-64. Puri, C., Olson, L., Pavlidis, I., Levine, J., & Starren, J. (2005). StressCam: non-contact measurement of users' emotional states through thermal imaging. Paper presented at the CHI '05 extended abstracts on Human factors in computing systems, Portland, OR, USA. Riley, A., & Burke, P. J. (1995). Identities and Self-Verification in the Small Group. Social Psychology Quarterly, 58(2), 61-73. Robinson, D. T., & Smith-Lovin, L. (1992). Selective Interaction as a Strategy for Identity Maintenance: An Affect Control Model. Social Psychology Quarterly, 55(1), 12-28. Robinson, D. T., Smith-Lovin, L., & Rogalin, C. (2004). Physiological Measures of Theoretical Concepts: Some Ideas for Linking Deflection and Emotion to Physical Responses During Interaction. Advances in Group Processes, 21, 77-115.

Measuring Emotion p. 42

Sapolsky, R. M. (2002). Endocrinology of the stress-response. In J. B. Becker, S. M. Breedlove, D. Crews & M. M. McCarthy (Eds.), Behavioral Endocrinology (pp. 409–450). Cambridge, MA: MIT Press. Schröder, T., & Scholl, W. (2009). Affective Dynamics of Leadership: An Experimental Test of Affect Control Theory. Social Psychology Quarterly, 72(2), 180-197. Skrandies, W. (1998). Evoked potential correlates of semantic meaning—A brain mapping study. Cognitive Brain Research, 6(3), 173-183. Skrandies, W., & Chiu, M. J. (2003). Dimensions of affective semantic meaning — behavioral and evoked potential correlates in Chinese subjects. Neuroscience Letters, 341(1), 45-48. Skrandies, W., Chiu, M.-J., & Lin, Y. (2004). The Processing of Semantic Meaning in Chinese Words and Evoked Brain Topography. Brain Topography, 16(4), 255-259. Smith-Lovin, L. (1995). Sociology of Affect and Emotion. In K. S. Cook, G. A. Fine & J. S. House (Eds.), Sociological Perspectives on Social Psychology (pp. 118-148): Allyn & Bacon. Smith-Lovin, L., & Heise, D. R. (Eds.). (1988). Analyzing Social Events: Advances in Affect Control Theory. New York: Gordon and Breach Scientific Publishers. Smith-Lovin, L., & Robinson, D. T. (1992). Gender and Conversational Dynamics. In C. Ridgeway (Ed.), Gender, Interaction, and Inequality (pp. 122-156). New York: Springer-Verlag. Socolinsky, D. A., Selinger, A., & Neuheisel, J. D. (2003). Face recognition with visible and thermal infrared imagery. Computer Vision and Image Understanding, 91(1–2), 72-114. Steptoe, A., & Sawada, Y. (1989). Assessment of Baroreceptor Reflex Function During Mental Stress and Relaxation. Psychophysiology, 26(2), 140-147. Stets, J. E. (2003). Justice, Emotion, and Identity Theory. In P. J. Burke, T. J. Owens, R. T. Serpe & P. A. Thoits (Eds.), Advances in Identity Theory and Research (pp. 105-122). New York: Klewer Academic Publishing. Stoolmiller, M., & Bank, L. (1995). Autoregressive Effects in Structural Equation Models. In J. M. Gottman (Ed.), The Analysis of Change (pp. 263-276). Mahwah, NJ: Erlbaum. Swann, W. B., Griffin, J. J., Predmore, S. C., & Gaines, B. (1987). The cognitive–affective crossfire: When self-consistency confronts self-enhancement. Journal of Personality and Social Psychology, 52(5), 881-889. Thoits, P. A. (1989). The Sociology of Emotions. Annual Review of Sociology, 15, 317-342. Troyer, L. (2008). A Strategy for Incorporating Social Intelligence in the Design of Virtual Environments. In J. Clay-Warner & D. T. Robinson (Eds.), Social Structure and Emotion (pp. 95-106). San Diego, CA: Elsevier. Turner, J. H. (2000). On the Origins of Human Emotions. Stanford, CA: Stanford University Press.

Measuring Emotion p. 43

Turner, J. H., & Stets, J. E. (2005). The Sociology of Emotions. New York: Cambridge University Press. Vrana, S. R., & Rollock, D. (2002). The role of ethnicity, gender, emotional content, and contextual differences in physiological, expressive, and self-reported emotional responses to imagery. Cognition & Emotion, 16(1), 165-192. Wexler, B. E., Warrenburg, S., Schwartz, G. E., & Janer, L. D. (1992). EEG and EMG responses to emotion-evoking stimuli processed without conscious awareness. Neuropsychologia, 30(12), 1065-1079. Wiggins, B., & Heise, D. R. (1987). Expectations, Intentions, and Behavior: Some Tests of Affect Control Theory. The Journal of Mathematical Sociology, 13(1-2), 153-169. Wolfe, W. L., & Zissis, G. J. (1978). The Infrared Handbook. Washington, D.C.: Office of Naval Research, Department of Navy. Zuckerman, M., & Lubin, B. (1985). Manual for the MAACL-R: The Multiple Affect Adjective Check List Revised. San Diego, CA: EdITS/Educational and Industrial Testing Service.

Measuring Emotion p. 44

Figure 1. Detecting Thermal Radiation from the Face

Measuring Emotion p. 45

Figure 2. Useful and Problematic Thermal Images

Measuring Emotion p. 46

Figure 3. Mouth Temperature for Individuals by Feedback Condition

Measuring Emotion p. 47

Figure 4. Mouth Temperature Change by Feedback Condition

Measuring Emotion p. 48

Figure 5. Mouth Temperature by Feedback Condition over Experiment

Measuring Emotion p. 49

Figure 6. Temperature Across Experiment by Facial Region and Feedback Condition

Measuring Emotion p. 50

Table 1.Unstandardized Coefficients for Effects of Affective Response on Temperature Change Across Three Facial Regions.

Brow Warming Eye Warming Cheek Warming

model 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

constant .340 (.048)

.436 (.208)

.382 (.377)

.364 (.248)

.513 (.398)

-.010 (.064)

.266 (.311)

-.245 (.255)

.307 (.363)

-.248 (.250)

.356 (.060)

.740 (.290)

.281 (.464)

.583 (.293)

.246 (.441)

Pos Affect -.010* (.004)

-.010^

(.006) -.017^

(.010) .009^

(.006) .008

(.008) .011^

(.006) -.013* (.005)

-.015* (.007)

-.005 (.011)

Evaluation .001 (.006)

.006 (.010)

.007 (.007)

.010 (.010)

-.005 (.009)

.016* (.008)

-.008 (.010)

.015* (.007)

-.005 (.008)

.004 (.012)

.014^

(.008) .008 (.012)

Potency -.017* (006)

-.018 (.013)

-.020* (.009)

-.021 (.014)

-.003 (.008)

-.004 (.009)

.000 (.012)

-.006 (.008)

-.014^

(.008) -.004 (.016)

-.028** (.009)

-.007 (.015)

Activity .009^

(.005) .005 (.008)

.012^

(.006) .007 (.009)

.003 (.007)

-.005 (.007)

-.001 (.009)

-.006 (.006)

-.001 (.006)

-.008 (.010)

.004 (.007)

-.003 (.010)

Deflection .014 (.024)

-.004 (.026)

.034^ (.018)

.040* (.018)

.047 (.030)

.027 (.029)

R2 070 091 078 173 175 031 008 244 027 355 056 044 104 205 075

Adj R2 060 059 - 054 101 010 020 - 029 112 - 057 202 047 017 - 001 152 - 074 ^ p<.10; * p<.05; ** p<. Standard Errors in parentheses

Measuring Emotion p. 51