Marinier_thesisproposal- Unified Theory of Emotions

7/27/2019 Marinier_thesisproposal- Unified Theory of Emotions

http://slidepdf.com/reader/full/marinierthesisproposal-unified-theory-of-emotions 1/50

Towards a Unified Theory of Emotion and Cognition

Bob MarinierUniversity of Michigan

Winter 2006Advisor: John Laird



2

1 INTRODUCTION..............................................................................................................................................3

1.1 WHY COGNITIVE ARCHITECTURES NEED EMOTION ......................................................................................3 1.2 WHY EMOTION NEEDS COGNITIVE ARCHITECTURES ....................................................................................3 1.3 WHAT GOOD ARE EMOTIONS? (HOW EMOTIONS MAY BENEFIT AI) ..............................................................3

1.4 WHAT IS EMOTION?.....................................................................................................................................4 1.5 OVERVIEW ..................................................................................................................................................5

2 CLASSIC COGNITIVE ARCHITECTURES ................................................................................................6

2.1 OPERATORS.................................................................................................................................................6 2.2 MEMORIES ..................................................................................................................................................6 2.3 MECHANISMS ..............................................................................................................................................6

3 THE EMOTION PROCESS: A FRAMEWORK ...........................................................................................8

3.1 APPRAISAL THEORY ....................................................................................................................................8 3.2 EMOTION-INDUCED CHANGES ...................................................................................................................11 3.3 RESPONSES TO EMOTION ...........................................................................................................................12

3.3.1 Coping .................................................................................................................................................12

3.3.2 Emotion regulation ......... ........... .......... ........... .......... ........... .......... ........... ........... .......... ........... .......... .13 3.4 PERSONALITY AND INDIVIDUAL DIFFERENCES...........................................................................................15

4 UNDERSTANDING THE DESIGN CHOICES ...........................................................................................17

4.1 APPRAISAL THEORY DESIGN CHOICES........................................................................................................17 4.1.1 How are the appraisal values generated? ......... ........... .......... ........... .......... ........... ........... .......... ........17

4.1.1.1 Comprehension and the decision-making process .....................................................................................17 4.1.1.2 Properties of the comprehension system.................................................................................................... 18 4.1.1.3 Building blocks of comprehension.............................................................................................................20 4.1.1.4 The comprehension process .......................................................................................................................21 4.1.1.5 Properties revisited.....................................................................................................................................24 4.1.1.6 Predictions .................................................................................................................................................25

4.1.2 What are the proper dimensions?........................................................................................................26 4.1.3 Are the emotions categories or modal spaces?............... .......... ........... .......... ........... .......... ........... ......26

4.1.4 Summary ..............................................................................................................................................27 4.2 POST-APPRAISAL CHANGES DESIGN CHOICES.............................................................................................27 4.2.1 How long does an emotion last and how is its intensity calculated? .......... .......... ........... .......... ..........27 4.2.2 What does an agent feel?.....................................................................................................................28 4.2.3 What cognitive changes should be modeled? ......................................................................................29

4.2.3.1 Mood state dependent retrieval and mood congruent retrieval...................................................................29 4.2.3.2 Categorization effects ................................................................................................................................ 30 4.2.3.3 Broaden and build ...................................................................................................................................... 31 4.2.3.4 Episodic memory ....................................................................................................................................... 31 4.2.3.5 Undoing .....................................................................................................................................................32 4.2.3.6 Priming effects...........................................................................................................................................32 4.2.3.7 Judgment effects ........................................................................................................................................ 33 4.2.3.8 Higher-level phenomena ............................................................................................................................ 33 4.2.3.9 Summarizing cognitive changes ................................................................................................................34

4.2.4 How do action and thought urges fit in?.............. ........... .......... ........... .......... ........... .......... ........... ......36 4.2.5 Summary ..............................................................................................................................................37 4.3 RESPONSES TO EMOTION DESIGN CHOICES ................................................................................................37

4.3.1 How does an agent respond to its feelings?............. .......... ........... .......... ........... .......... ........... .......... ...38 4.3.2 What responses should be incorporated? .......... ........... .......... ........... .......... ........... ........... .......... ........39 4.3.3 Summary ..............................................................................................................................................40

5 EXPERIMENTAL APPROACH....................................................................................................................41

5.1 PRIOR EVALUATIONS .................................................................................................................................41 5.2 DESIGNING NEW EVALUATIONS .................................................................................................................42



3

5.2.1 Accelerating reinforcement learning .......... ........... .......... ........... .......... ........... ........... .......... ........... ....42 5.2.2 Improving metacognitive strategies ........... .......... ........... .......... ........... .......... ........... .......... ........... ......43

5.3 CHECKING PREDICTIONS ...........................................................................................................................44 5.4 SUMMARY .................................................................................................................................................44

6 PLAN.................................................................................................................................................................46

7

REFERENCES.................................................................................................................................................48

1 Introduction

1.1 Why cognitive architectures need emotion

Cognitive architectures promise to one day explain how human cognition arises from a set of basic architectural mechanisms. Informally, we all know that emotion plays a major role in ourlives, often influencing our behavior and our thoughts. More formally, psychologists havedocumented several phenomena that show such changes. If cognitive architectures ever hope toexplain all of human behavior, then they need to include a theory of how emotion integrates withthe architecture. Furthermore, cognitive architectures must support the processes that lead to

emotion in the first place.

1.2 Why emotion needs cognitive architectures

Like much of psychology outside of cognitive architectures, emotion psychologists successfullydocument various phenomena but then generate theories that are either too abstract or which failto integrate with phenomena from other areas of psychology. In other words, emotionpsychologists are not exempt from Newell’s (1990) criticisms of psychological research in hiscall for a focus on unified theories of cognition. Current attempts to create process models of emotion essentially start from scratch instead of building on existing, tested architectures (c.f.Smith & Kirby 2001). By building on existing, independently-motivated architectures, emotiontheories will be subject to a much broader set of constraints, narrowing the possible set of

plausible theories and forcing greater detail. The increase in detail may even allow for morequantitative predictions, such as the timing of emotional events.

1.3 What good are emotions? (how emotions may benefit AI)

Old philosophy and modern “common sense” dictates that emotions are a distraction fromlogical decision making and, in short, we would be better off without them. Certainly there aretimes when our judgment seems clouded by our emotions, resulting in counterproductivebehavior. However, case studies involving individuals who have lost the ability to experienceemotions due to brain damage indicate that emotions have positive roles to play as well. Theseindividuals have no difficulty in most other areas, including speech, motor, and memory.However, they seem to suffer from an inability to plan, learn, and understand the consequences

of their actions (Damasio 1994). In other words, they are not capable of leading “normal” lives.The implication is that emotion plays an important functional role in generating good behavior.

Furthermore, from an evolutionary perspective, emotions help decouple sensory input frommotor responses by allowing an agent to classify the world relative to its goals and then respondto that more general classification instead of the specific details of the situation (Smith &Lazarus 1990). That is, emotions are a mechanism that is able to identify the aspects of thesituation that are most important for survival and learning – emotions are evolution’s lessons



4

about what is important. Conversely, emotions can also result in useful automatic responses,including motivation, attention shifting, and physiological preparation for action.

Finally, emotion may play an important role in metacognition (Flavell 1979, 1987). Simply put,metacognition is “thinking about thinking.” For my purposes, metacognition is the knowledge,

mechanisms and strategies used to regulate cognition (e.g. select appropriate cognitive strategies)in order to generate intelligent behavior. This includes planning, learning, and goal management(e.g. monitoring progress towards goals and even giving up when appropriate).

One aspect of metacognition is called metacognitive experience (Flavell 1979, 1987).Metacognitive experience is defined as “experiences that are cognitive and affective” (Flavell1987). As we will see, emotions (or more accurately, feelings) fit this definition of metacognitive experience. Emotions effectively summarize the relationship between a situationand one’s goals (see section 3.1), which is critical for assessing the progress one is makingtowards those goals. Furthermore, when someone copes with his emotions, he is changing hiscognitive activities in response to this evaluation. Thus, some forms of coping are examples of

metacognitive strategies (i.e. strategies used to change one’s cognitive processing; see section3.3).

From an AI perspective, if we can identify how emotions help humans, then we may be able todesign computational systems which are also able to benefit from similar capabilities. It is evenpossible that we will discover that some existing computational systems already have featureswhich we would identify as a kind of emotion.

1.4 What is emotion?

One might expect that a thesis on emotion will concretely define emotion. Unfortunately, asdiscussed by Smith & Lazarus (1990), concrete definitions of emotion are not possible.

Emotions seem to include many related phenomena, such that there are many edge cases that donot fit most definitions, but also are not easily dismissed as unemotional in nature. That said, Iwill later identify what is, for all intents and purposes, emotion in my system. However, byassigning the label “emotion” to this part, I am not trying to make any strong theoretical claim;rather, I am merely making it easier to discuss the system and its components.

In general, there are at least three ways we can think of emotion.1) Emotion as a state. This is probably the most common popular notion of emotion. For

example, a person is angry means a person is in the angry state.2) Emotion as a process. Many emotion theorists talk about emotion as a complex dynamic

interaction between cognition, physiology, and social elements. It doesn’t make much

sense to “freeze” this dynamic system and try to label it, because the “state” is constantlychanging. Thus, emotion is really more about the changing interactions and less aboutany particular moment in time.

3) Emotion as an (indirect) knowledge source. From a cognitive architecture’s perspective,emotion provides additional information that can be used to make decisions and aidlearning. It is indirect because, unlike many other sources of knowledge, the agent doesnot query it for information. Rather, some information is forced upon the agent. But (asdiscussed in the paper), this information is not necessarily a direct reflection of the



5

agent’s emotions. Furthermore, emotion can impact decision making “behind the scenes”by manipulating how the architecture works.

1.5 Overview

A long-term research program would investigate the integration of physiological, cognitive and

sociocultural aspects of emotion and how they impact behavior. Given the need for a narrowerfocus and my experience with Soar, a cognitive architecture, I will not investigate thephysiological or sociocultural aspects of emotion for this thesis. On the cognitive side, asdiscussed in section 4.2.3, it turns out that most of the “knobs” necessary to integrate emotionalchanges at the architectural level do not yet exist (although many are under active developmentby others). Thus, my research will focus primarily on the cognitive antecedents of emotion(section 4.1) and the cognitive responses to emotion (section 4.3). This will take the form of adomain independent comprehension system that allows an agent to understand the situation withrespect to its goals (the basis of the appraisal theory of emotion, section 3.1).

In general, evaluation of emotion systems is very difficult (see section 5.1). However, by

providing emotional state to the reinforcement learning system (Nason & Laird 2004), I shouldbe able to demonstrate that an emotional agent can learn faster than a non-emotional agent(section 5.2.1). I may also be able to demonstrate improved support for metacognitive strategiessuch as goal management via the comprehension system that supports the emotion system(section 5.2.2). In conclusion, my research will create a general comprehension system thatsupports the induction of emotion and then utilizes the resulting feelings to help choose actions.Furthermore, I will answer the question, does emotion (as realized in this system) help improvelearning and task performance? This research will also provide a concrete framework for futureresearch involving cognitive mechanism integration, physiological integration, and socioculturalintegration.

In the remainder of this paper, Section 2 will describe what is meant by “cognitive architecture.”Section 3 will describe the framework for my emotion research. Section 4 will describe thekinds of choices that must be made in each part of the framework, including the ones I intend toexplore. Section 5 will discuss a possible experimental approach. Section 6 will summarize theintended research program.



6

2 Classic cognitive architecturesIn this section I will describe some general cognitive architecture components in order to providecontext for my research. While I have Soar in mind when I describe these components, thesedescriptions should apply to many cognitive architectures. How emotion integrates with this

classical architecture will be discussed throughout the rest of the paper.

2.1 Operators

Behavior is generated by repeated selection and application of operators. Operators are theprimitive “actions” that an agent can take in its thinking process. Some operators may merely beinternal, “mental” actions; others may be realized as motor commands that affect the world. Anoperator is composed of two parts: proposals and applications. A proposal describes theconditions under which the operator may be chosen. An operator may have many proposals,allowing for disjunction. An application describes what to do once an operator has been chosen.An operator may have many applications, each doing different things and applying underdifferent conditions. Operator selection is mediated by knowledge that assigns a numeric

preference to each operator. The numeric preference represents how much that operator is“worth” under those conditions – that is, how much future reward the agent expects to get bychoosing that operator.

2.2 Memories

There are two broad categories of memories – long-term memories and short-term memories.

There are at least three kinds of long-term memories: procedural, episodic, and semantic.Procedural memory contains knowledge about what the available operators are, when to performthem, and how to perform them, as well as simple entailments about the situation. It is fast,requires an exact match and allows for parallel retrievals. Episodic memory contains recordingsof previous situations that an agent has been in. It is probably slower, allows for partial matches,and only allows for serial retrievals. Semantic memory contains facts that the agent knows, butwhich are not tied to a particular previous situation. It is also slow, allows for partial matches,and only allows for serial retrievals. During episodic and semantic retrievals, what gets retrievedis determined by the “goodness” of the match, which may be biased by things like activation (of both cue and memory), noise, and thresholds. All of these long-term memories are sources of

knowledge that the agent can use to help choose and apply operators.

Short-term memory is the place where knowledge from the various long-term memories andother inputs, such as perception, are brought together so they can be reasoned about together forthe purpose of choosing an operator. For my purposes, I will not differentiate between thedifferent kinds of short-term memories.

2.3 Mechanisms

A cognitive architecture contains several mechanisms that allow the various memories to work together and provide a means for choosing and applying operators. In addition to the retrievalmechanisms described in section 2.2, there are learning mechanisms that create new long-termmemories. Those will not be important for this research.



7

At any particular time, there may be multiple operators that can apply in a given situation. Inthis case, the decision procedure takes the available knowledge into account in order to choosethe best one. Sometimes it may choose one of the operators randomly, weighted by its numericpreference. In that case, the agent may get some reward feedback which allows it to adjust the

numeric preferences of the operators via reinforcement learning to improve their accuracy.

Figure 1 shows how these various parts fit together.

Short-

term

Memory

Long-term Memories

P r o c

e d u r

a l

Episodic

S e m a n t i c

Decision Procedure

B o d y

B o d y

P r o p o s a l &

S e l e c t i o n

R e i n f o r c e m

e n

t

L e a r n i n g E n c

o d i n

g &

R e t r i e

v a l

Perception & Motor

Figure 1: A basic cognitive architecture. The ellipse shows central cognition as separate from but still part of

the body.



8

3 The emotion process: A frameworkI will begin by giving an overview of the emotion process, which will provide the framework formy research. I will then go over the design options for each part of the process.

The emotion process begins with the agent-environment transaction. This is more than just theenvironmental situation the agent happens to be in at the moment – it is the relationship betweenthe situation and the agent’s goals (Lazarus 1991). It is the understanding of this relationshipthat leads to the emotion state. Emotional state, in turn, may cause (or be defined by) changes ina variety of systems – facial changes, physiological changes, cognitive changes, action andmotivational urges, and, importantly, the actual perceived feeling. Upon noticing some of thesechanges, the agent may then deliberately respond to them, which may lead to changes in variouspoints in the process, making a cycle. Figure 2 shows the basic emotion process without anycycles (I will develop the diagram as I introduce more concepts).

Figure 2: Basic emotion process

I will describe each step of the process including some of the alternative approaches that mightbe taken.

3.1 Appraisal theory Appraisal theory describes how an agent generates an emotion from the agent-environmenttransaction. Appraisal theory postulates that an agent evaluates the transaction along variousdimensions which help categorize the transaction. Appraisal can be thought of as ametacognitive strategy – that is, an agent’s way of monitoring its progress towards its goals.There are many different appraisal theories (see Roseman & Smith 2001 for an overview), butcommonly proposed dimensions include things like novelty, goal relevance, goal conduciveness,causality, and coping potential. A more complete example is shown in Table 1 (Scherer 2001).



9

Appraisal Objective Appraisal Dimension Main ideas

Novelty Suddenness, familiarity, predictability.

Intrinsic pleasantnessPleasantness of object itself, independent of current goals. May be acquired.Relevance Detection

Goal relevance Does stimulus result in outcomes that impactmajor goals?

Causal attributionWho or what is responsible? What were theintentions?

Outcome probability How likely are certain consequences?

Discrepancy fromexpectation

Degree can be determined by the number of features that fit the original expectation.

Goal/needconduciveness

How conducive (or not) is the event to helpingme attain my goals?

Implication Assessment

UrgencyAre high-priority goals endangered? Willwaiting make things worse?

Control To what extent can an event be controlled bynatural agents?

PowerTo what extent can an event be controlled byme (directly or via influence on others)?Coping Potential

DeterminationAdjustment

If I fail to change the event, to what extent canI live with the consequences?

Internal standardsTo what extent is my behavior in line with myself ideal and moral code?

Normative SignificanceEvaluation

External standardsTo what extent is my behavior in line withsocial norms?

Table 1: An example set of appraisal dimensions (from Scherer 2001).

Appraisal theory also specifies a mapping from appraisal values to emotional states. Forexample, a theory might say that if transaction is goal relevant, non-conducive, caused bysomeone else, and the agent has to power to do something about it, then anger may result. If theagent is powerless to do anything, then fear may result. A more complete example is shown inTable 2 (Scherer 2001). The resulting emotion is an important metacognitive experience thatsummarizes how things are going for the agent. This information can be used by other processes(e.g. coping; see section 3.3) to help generate intelligent behavior.



10

Enjoyment/ Happiness

Elation/ Joy

Displeasure/ Disgust

Contempt/ Scorn

Sadness/ Dejection

DespairAnxiety/ Worry

Relevance

Novelty

Suddenness lowhigh/

mediumlow high low

Familiarity low low very low

Predictability medium low low low

Intrinsic pleasantness high very low

Goal/need relevance medium high low low high high medium

Implication

Cause: agent otherother/ nature

other/ nature

Cause: motive intentionalchance/ intentional

intentionalchance/ negligence

chance/ negligence

Outcome probability very high very high very high high very high very high medium

Discrepancy fromexpectation

consonant dissonant

Conduciveness high very high obstruct obstruct obstructUrgency very low low medium low low high medium

Coping potential

Control high very low very low

Power low very low very low low

Adjustment high medium high medium very low medium

Normative

significance

Internal standardscompatibility

very low

External standardscompatibility

very low

Table 2: An example of a partial mapping from appraisals to emotions (from Scherer 2001). Blank entries

indicate appraisals that play either no role or an indeterminate role for that emotion.



11

Agent-Environment

Transaction

Emotion-Induced

Changes

Responses to

Emotion

Appraisal E m o t i o n

Figure 3: Appraisal mediates between the agent-environment transaction and the various changes in the

emotion-induced changes.

Figure 3 shows how appraisal and emotion fit into the appraisal process.

3.2 Emotion-induced changes

Once the transaction has been appraised, the agent’s architecture may automatically respond in anumber of ways. In humans, there are often physiological changes (e.g. heart rate, bloodpressure, skin conductance, etc), facial expression changes, vocal changes, and so on (Cacioppoet al 2000; Keltner & Ekman 2000; Bachorowski 1999)

1. More interesting to me are the

cognitive changes (e.g. possible changes in operator selection, learning, memory retrieval, etc),action and thought urges, and subjective feelings. Damasio (2003) defines feeling as “theperception of a certain state of the body along with the perception of a certain mode of thinkingand thoughts with certain themes.” Since feelings are based in the body, they may have non-emotional sources, but emotions always result in feelings. Indeed, Damasio primarily defines

emotions by their physiological impact. Since I am not exploring physiology, I will use asimpler definition: feelings are an agent’s perception of its emotional state.

1 It may be that some changes are not mediated by emotion and are actually linked to specific appraisals (Smith &Kirby 2001). These kinds of changes provide insight into how cognition and the body are linked, but are beyond thescope of this research.



12

Figure 4 shows the emotion process with the emotion-induced changes I will discuss in thispaper.

Agent-Environment

Transaction

Emotion-Induced

Changes

Responses to

Emotion


Thought-action urges

Cognitive ∆

Subjective feelings

Figure 4: Some of the changes caused by emotion include thought-action urges, cognitive changes, and

subjective feelings.

3.3 Responses to emotion

The emotional state, for the purposes of this research, will be some point in the n-dimensionalappraisal space generated by the appraisal process. Regions within this n-dimensional space maycorrespond to what people commonly think of as emotion. That is, cultural knowledge allows usto classify many different emotion points as “anger,” “fear” or “happiness.”

There are at least two overlapping ways to discuss responses to emotion: coping and emotionregulation.

3.3.1 Coping

Coping is what an agent does in order to improve or maintain his feelings. An agent can notreally deal with his emotions directly since all it knows about them is what it can perceive (i.e.his feelings). There are two classes of coping: problem-focused and emotion-focused.

Problem-focused coping is taking actions in the world to change the environment part of theagent-environment transaction. For example, if something in the environment is causing fear,the agent might run away so it’s no longer being exposed to that stimulus. If the agent is angry,



13

it might take actions that fix whatever the problem is. If the agent is happy, it might take actionsthat protect the source of its happiness.

Emotion-focused coping is taking internal actions to change the agent part of the agent-environment transaction. For example, if an agent determines its goals are unachievable, then it

might give up on them, thus relieving the negative emotions caused by its repeated failures.Emotion-focused coping is a metacognitive strategy because the agent is changing the way itthinks in order to help it make progress in the world (even if that progress is now towards newgoals). Gratch & Marsella (2004) list several types of coping which I have reproduced in Table3.

Coping in general provides a way to manage one’s goals, be it forming new subgoals to get back on track (problem-focused coping) or giving up on some goals (emotion-focused coping).

Active coping: taking active steps to try to remove or circumvent the stressor.

Planning: thinking about how to cope. Coming up with action strategies.Problem-focused

Coping Seeking social support for instrumental reasons: seeking advice, assistance, orinformation.

Suppression of competing activities: put other projects aside or let them slide.

Restraint coping: waiting till the appropriate opportunity. Holding back.

Seeking social support for emotional reasons: getting moral support, sympathy, orunderstanding.

Positive reinterpretation & growth: look for silver lining; try to grow as a person as aresult.

Acceptance: accept stressor as real. Learn to live with it.

Turning to religion: pray, put trust in god (assume God has a plan).

Focus on and vent: can be function to accommodate loss and move forward.

Denial: denying the reality of event.Behavioral disengagement: Admit I cannot deal. Reduce effort.

Mental disengagement: Use other activities to take mind off problem: daydreaming,sleeping.

Emotion-focusedCoping

Alcohol/drug disengagement.Table 3: Examples of coping strategies (from Gratch & Marsella 2004).

3.3.2 Emotion regulation

Emotion regulation is in some sense more far-reaching than coping. Whereas coping is typicallythought of as a response to emotion, emotion regulation also covers manipulation of predictedemotions. Emotion regulation can be divided into two areas: antecedent-focused and response-

focused.

Antecedent-focused emotion regulation is choosing or avoiding environments that promotecertain emotions. Some possible types include situation selection (choosing or avoiding asituation entirely), situation modification (changing a situation) and attentional deployment(choosing to attend to or ignore certain aspects of a situation). In general, then, the agent hassome prediction of what an emotion might be in some situation, and takes steps to avoid it orreinforce it.



14

Response-focused emotion regulation encompasses response modulation, which is activelytrying to change your post-appraisal responses (e.g. change one’s facial expression, suppressone’s anger). Since response-focused emotion regulation seems to require the ability tomanipulate physiology (which I am not including) or lower-level cognitive functions (most of

which are not available to me; see section 4.2.3), I do not intend to include response-focusedemotion regulation in my research.

Clearly there is a large overlap between coping and emotion regulation. In fact, one can arguethat antecedent-focused emotion regulation and coping are pretty much the same thing. After all,coping can only be successful if there is some (possibly implicit) prediction that the coping effortwill somehow improve or maintain the agent’s emotional state. The antecedent-responsedifference can be resolved by noting that the emotion process is a cycle, and thus what can beinterpreted as a response to one emotion may also be interpreted as a prediction of the nextemotion. Thus, the primary difference between coping and antecedent-focused emotionregulation is the set of dimensions used to describe them. It may be possible to transform the

antecedent-focused strategies into coping strategies. For example, suppose a woman decides notto go to a particular bar because she is afraid that her ex-boyfriend might be there, which couldlead to an angry confrontation. The antecedent-focused view would describe her decision as anattempt to prevent anger. The coping view would describe her decision as a response to her fear.For my research, I will probably focus on responding to emotions and not consider explicitemotion anticipation. That is, I intend to follow the coping model more than the emotionregulation model.

Figure 5 shows how responses to emotion can impact the rest of the emotion process,transforming it into a cycle.



15

Agent-Environment

Transaction


Cognitive ∆

Subjective feelings

Responses to

Emotion


C o p i n g

R e g u l

a t i o n

Figure 5: Responses to emotion include changing the agent-environment transaction and attempting to

directly manipulate the emotion-induced changes.

Table 4 summarizes the parts of the emotion process that I will explore. Section 4 contains a

more detailed explanation of what I have explored and what I will explore.

Emotion process component Comments

Agent-environmenttransaction

This is part of the research in the sense that it needs to berepresented in order to be appraised.

Appraisal I will explore what appraisals to use and how to generate them.

Emotion I will explore which emotions to use and how to represent them.

Post-appraisal changes I will explore those elements listed in the emotion processfigures. I will not explore physiology.

Responses to emotion I will explore this from the perspective of coping, notregulation.

Table 4: Summary of what I will explore about the emotion process.

3.4 Personality and individual differences

Before closing this section, I want to briefly mention personality and individual differences. I donot intend to explore the effects of personality, but it may help to have an understanding of howit fits into the framework.



16

Personality is a property that can impact each stage of the emotion process. For example, it mayimpact the appraisals that are generated, and the coping strategies used. There are at least twomajor sources for personality differences – differences in knowledge, and differences inarchitectural parameters.

Differences in knowledge can come from a variety of sources. A culture can influence whatdifferent genders and groups consider appropriate behavior, leading to different emotion profiles.For example, women are more likely to express sadness whereas men are more likely to expressanger (Citrin et al. 2005). Furthermore, Gross & John (2003) show that minorities in the U.S. aremore likely to engage in suppression than European Americans.

Differences in architectural parameters may come in the form of subtle differences in workingmemory capacity or long-term memory retrieval times or noise. They can also take the form of physiological differences – for example, there may be differences in heart rate and skinconductance changes. Berenbaum (2002), for example, identified links between personalitytraits and pleasure reactions.

Finally, Feldman Barrett & Gross (2001) describe a concept called emotional intelligence whichbrings some of these differences together. Emotional intelligence is how good people are atperceiving their current emotions and effectively regulating them. While they stop short of providing a quantitative measure like IQ, it is safe to say that people differ in their emotionalintelligence, which in turn impacts their emotional states over time.



17

4 Understanding the design choicesIn the previous section I introduced the emotion process framework in which I intend to conductresearch. However, there are many decisions that must be made about each part of theframework in order to arrive at an actual implementation. In this section I will discuss design

decisions related to appraisal theory, what happens after the appraisals have been generated, anddesign considerations for how agents can respond to emotions.

In general, my design philosophy is to avoid placing the research of others on my research path.Thus, while there are many interesting ideas related to how emotion integrates with variouscognitive mechanisms, as we will see in section 4.2.3, the relevant mechanisms do not yet existin Soar, and while many of them are under development, I will not wait for them to finish beforeI can conduct my research. Because of this, much of my work will be in generating andresponding to appraisals, and not in architectural integration. Those areas of generation andresponse which seem to rely on these unimplemented mechanisms will have to be finessed (forexample, even though semantic memory does not exist, it is possible to “fake” many aspects of it

using procedural memory).

4.1 Appraisal theory design choices

There are several choices that must be made in determining how to properly integrate appraisaltheory in the emotion process:

1) How are the appraisal values generated?2) What are the proper dimensions?3) Are the emotions categories or modal spaces?

4.1.1 How are the appraisal values generated?

One of the key issues in appraisal theory is why and how appraisal values are generated. Ihypothesize that appraisals are supported by comprehension; furthermore, many appraisals arerequired by the comprehension process, and thus “fall out” of it.

In order for an agent to do something about the agent-environment transaction, it mustunderstand the transaction. Stated another way, we can ask, “How does the agent comprehendwhat is going on and how it relates to its goals?” This question of comprehension is perhaps thekey unaddressed issue in appraisal theory. Existing theories assume that humans can do this, anddeflect any issues to other areas in psychology or artificial intelligence (e.g. Roseman (2001)refers to causality research and Gratch & Marsella (2004) rely on decision-theoretic planning). Icannot hope to solve this problem completely, but I will attempt to address it adequately enoughto support the rest of my research. First, I will discuss where comprehension fits into the overall

decision process, the properties of comprehension, and the concepts necessary to understand myapproach. Then, I will discuss the actual design of the comprehension system.

4.1.1.1 Comprehension and the decision-making process

Newell (1990) discussed comprehension in the context of the PEACTIDM cycle (perceive,encode, attend, comprehend, tasking, intend, decode, motor):

1) Perceive: Raw perceptual inputs



18

2) Encode: Interpreting the raw inputs in a domain-independent way.3) Attend: Attending to an input.4) Comprehend: Understanding the input in the context of what the agent is doing.5) Tasking: Performing maintenance on one’s goals.6) Intend: Choosing a course of action.

7)

Decode: Translating the action choice into motor commands.8) Motor: Executing motor commands.

Perception and motor are considered to be outside of central cognition, and encode and decodeare on the boundary. This sequence is mostly fixed because of the constraints between the steps(e.g. one can’t comprehend without first attending, etc). There are some possible exceptions.For example, an agent may be able to Intend directly from some raw Perception. Tasking mayalso move around; for example, it may not be necessary to update one’s goals after every cycle,or it may make sense to perform goal maintenance before comprehension has taken place.

In my implementation, Perceive is controlled by an external system which manages the agent’s

perceptual “buffer” (i.e. central cognition’s access to perception). Encoding is accomplished bya set of parallel elaboration rule firings which annotate input with additional information. Attendis an operator which chooses one of currently encoded inputs to process next. Comprehend iswhat the agent does with the encoded input; I will go into detail below. Tasking is anything theagent does to maintain its goals. For example, the system has an operator which generates a newdeclarative goal if the agent doesn’t currently have one. Intending is an operator the agent usesto decide what to do in the world. Decoding will simply be the process of sending actual motorcommands to the motor processor, which is an external system.

4.1.1.2 Properties of the comprehension system

Comprehension is at the center of the PEACTIDM process as it is the one place where

knowledge of the environment (PEA) combines with knowledge of the agent’s goals (T) anddesires to create a representation that can be the basis for action (IDM). My hypothesis is thatmany appraisals are side effects of comprehension. It might appear that I am merely relabeling“appraisal” and calling it comprehension, but my point is that independent of any theory of emotion, a cognitively capable agent must comprehend. More crudely, my hypothesis is thateven Mr. Spock would have a comprehension process, but he would not have the associatedprocesses to compute appraisals and generate the associated emotions. Thus, comprehension is anexus for the integration of cognition and emotion.

The comprehension process is severely constrained by the fact that it is embedded in aknowledge-rich agent with an ongoing existence that must interact with a continually changingenvironment. I also consider additional constraints that arise from what is generally known abouthuman psychology. Many of these restrictions are borrowed from NL-Soar (Lewis 1993), apsychologically plausible natural language processing system based on a comprehension modelof language.

1) Domain independent: The agent should not need separate comprehension systems foreach domain (although there may be domain-specific information which aids theprocess).



19

2) Limited working memory: The agent should not require an arbitrary amount of workingmemory. NL-Soar’s approach to this (and mine as well) is to only represent oneinterpretation at a time.

3) Incremental: The agent should not have to wait until all inputs necessary for a completeunderstanding are available. It should be able to build up its comprehension from smaller

pieces.4) Happens over time: Situations unfold over time, so the agent should be able to constructand refine its comprehension over time.

5) Supports immediate comprehension: The agent should be able to create someunderstanding of the situation from the very first moments of the situation. If the agentwaits it is wasting time that it could be using to do some processing, no matter howspeculative. Furthermore, the agent may want to respond to the situation immediately.Finally, unlike language, which contains markers that indicate logical stopping points(e.g. periods, commas, pauses in speech, etc.) to try to comprehend the words, situationsdo not. The world unfolds as an endless stream of events.

6) Supports hierarchical comprehension: Comprehending a series of events requires not only

understanding the individual events and the connections between individual events, butalso how they fit together at more abstracts levels. Thus, it is not just that an agent isgetting into the car, starting the engine, driving down the road, etc. Rather, it is that thisagent is driving to work, which can provide context for future comprehension andprediction.

7) Supports prediction: A true understanding of the situation includes a prediction of what isgoing to happen next. Prediction aids future comprehension for familiar situationsbecause it allows the agent to confirm if an event is consistent with the current predictioninstead of attempting to process the event in isolation. Prediction also allows the agent toprepare for future events.

The property of “immediate comprehension” leads the agent to require at least two supportingmechanisms:

8) Immediate ambiguity resolution: Many events will be ambiguous when encountered inisolation. One approach is to defer commitment until a unique interpretation is clear;however, this can result in a combinatorial blowup in processing and memory usage asmultiple ambiguous events are encountered. An alternative is for the agent to commitimmediately to a particular interpretation (i.e. the “best” one), which avoids thecombinatorial blowup and supports immediate comprehension.

9) Error recovery: Since the correct interpretation is ambiguous, the agent may choose thewrong one. Thus, the agent must have the capability to recover when it discovers it hasmade an incorrect assumption. As in NL-Soar, I expect that recovery will usually involvea local repair to its interpretation, but can also require a complete reinterpretation in“garden path” situations.

The comprehension process is essentially its own weak method. That is, it provides a universalframework for solving problems. Soar itself provides a lower-level framework for weak methods (Laird & Newell 1983), but comprehension provides additional useful structure.



20

4.1.1.3 Building blocks of comprehension

In this section I will discuss the theories I am using to construct a comprehension system in acognitive architecture. For now, I intend to construct the comprehension system on top of thecognitive architecture using the pieces I describe here. However, given that comprehension issuch a core process to the agent, it may make sense in the future (post thesis) to incorporate some

architectural support for these building blocks (I will not discuss this possibility further).

With regards to comprehension, Newell (1990) said:

“Soar interprets the environment by applying comprehension operators to specificaspects of an environment it wants to comprehend….What gets produced byexecuting a comprehension operator is a data structure in the current state that isthe comprehension, by virtue of its being interpretable by other parts of Soar indoing other tasks.”

The data structure I have chosen is a schema (Rumelhart 1980). A schema is actually more than

just a data structure – it is a concept which consists of a variablized data structure and knowledgeabout how that data structure should be instantiated. For example, I may have a schema for theconcept “buy” that looks like this:

Schema: BuySeller: <S>Buyer: <B>Price: <P>

The items in brackets are variables with some ranges. Once some of the variables have been set,the agent can infer the values of some of the others (or at least possible values). For example, if

<P> is known to be greater than $100, then the agent might infer that the list of buyers couldinclude people like Bill Gates and Richie Rich, but not Bob Marinier. This inference mightdepend on another concept, “expensive” which the agent could use to determine if it could be thebuyer. Of course, what is expensive is inherent to this agent’s concept; Richie Rich wouldprobably not consider $100 to be expensive, but Bob Marinier would.

Not all elements in a schema are necessarily variablized. Some may be concrete values that areinherent to the concept.

My comprehension system is actually built on event schemas (Zacks & Tversky 2001). For mypurposes, an event is a period of time in which the primary actor and the action he is engaged in

do not change. This is a simplification of Talmy (1975) in which he identified figure, motion,path and ground as being the key identifiers of an event.

The reason for focusing on events is that situations seem to be composed of series of events, andit is the understanding of these events that let an agent choose actions to initiate its own events.Schema theorists would probably argue that schemas are used to comprehend everything, not justevents, but that is well beyond the scope of my research.



21

To get an idea of how event comprehension works, suppose I have a schema for the “crossing thestreet” event:

Schema: Crossing the streetEvent 1: Step down from curb

Event 2: Walk across streetEvent 3: Step up onto curb

As you can see, the abstract event “crossing the street” is defined in terms of more concreteevents. These, in turn, may be broken down further. At some point, however, an abstract eventwill be composed of ground events which are the most basic units. These ground events are theevents that the agent recognizes when it encodes raw inputs. They also correspond to actionsthat the agent can directly execute in the world. For simplicity, I have not shown the explicitactors and actions, which would typically be variablized, in this example.

Now suppose the agent observes someone stepping down from a curb. It can now recognize that

this is the first step of the “crossing the street” schema. If it commits to this interpretation, it canmake predictions about what will happen next (e.g. that the person will walk across the street andstep up onto the other curb). It can also infer something about the other agent’s goals (e.g. that itwants to cross the street). Perhaps most importantly, the agent can infer what the meaning of thisevent is with respect to its goals – is it ok for me if this person crosses the street? When the nextevent occurs, this may confirm the predictions or the agent may have to reinterpret using someother event schema. The initial choice of event schema may be impacted by several factors; forexample, if the agent wants the person to cross the street, then it may be biased to interpret theinitial event that way.

As alluded above, the agent’s goal can also be represented by an event schema. That is, the goalschema is just whatever event the agent wants to occur. Each subevent of the goal schema isreally a subgoal, then. That is, the agent can represent a goal hierarchy using event schemas.These declarative goals should not be confused with Soar’s architectural goal stack. Soar’sarchitectural goal stack is used to implement the comprehension process, but the agent’s task goals (the ones that it actually cares about, in some sense) are represented declaratively as eventschemas.

4.1.1.4 The comprehension process

The comprehension process basically creates and maintains event schemas which represent theagent’s understanding of what is currently going on.

So suppose that an agent already has a schema it has committed to based on prior processing. Itperceives and encodes an event and then attends to it. Since it is an event, it executes its“comprehend event” operator. This operator breaks down into two steps: determine thediscrepancy from expectation, and update the current interpretation (i.e. the event schema).Determining the discrepancy from expectation is necessary in order to decide how theinterpretation should be updated. For example, if the event exactly matches the agent’sprediction, updating the interpretation may just be a matter of denoting that the event hasoccurred. On the other hand, if the event is slightly different (i.e. of the same type but with



22

different variable bindings), then the event may need to be swapped in for the prediction, and thefuture predictions updated to reflect the latest information. Finally, if the event is completelydifferent than what was expected, the agent may need to discard the current interpretation andreplace it with a reinterpretation that is consistent with its observations.

This is the first example of how some appraisals may “fall out” of the agent’s processing. The“determine discrepancy from expectation” operator is an appraisal from Scherer’s (2001) theory,but similar ones exist in other appraisal theories.

One of the design decisions I made with the comprehension system is to require the agent toalways have a prediction of what it thinks will happen next. As argued earlier, an agent that doesnot have some prediction of the future does not really understand the situation. Furthermore,many appraisals rely on having some prediction (e.g. outcome probability, discrepancy fromexpectation, etc.) Realistically, there may be some situations in which an agent cannot make areasonable prediction (possibly resulting in confusion), but as a simplification I will not considerthat possibility in my research. Thus, when the agent reaches the end of its current schema (e.g.

if the person finishes crossing the street), the agent needs to determine what the next schemashould be. One way of doing this is to determine what abstract schema the “crossing the street”event is part of. For example, given the particular person and street, the agent might suppose thatthe larger event that is taking place (i.e. the goal of the person) is to “go to work.” The nextevent in the “go to work” schema may be to “enter the building,” which in turn requires “walkingup to the building” and “opening the door.” Thus the agent might predict that the next event thatwill take place is “walking up to the building.” As this demonstrates, predicting the next eventrequires a series of abstractions and specializations.

Figure 6 shows the comprehension aspects of the PEACTIDM process, and Figure 7 shows howevent comprehension fits into the emotion process. As depicted in Figure 6, once the agent hascomprehended the event, it can generate another appraisal, “goal/need conduciveness.” Thisappraisal is different in that it does not “fall out” of the comprehension process, but ratherfollows it. Since it isn’t required by the agent’s processing, the agent may only generate it if ithas enough resources to do so (e.g. time). Even if there is another event waiting to be processed,the agent could decide to delay or skip that in favor of doing this appraisal. Perhapsreinforcement learning could play a role in helping to determine the best choice to make.

In general, appraisals can be divided into at least 3 groups: 1) automatic appraisals (e.g. noveltymay be computed automatically by the high-level vision or long-term memory systems), 2)deliberate but required appraisals (i.e. those appraisals which “fall out” of the comprehensionprocess), and 3) deliberate but optional appraisals (i.e. those which the agent may do if it hastime). This is not to imply that the agent can choose not to do optional appraisals. Rather,continued comprehension may take precedence over these appraisals, so they may be skipped if events are occurring rapidly. Table 5 shows some possible appraisals and what types they mighthave.

These particular appraisals are merely exemplars. Rather than take a firm stance on the exact setof appraisals at this point in my research, I view the comprehension process as providingconstraints on what the set of appraisals should include. That is, some appraisals may fit more



23

naturally than others into the comprehension process. Of course, the constraint goes both ways inthat the comprehension process must allow for many common appraisals.

Novelty and intrinsic pleasantness are likely properties that arise during the encoding process,which is mostly automatic. The agent may get causality automatically when it chooses a schema.

That is, the schema itself may include the causality information in its structure. That is not to saythat determining causality in general is not a complex process that is not automatic; presumablythe agent must sometimes go through that process when it is generating a new schema tounderstand a novel set of events.

The distinction between required and optional deliberate appraisals may lie primarily in what isrequired to understand what is happening in order to immediately act vs. what is required tounderstand the relationship of what is happening to the agent’s goals. This distinction is slightlymuddied because the agent’s interpretation is colored by its goals. Thus, I consider goal/needrelevance to be required; it may occur as part of the attend process (i.e. the agent may ignoreevents irrelevant to its goals). Outcome probability is probably also required, although I do not

yet know how it fits into the comprehension process. Goal/need conduciveness and copingpotential, on the other hand, do not seem to be critical to the immediate understanding of what ishappening, but clearly it is in the agent’s interest to do these if it can.

Figure 6: The comprehension process so far, based on PEACTIDM. Left to right is roughly the operator

sequence, and top to bottom shows how some of the steps may break down into architectural subgoals.

Double-lined boxes depict operators expected to require architectural subgoals.

AttendComprehend

EventDetermine Goal/Need

Conduciveness

Determine Discrepancyfrom Expectation

UpdateSchema

Reinterpretevents

Generate ConcretePrediction

Encode

Perceive

AbstractSchema



24

Agent-Environment

Transaction


Cognitive ∆

Subjective feelings

Responses to

Emotion


C o p i n g

R e g u l

a t i o n

C o m

p r e

h e n s i o

n

Figure 7: Comprehension is the process that results in appraisal.

Appraisal Type Appraisal

NoveltyIntrinsic pleasantness

Automatic

Causality

Goal/Need relevance

Discrepancy from expectation

Deliberate-required

Outcome probability

Goal/need conducivenessDeliberate-optional

Coping potentialTable 5: Possible types for common appraisals.

4.1.1.5 Properties revisitedSo, does the comprehension process fulfill each of the requirements I described earlier?

1) Domain independent: The system works on events, which are a domain-independentrepresentation of what is happening. There can be domain-dependent knowledge used incomprehension, but the overall process can be used in any situation.

2) Limited working memory: The system only represents one goal and one interpretation ata time. In order to determine what subgoal is in progress or to generate a next prediction,



25

it destructively abstracts and specializes the current interpretation. This could lead tointeresting phenomena in which an agent loses track of what exactly it is doing, but ingeneral I expect the activations of the right things to be high, leading to a low error rate.

3) Incremental: The system can attend to and process one event at a time.4) Happens over time: Since events occur over time, the agent processes them over time.

The delay between events is actually necessary in order for the agent to complete itsprocessing without becoming overwhelmed. The agent may even be able to utilize extratime it has to generate additional appraisals.

5) Immediate comprehension: The agent commits to a current interpretation schema startingwith the first event.

6) Supports hierarchical comprehension: The event schemas describe events at multiplelevels, directly supporting hierarchical comprehension.

7) Supports prediction: The hierarchical nature of comprehension and event schemasdirectly support prediction. Once a high-level schema has been recognized, thesubsequent events that it contains are predictions.

8) Immediate ambiguity resolution: If multiple interpretations are available, the agent will

pick one that best fits its goal, was used most recently, or randomly.9) Error recovery: There are at least two levels of error recovery: a simple level, in whichthe schema structures are correct but the variable bindings need to be updated, and acomplex level in which the structure itself is incorrect and the agent needs to choose anew schema. The simple level may be implemented via a truth maintenance system thatautomatically updates the variable bindings used in the next predicted event. Thecomplex level may utilize arbitrary processing to find a schema that fits the wholesequence. For example, the agent can recall the most recent event sequence fromepisodic memory and try to find a schema that matches that set. I do not intend toexplore this complex level if possible. I also have not yet explored to what degreeintermediate levels of error recovery are possible.

4.1.1.6 Predictions

From this theory I can derive a number of predictions.

First, because the comprehension process results in immediate comprehension, if the agent’sperception of a stream of events is interrupted, the agent should have an understanding of on thesituation up to that point (which will often include predictions as to what might happen next andwhat other agents are trying to do). Thus, an interrupted agent will behave based on theinformation received so far, but that behavior may be flawed in predictable ways due to mistakesin the agent’s interpretation (similar to garden path phenomena in language processing).

Furthermore, the comprehension process imposes a partial ordering constraint on appraisalgeneration. For example, as shown in Figure 6, the “Discrepancy from Expectation” appraisaloccurs before the “Goal/Need Conduciveness” appraisal. This is consistent with Scherer’s (2001)theory which also hypothesizes a sequential ordering. However, the reasons for the hypothesesdiffer. Scherer’s reasoning is that it would be a waste of processing resources to do someappraisals when the results of others show that they are irrelevant. Thus, for example,“Discrepancy from Expectation” comes after “Goal Relevance,” because if an event is not



26

relevant, then the agent should just ignore it. By contrast, our theory imposes an orderingbecause of the functional requirements of the comprehension process.

A corollary of this is that our model allows for the possibility that different emotions willinherently require different amounts of processing. If some emotions only require appraisals that

occur earlier in the comprehension process, then those emotions will take less processing thanthose that require more appraisals. The idea that different emotions may require differentamounts of “cognitive activity” is not actually new (Lazarus 1982). This may lead to timingeffects.

Given that comprehension takes time, if there are tight time constraints, some appraisals may notget generated, meaning that under certain time pressures, some emotions may not occur, or maybe based on appraisals that were generated in earlier situations. Under extreme time constraints,comprehension itself may not be possible, leading to purely reactive behavior. In between, it maybe possible that an agent “misses” some of the events, leading to flawed interpretations.

Finally, our model supports appraisals that can happen at different time scales. Some appraisalsmay be based on comprehension in novel circumstances that require multiple retrievals fromlong-term memory, or even significant internal problem solving to understand the situation,while others could be based on comprehension in well practiced situations where essentiallyreactive comprehension is possible. These differences in comprehension processing can lead tovery different time scales for generating appraisals. Combining this with the previous two points,I predict that the complete appraisal (and thus the emotional reaction) can change over time asthe comprehension of the situation evolves over time.

4.1.2 What are the proper dimensions?

Every appraisal theory makes slightly different claims about what the dimensions are. However,

often times the differences really seem to be splitting hairs, and it’s unlikely that the exactdimensions I choose will have a large impact on this research. That said, I will favor thosedimensions that seem to best fit into the comprehension process. This is an additional constraintthat appraisal theorists have not yet considered. In fact, the constraint is likely to go both ways –I expect to modify the comprehension process in order to account for appraisals that the literaturedeems necessary. For example, the process as I have described it does not yet account fordetermining if events are relevant (it assumes that they are). Finally, I intend to ignoredimensions like internal and external standards compatibility because they seem to require asignificant amount of cultural knowledge which is beyond the scope of my research. Besides,according to Table 2, many emotions are possible without them, so this does not a limit my basicresearch program.

4.1.3 Are the emotions categories or modal spaces?

Looking at Table 2, it may look like the emotions are strict categories. Indeed, some appraisaltheories postulate strict categories (e.g. Roseman 2001). Some appraisal theories, however, think of these emotion labels as regions within the n-dimensional space defined by the appraisals(Scherer 2001). Some of the appraisal dimensions, like causality, are categorical, but others, likeoutcome probability, are probably continuous. Thus, an emotion is really some region of the n-dimensional space. Within what I might call the anger region are an infinite (or at least large)



27

number of variations of anger, corresponding to each point within the space. Some regions in thespace may not have any cultural label at all, and the edges between labeled regions may be fuzzy(i.e. how a person decides to label such a region may be inconsistent).

I intend to represent emotions as n-dimensional points internally, but the agent may label its

feelings and work primarily with those labels.

4.1.4 Summary

The appraisal theory design decisions are summarized in Table 6.

Open Questions Furtherresearch?

Comments

How are theappraisal valuesgenerated?

Yes I will refine the situation comprehension process tosupport an appropriate range of deliberate and automaticappraisals.

What are the

proper appraisaldimensions?

Yes This will be strongly influenced by what fits well into the

comprehension process (and the comprehension processwill be strongly influenced by what it needs to fit in).

Are the emotionscategories ormodal spaces?

No I have decided to use an n-dimensional representation forthe emotions.

Table 6: Summary of appraisal theory design decisions.

4.2 Post-appraisal changes design choices

There is a plethora of appraisal or emotion-induced changes cataloged by the literature:physiological, neurological, facial, vocal, cognitive, etc. Most of these are not really appropriatefor integration with a cognitive system like Soar until it is able to incorporate a detailed

physiological model, so deciding what to focus on is important. There are also a number of detail questions that need some attention in order to implement a system.

1) How long does an emotion last?2) How is emotion intensity calculated?3) What does an agent feel?4) What cognitive changes should be modeled?5) How do action and thought urges fit in?

The first two questions are actually related, so I will address them together.

4.2.1 How long does an emotion last and how is its intensity calculated?When we think of temporally extended emotions, we are really entering the domain of mood .Most of the emotion literature does not carefully distinguish between emotions and moodsbecause concrete definitions for either have not been settled on (and may not be possible (Smith& Lazarus 1990)). However, Rosenberg (1998) identifies some useful properties; primarily thatmood has a longer temporal duration than emotion and that mood and emotion should influenceeach other.



28

One possible approach is to represent mood as a baseline in the n-dimensional emotion space towhich emotion decays once its generating appraisals have disappeared. Emotion, in turn, pushesthe baseline in the direction of the emotion. Mood probably also decays towards some neutralpoint, but much more slowly than emotion. It also makes sense that the intensity of a newemotion would be enhanced or subdued by the current mood depending on whether the mood

was congruent or not with the emotion.

This approach makes some assumptions and raises some difficult questions. In particular, thisassumes that moods and emotions are of the same kind, which implies that there is acorresponding mood for every emotion. There is also the question of how discrete dimensionsdecay to baseline – how does causality decay from, for example, you to me? Finally, there isevidence that the baseline mood for humans is slightly positive (Cacioppo & Gardner 1999) butis it happy, elated, interested, or something else?

Previous work by Gratch & Marsella (2004) ignored the notion of mood but still maintainedinteractions between emotions by creating a matrix that described which emotions offset each

other and to what degree. The intensity of a new emotion was combined with the previousemotion using this matrix to determine the final intensity of the new emotion. For example, if the previous emotion was mild anger and the new emotion was strong happiness, the finalemotion might be moderate happiness.

Given the complexity of exploring the available options, and the lack of guidance that theliterature can currently provide, my approach is to keep it simple. Thus, my initialimplementation will ignore moods and interactions between emotions in favor of the simplestmodel: the emotional state is a direct result of the appraisals. If I have time, I may explore one of these other approaches.

Even with the simple approach, the question of how the intensity is generated is still unanswered.Gratch & Marsella had continuous appraisal dimensions corresponding to likelihood anddesirability, which they simply multiplied together to get intensity. I will have more metricdimensions to work with, so the development of a function that combines the appropriate onestogether into a single value will be necessary. Again, I intend to start with a simple model andonly explore further as necessary.

4.2.2 What does an agent feel?

Damasio (1994) makes a sensible distinction between what an agent’s objective emotional stateis, and what it subjectively perceives its state to be. What it perceives is called its feelings.Work by Feldman Barrett (2004) also supports the notion that some people are more sensitive

than others to their current emotional states, indicating differences in perception. It is importantfor the agent to feel something – otherwise, there is no way for the agent to cope with itsemotional state (since it doesn’t know anything about it). The simplest feeling implementationwould just give the agent direct access to its emotional state – in my implementation, this meansmaking the n-dimensional point that describes the current location in appraisal space available toperception. The agent will then need a set of rules that are able to recognize the point asbelonging in a particular modal space (e.g. anger) so the agent can label it for its own purposes.



29

These rules represent a combination of cultural knowledge and personal experience about howvarious feelings related to other changes in its past2.

4.2.3 What cognitive changes should be modeled?

A good first guess at what aspects of cognition emotion impacts and when might be “everything”

and “always.” There is evidence that the direct impact of emotion is more limited than this.

I will describe several phenomena of interest and attempt to identify the cognitive functionalityrequired to support the phenomena. As we will see, Soar does not yet support most of therequired functionality, making it difficult to include these phenomena in my research.Furthermore, some of the phenomena require an in-depth understanding of how humans docertain tasks (i.e. categorization), which are really outside the scope of my research.

First I will describe mood state dependent retrieval and mood congruent retrieval, which willgive insight into why not all of cognition is directly impacted by emotion. I will then describeseveral other phenomena including categorization effects, the broaden and build theory, episodic

memory effects, undoing, priming effects, judgment effects, and higher-level phenomena thathave not been specifically studied in the literature.

4.2.3.1 Mood state dependent retrieval and mood congruent retrieval

Forgas (1999) describes two phenomena, mood state dependent retrieval and mood congruentretrieval, that some researchers have had difficulty reproducing. Forgas analyzed the specifictasks that were used to test the phenomena and concluded that different processing strategieswere used for different tasks, and that some of those strategies are not impacted by emotion.Those processing strategies appear to map fairly directly onto various cognitive architecturalcomponents, which implies that some of those components are unaffected by emotion. I willbriefly describe Forgas’s work below.

Mood state dependent retrieval is the phenomena that if someone is in mood X when he encodessome information, then it will be easier to recall that information later if he is in mood X again. Mood congruent retrieval is the phenomena that if someone is in mood X when he tries to recallsome information, the retrieval will be biased towards information related to that mood (Forgas1999). Forgas says that mood is intended to refer to “weaker, more enduring and lessconsciously accessible affective states” than emotion. For my current research, however, whichlacks a strong distinction, I think it is safe to drop this difference.

Forgas (1999) reviewed various experiments that tested these phenomena, some unsuccessfully,and found that different experimental setups required different processing strategies, and only

some of these strategies were impacted by emotion. He describes four different processingstrategies that humans seem to use to solve problems:

1) Direct Access processing strategy: “The simplest method of performing a cognitive task,based on the strongly cued retrieval of stored cognitive contents.” Knowledge is

2 I say “represent” because I do not intend to explore how the agent learns these rules.



30

represented in a “crystallized” form which is accessed quickly and automatically. In acognitive architecture, this strategy maps onto using procedural memory.

2) Motivated processing strategy: “Occurs when information processing is guided by astrong, pre-existing objective.” Knowledge is found via a goal-directed search. In acognitive architecture, this strategy maps onto using architectural subgoals.

3)

Heuristic processing strategy: “Occurs when subjects have neither a crystallized responsenor a strong motivational goal…and they lack either personal involvement or sufficientprocessing resources.” Knowledge about emotions (“affect-as-information”) may bebrought to bear. In a cognitive architecture, this maps onto having a representation of thecurrent emotional state in working memory (i.e. a feeling) which can then be used byother mechanisms (e.g. procedural memory).

4) Substantive processing strategy: “People need to select, learn, interpret and processinformation about a task, and relate this information to pre-existing knowledge structuresusing memory processes.” This is a “default” strategy used when none of the othersapply. In a cognitive architecture, this maps onto using slower, more error pronemechanisms like episodic and semantic memory.

Forgas claims that each of these strategies is successively more effortful, and thus, for example,the substantive processing strategy will only be used if the earlier strategies fail. Forgas keyclaim is that only the substantive processing strategy is directly impacted by emotion. Thus, thereason some experiments were unable to reproduce the mood state dependent retrieval and moodcongruent retrieval phenomena is because the nature of those experiments allowed other non-substantive processing strategies to be used. For example, if the processing required was toosimplistic, then subjects would not have to use the substantive processing strategy, and thus theeffects of emotion would not show through.

This is an interesting result, because it means that we don’t need to consider how emotionimpacts rule firings or impasses; essentially, the traditional mechanisms in Soar are unaffected.For this reason, these and other cognitive phenomena discussed in this section are unlikely to fitinto the traditional Soar architecture that is currently available to me and thus I will not be able toexplore them.

It seems that the cognitive functionality required to explain these retrieval phenomena ismetacognitive knowledge that associates emotion (or some region in the n-dimensional emotionspace) and knowledge that was acquired in the context of that emotion or is somehow related tothat emotion. Given Forgas’s theory, the cognitive components most likely to be influenced byemotion are semantic and episodic memory.

4.2.3.2 Categorization effects

Isen & Daubman (1984) present a series of experiments in which subjects have to judge howwell various items fit into categories. Those induced with positive emotion tend to rate poorexemplars higher than those induced with neutral or negative emotion (although those in thenegative emotion condition rate items higher than the neutral condition as well). In anotherstudy, subjects grouped colored chips however they wanted to. Similar to the previous studies,those induced with positive emotion created the fewest groups, and those with negative emotioncreated fewer groups than those in the neutral condition. The negative emotion effects did not



31

show up in all studies with significance, although this may have been due to a flaw in theexperiment design.

The authors argue that these phenomena point to a change in processing induced by positiveemotion, and possibly also negative emotion. Some possibilities include that emotion provides

or induces extra features that can be used when cueing retrieval, which may make categoriesseem more inclusive (since all the items may seem to have a positive emotion feature, forexample). It’s unclear, however, how an emotion feature comes to be included with an item in,for example, semantic memory. This seems to require a deeper understanding of how humans docategorization and grouping, which is beyond the scope of my research.

4.2.3.3 Broaden and build

Traditionally, theorists have claimed that emotions are paired with specific action tendencies.Fredrickson & Branigan (2005) observed that this really only applies to negative emotion, andthat attempts to define the action tendencies for positive emotions has resulted in vaguetendencies. Rather than resulting in action tendencies, Fredrickson & Branigan suggest that

positive emotions induce a change in the style of cognitive processing. They propose thatemotions are associated with thought-action repertoires. Specifically, something (unidentified)is broadened during positive emotions, leading to larger though-action repertoires for positiveemotions. Negative emotions, by contrast, result in a narrowed thought-action repertoire. Thisbroadened thinking can result in the development of new resources, including physical, social,and cognitive resources. That is, broadened resources can lead to the recognition of newrelationships that the agent can learn about. For example, Isen & Daubman (1984) showed thatsubjects in a positive emotional state were more likely to classify elevators and camels as typesof vehicles. The agent can then use that knowledge later to help form a more effective responseeven during negative emotions. While the theory of broaden and build goes beyond cognitiveeffects (e.g. social effects), I will only be exploring cognitive effects.

This kind of control makes sense when one considers that negative emotion situations areprobably unsafe, and thus an agent should not be trying to explore and learn, but rather should betrying to use what it already knows to survive. Conversely, positive emotion situations areprobably safe, and thus the agent can take advantage of the opportunity to explore and learn newthings that may be useful to it in unsafe situations.

There are a few different ways that broaden and build may integrate with a cognitivearchitecture. The important thing is that there needs to be some metacognitive information thataffects what knowledge is brought to bear or what decisions are made. Since broadening andnarrowing can be thought of as a one-dimensional concept, we can think of the architecture

having a “knob” that can be turned to change how some components work. Possible knobsinclude the exploration rate in reinforcement learning and the retrieval threshold and the noiselevel in semantic and episodic memory.

4.2.3.4 Episodic memory

Philippot & Schaefer (2001) describe what they call a hierarchical autobiographical memory(ABM). The hierarchy ranges from general to specific. General ABMs are things like, “When Iwas a child, we spent summers camping.” Specific ABMs are things like, “That one time we



32

were camping and I got lost in the woods.” Given these definitions, it seems that specific ABMsmap onto what I have called episodic memory, whereas general ABMs seem more like semanticmemories about episodic memories. For example, I know that I went camping frequently as achild – this is a fact, and not one that I learned at a specific time.

Philippot & Schaefer claim that only general ABMs are affected by emotional state. They citeseveral studies. For example, Philippot & Dozier (1996) had subjects watch an anger-inducingfilm. The subjects then reported the ABMs that spontaneously occurred to them and theemotional intensity felt during the film. After the film, the subjects rated each ABM for itsemotional intensity. Independent judges then classified the ABMs as specific or general. Therewas a positive correlation between the emotional intensity felt during the film and both thenumber of reported ABMs and the emotional intensity of the reported general ABMs, but therewas no relationship with the specific ABMs.

Following my inference that general ABMs are semantic memories and specific ABMs areepisodic memory, these results show that episodic memory is not directly impacted by emotion,

but semantic memory is. This fits in with some of the ideas about how broaden and build mightintegrate with semantic memory.

This phenomenon, then, helps refine our understanding of how broaden and build might integratewith cognitive architectures in that it may eliminate episodic memory’s mechanisms ascandidates, leaving just semantic memory.

4.2.3.5 Undoing

Fredrickson & Levenson (1998) describe a phenomenon called undoing. Undoing is the ideathat positive emotions undo some of the effects of negative emotions. For example, during fearone’s heart rate may increase dramatically. A subsequent positive emotion will cause the heart

rate to return to baseline more quickly than if the effect were just allowed to “wear off” on itsown. Importantly, if someone is already at baseline, then inducing a positive emotion will haveno effect.

In terms of broaden and build’s cognitive effects, undoing may reverse the narrowing effects of negative emotions and cause things like the exploration rate, retrieval thresholds and noise toreturn to their base levels.

It seems that there needs to be metacognitive information that tells the architecture that it is ok toreturn to baseline. The reason the change is not instantaneous is because humans are biologicalsystems that generally do not support instantaneous changes (at least at the physiological level).

Since I am not exploring physiology or the broaden and build phenomenon, it does not makesense for me to explore this phenomenon.

4.2.3.6 Priming effects

Neumann (2001) describes an experiment in which subjects are primed with a sentence-formingtask. Subjects took a phrase such as “make some calls” and changed it into a self- or other-referenent sentence such as “I make some calls” or “He makes some calls.” They were thenplaced in an ambiguous situation that could invoke anger or guilt. Those who had been primed



33

with self-referent sentences were more likely to express guilt whereas those primed with other-referent sentences were more likely to express anger.

Neumann’s goal in this study was to show that attribution of causality caused an emotionalresponse. This is consistent with the appraisal theories I discussed earlier. What I find

interesting, however, is the priming effect itself. This study shows that a primable mechanism isinvolved in generating appraisals. Given the restrictions on the mechanisms that can be involved(i.e. not procedural memory) and the nature of the priming task, it seems likely that semanticmemory is used in some way to generate at least the causality appraisal (and likely others aswell).

4.2.3.7 Judgment effects

Lerner & Keltner (2001) show that subjects who feel anger are optimistic in their risk assessments whereas those that feel fear are pessimistic. This goes against traditional literaturewhich had demonstrated a valence effect – positive emotions lead to optimistic risk assessmentsand negative emotions lead to pessimistic risk assessments. Lerner & Keltner show that in

ambiguous situations, the overriding factor in determining how risk is assessed is the level of certainty and control, not valence. Anger is characterized by high certainty and control whereasfear is characterized by low certainty and control, leading to the observed phenomenon. Theyalso show that anger and happiness are nearly indistinguishable in risk assessment tasks, which isconsistent with their view that happiness is also characterized by high certainty and high control.Finally, they show that in unambiguous situations, the certainty/control effect disappears (sincethere isn’t a need for interpretation) and the traditional valence effect emerges. These effects areduplicated in a variety of ways, including judgments about others vs. the self and actuallyinduced emotions vs. personality traits.

I don’t have any good ideas about how this would work in a cognitive architecture. It seems that

we need a deeper understanding of how the judgment process works in order to see how thisphenomenon could fit in. There has been work on how humans do not use “rational” theories inestimating probabilities and other related judgments (e.g. Kahneman & Tversky 1973), but that isbeyond the scope of my research.

4.2.3.8 Higher-level phenomena

The phenomena described so far are fairly low level, and their effects may be difficult to testsince the impact on agent behavior is indirect. There are other more obvious phenomena that, tomy knowledge, have not even been cataloged, perhaps because they are so obvious.

The first example is something I will call retroactive comprehension. Consider this scenario:

Jack has a daughter Molly. Molly loves Chinese food.

Jack: How about we get some Chinese food tomorrow night?Molly: That would be great! Wait, I’m not going to be here tomorrow!

What happened here was that Molly’s initial interpretation failed to take into account theknowledge that she wasn’t going to be home for dinner tomorrow night. Thus, she had an initial



34

positive reaction (probably excitement). However, she then reinterpreted in light of thisinformation she suddenly remembered, and that caused a change in her agent-environmenttransaction, which resulted in a negative emotion (probably dismay).

The comprehension process described earlier doesn’t explicitly account for this possibility

because it only updates the current interpretation in response to new events. However, we cansee that there is a temporal aspect to the interpretation of the information (or the retrieval of relevant information). Possibly there is a delay in the retrieval of the critical information that shewill not be home tomorrow, or there may be multiple retrievals involved. In either case,comprehension proceeds with the information it has (as expected given its property of immediatecomprehension). Thus, when the critical information finally is available, a reinterpretation isrequired, resulting in the realization.

Unfortunately, Soar does not currently have any retrieval mechanisms that incorporate delayedretrieval. Even if multiple retrievals are used, it is unclear what the separate retrievals are.Given these difficulties, it is unlikely that I will be able to reproduce this phenomenon, but I

hope to at least gain a better understanding of the required mechanisms, including how thecomprehension process can support this.

I do not currently have other high-level phenomena to report, but I hope to add to this list overtime.

4.2.3.9 Summarizing cognitive changes

In summary, there are numerous interesting phenomena regarding how emotion might impact thelow-level cognitive processing, but Soar is not currently in a good position to address them, andsome require a deep understanding of other areas that are beyond the scope of my research.However, research is currently underway to add much of the required functionality to Soar,

which may make some of these viable options for future (post thesis) work. Table 7 summarizesthe phenomena described in this section.



35

Phenomena Required Functionality/ Understanding

Possible mechanisms

Mood state dependentretrieval, mood congruentretrieval

Metacognitive knowledge thatassociates emotion andknowledge that was acquired

in the context of that emotionor is somehow related to thatemotion.

Nodes and activation insemantic memory.

Categorization Mechanism that allowsemotion to be used as a cuefor category information. Adeeper understanding of howhumans categorize.

Unknown, but may beencompassed by broaden andbuild.

Broaden and build One-dimensionalmetacognitive information thataffects what knowledge is

brought to bear or whatdecisions are made.

Exploration rate, noise level insemantic memory (notepisodic memory).

Episodic memory N/A Episodic memory is notdirectly affected by emotion.

Undoing Metacognitive informationthat tells the architecture it canreturn to baseline.

Same as broaden and build.

Priming Metacognitive informationthat describes how recentlyinformation has been used.

Semantic memory can beprimed.

Judgment A deeper understanding of

how humans judge things.

Unknown.

Molly’s reinterpretation Temporal aspects toinformation retrieval orinterpretation.

Semantic or episodic retrievaldelays or multiple retrievals.

Table 7: Summary of cogitive-emotional phenomena.

Figure 8 shows how central cognition with emotion might look. Emotion is representedalongside short-term memory because it is a non-symbolic analogue to short-term memory’ssymbolic representation of the situation.



36

Short-

term

Memory

Long-term Memories

P r o c

e d u r

a l

Episodic

S e m a n t i c

Decision Procedure

B o d y

B o d y

Emotion Appraisal

C onf i d en c e P

r o p o s a l &

S e l e c t i o n

R e i n

f o r c e m e n t

L e a r n i n g E n c o d

i n g &

R e t r i e v

a l

Feelings & AppraisalPerception & Motor

N o i s e

, T h

r e s h

o l d ,

A c t i v a t i o

n &

A p p

r a i s a l

Figure 8: Basic central cognition augmented with emotion.

4.2.4 How do action and thought urges fit in?

In the previous section, the broaden and build theory arose out of a desire to explain why actiontendencies don’t make sense for positive emotions. This led to the notion of thought-actionrepertoires; that is, sets of possible thought-actions which might be “broadened” or “narrowed”depending on the current emotional state.

While I do not intend to explore broaden and build, incorporating the notion of thought-actionrepertoires makes sense independently. That is, thought-action repertoires supply the possiblethings an agent can do, and the agent must have choices if it is to be interesting. Furthermore, itmay make sense to incorporate an actual urge to do something beyond it merely being available.

Indeed, the idea of thought-action repertoires fits naturally into cognitive architectures. From acognitive architecture perspective, actions and thoughts are just two manifestations of operators.Operators can send motor commands to the motor processor, leading to actions in the world, butthey can also be used to support internal processing (e.g. mental exploration of the possibilities).Thus, thought-action repertoires map naturally onto proposed operators. A simplistic view of

urges might be that an urge to do or think something is synonymous with it being an operatoravailable for selection. It is also possible that urges help distinguish among the possibleoperators. That is, an urge may be some metacognitive information which tells the decisionprocedure how important or likely useful an operator is. This metacognitive information may just be the value of the operator as learned via reinforcement learning. However, this does notaccount very well for how urges may be stronger when emotions are more intense. Thus, analternative (possibly complimentary) explanation is that operators may have activation which is



37

linked to the intensity of the feeling that was tested as part of the operator’s proposal. Followingthis line of thinking, activation would need to be integrated with operator values.

Research versions of Soar are just beginning to incorporate working memory activation;however, given the infancy of its usage and the lack of clear functional gains for my work, I do

not intend to explore this more sophisticated notion of thought-action urges. However, Soar byits nature supports thought-action repertoires, and thus the simple sense of thought-action urgeswill (indeed must) be included.

4.2.5 Summary

The post-appraisal design decisions are summarized in Table 8.


Comments

How long does anemotion last and

how is itsintensitycalculated?

Yes The intensity aspect will probably be based on some of theappraisals, whereas how long the emotion lasts will

probably be finessed for now (i.e. it will last until the nextone, or some fixed number amount of time).

What does anagent feel?

No I will give the agent the n-dimensional emotion point.

What cognitivechanges should bemodeled?

No Changes at the cognitive level seem to requiremechanisms that Soar does not yet possess and a deeperunderstanding of various human strategies that are outsidethe scope of my research. Thus, I cannot model thesephenomena.

How do action

and thought urgesfit in?

No Soar already supports the simple notion of action and

thought urges, but does not currently support themechanisms that would probably be required for a moresophisticated view.

Table 8: Summary of the post-appraisal design decisions.

4.3 Responses to emotion design choices

As described earlier, there are several ways an agent can respond to its emotions. This subject isreally more accurately called “responses to feelings” since the agent can only respond to itsinputs, and that means feelings, not emotions, in this case. In Figure 7 we see an unlabeledtransition from the emotion-induced changes, including feelings, to responses. What is thistransition? That is, how does the agent go from a feeling to a response (either internal or

external)? Similar to the appraisal process, the agent must comprehend , in some way, its feelingsin order to respond to them. My questions are:

1) How does an agent respond to its feelings?2) What responses should be incorporated?



38

4.3.1 How does an agent respond to its feelings?

Just like the appraisal process, the agent must comprehend its feelings in order to respond tothem. This comprehension process will help determine what kinds of responses are possible andwhat they mean in terms of the agent’s internal structures and operators.

Once again, I will try to follow the PEACTIDM model. An emotion is first perceived; that is theagent gets the n-dimensional emotion point. Next, this point is encoded. The encoding processmay include providing a label or other augmentations that will make it easier to use the feeling3.Next, the agent has to attend to the encoded feeling. In the simple cases I will focus on, therewill probably not be much, if any, competition with other encoded inputs for attention.

Once the feeling is being attended to, the agent will try to comprehend it. I have not spent muchtime on this yet, so I can only speculate what will be involved. The most important aspect, Ithink, will be to contextualize the feeling. That is, to link the feeling to the agent’s interpretationof the situation. Thus, for example, an agent can know who or what is causing it to be angry.This might seem a little backwards – after all, the agent generated the appraisals in the first

place, so of course it knows who’s causing it to be angry. This misses an important point,though – an agent never generates an “anger” appraisal. At best, it is only generating a“causality” appraisal (in this case), and also generating other appraisals. It does not containaccessible knowledge about how the combination of those appraisals is going to lead to anger(that knowledge is contained in the architecture and cannot be reasoned about). Thus, an agentmight have knowledge that if it is angry, the cause of that is whatever the situationcomprehension structures say the causal agent is. But this linking knowledge must exist – theagent does not get this information for free. It is also not necessarily the case that everyappraisal is important and thus must be linked to the feeling. As shown in Table 2, manyemotions are based on only a subset of the appraisals. Thus, the function of the comprehensionprocess is to pull out the relevant information so the agent can use that to form a response.

Once an agent understands it feelings, it can try to cope with those feelings. Coping mayactually combine tasking and intending (in the PEACTIDM sense). For example, if the agent isnot making progress towards its main goal, it will probably be feeling bad. If it is feeling angrythen, since anger is often accompanied by the belief that one has the power to change thesituation, it may decide to engage in problem-focused coping. That is, the agent may form asubgoal that gets it back on track and then begin executing actions in the world to achieve thatsubgoal. Forming the new subgoal is a tasking operation, and pursuing it involves intending. If,on the other hand, the agent is feeling afraid then, since fear is often accompanied by the belief that one is powerless to change the situation, it may decide to engage in emotion-focused coping.For example, the agent may discard its original goal and form a new one. This is a tasking

operation. Once it has a new goal, it can start pursuing it, which involves intending. It may notgo directly from tasking to intending in this case – it may have to go through the situationcomprehension process again as it tries to understand how the situation relates to its new goal,etc.

3 I do not intend to explore possible individual differences in how feelings are labeled.



39

Figure 9 depicts the feeling comprehension process. Figure 10 shows how feelingcomprehension fits into the emotion process.

Figure 9: The feeling comprehension process so far, based on PEACTIDM. Left to right is roughly the

operator sequence, and top to bottom shows how some of the steps may break down into architectural

subgoals. Double-lined boxes depict operators expected to require architectural subgoals.

Agent-Environment

Transaction


Cognitive ∆

Subjective feelings

Responses to

Emotion


C o p i n g

R e g u l

a t i o n

C o m

p r e h

e n s i o n

C o m

p r e h

e n s i o n

Figure 10: Responses to emotion result from comprehending one's current feelings.

4.3.2 What responses should be incorporated?

As just described, several feeling responses naturally fall out of the feeling comprehensionprocess when we consider what structures are available for manipulation.

Generating new goals to replace old ones and reinterpreting the situation are ways of changingthe agent part of the agent-environment transaction – that is, they are forms of emotion-focusedcoping.

Attend Cope

GenerateNew Goal

Generate NewSubgoal

IntendReinterpretEvents

Encode ComprehendFeeling

ContextualizeFeeling

Perceive

Emotion-focused Problem-focused



40

On the other hand, Intending is a way of changing the environment part of the agent-environmenttransaction – that is, it is a form of problem-focused coping. Generating new subgoals supportsmaking these changes.

The remainder of the response strategies I described in section 3.3 deal primarily withanticipation and physiology. I do not intend to explore these areas. I also do not intend toexplore reinterpretation if possible.

4.3.3 Summary

The responses to emotion design decisions are summarized in Table 9.


Comments

How does anagent respond toits feelings?

Yes I will develop a feeling comprehension process that allowsan agent to understand its feelings in the context of itsinterpretation of the situation. This will allow it to

effectively cope with the situation by forming new goalsand taking actions.

What responsesshould beincorporated?

Yes Deciding which responses make sense to include willdepend on the design feeling comprehension process (andvice versa). The tasks that we design for evaluation mayalso suggest an appropriate set of responses.

Table 9: Summary of the responses to emotion design decisions.



41

5 Experimental approachEvaluating computational emotion models is simultaneously necessary and difficult. An idealevaluation would compare the comprehension process to human data at the cognitive level;unfortunately, that kind of evaluation is out of reach. Instead, I have a more conservative goal of

answering the question, can emotion (as supported by comprehension) improve learning and task performance?

In this section I will briefly review how other emotion models have been evaluated in order to setthe stage for appropriate evaluation. Then I will describe what kinds of results I hope to achieve.

5.1 Prior evaluations

Evaluating psychological models is very different from evaluating computational models. Manypsychological models are not defined in a way that makes computational testing feasible; forexample, in a review of models exploring the relationship between emotion and memory,Philoppot & Schafer (2001) state, “An important limitation of the four models presented in this

section is that they are very general and abstract, and, consequently, difficult to test empirically.”Furthermore, the kinds of emotion experiments that are done in psychology tend to require acomplex setup (from the point of view of cognitive models) to induce the emotion (such aswatching videos, writing essays, or being placed in a situation), and then the subject is asked toperform some simple task or to report his feelings. The complexity of the setup is unsuitable forreproduction in a computational system. Most of the difficulty in psychological testing isinducing and detecting emotional states (Larsen & Fredrickson 1999). Fortunately, in acomputational system we can always directly read the emotional state. However, induction maybe non-trivial – we can’t just set the emotional state because then the agent won’t have acorresponding situation interpretation to provide it with context. And we have no basis withwhich to measure the accuracy of the resulting behavior.

Hudlicka & Zacharias (2004) and Marinier & Laird (2004) both took the approach of showingthat their emotional systems influenced agent behavior. In Hudlicka & Zacharias’s MAMIDsystem, they showed that agents with different stereotypical personalities made differentdecisions in the same scenario. In Marinier & Laird’s Soar-Emote system, they showed thateach subpart of the emotion system (physiological, cognitive and social) independentlyinfluenced aggregate behavior, as well as the full combination. They also demonstrated thathistorical information can influence behavior. These results are a good first step, in that anegative result would indicate a major flaw in those systems. However, these results do notindicate if these are the “good” changes (either in terms of psychological accuracy or improvedperformance).

Gratch & Marsella (2004) describe three different approaches to evaluation: believability,cognitive dynamics, and social impact. Traditionally, they say, it is common to evaluate thebelievability of the agents. Unfortunately, there is no commonly accepted definition of believability, and even agents that have been explicitly designed to be unrealistic have been ratedas believable.



42

To explore cognitive dynamics, they propose placing the agent in situations described by theStress and Coping Process Questionnaire (Perrez & Reicherts 1992). This questionnaire askspeople how they would respond to various situations, including how they would cope; actualagent behavior can then be compared to human responses. There are a few issues with thisapproach. One issue is that the questionnaire relies on suitable social behavior, which in turn

requires cultural knowledge. I do not intend to address cultural issues in this research, andwithout a suitable cultural theory, this may degenerate into “programming” the agent to generatethe correct behavior. Furthermore, the time scale of many of the scenarios in the questionnaire istoo long (hours, weeks) or unclear. Finally, respondents are merely asked to introspect aboutthese situations, and thus their responses may not accurately reflect what a person actually in thesituation would do; rather, they may merely be using their own “folk psychology” theories toguess at what they might do. This approach may still be worth investigating in the future, butdoes not seem well suited to my current research.

Finally, to explore social impact, they identify a phenomenon called social referencing. Socialreferencing is when one’s appraisal of an ambiguous situation is influenced by the appraisals of

others. They propose forcing a human to make a decision in an ambiguous case and varying thedecision preferences, expressed non-verbally, of other agents, to see if the human can beinfluenced. This kind of test actually seems to be a test of facial expression and body languagesystems more than a test of internal emotion systems. Since I am not exploring physiological orsocial interactions, this kind of test is not appropriate for my work.

5.2 Designing new evaluations

What none of these evaluation measures explore is improved task performance. Hudlicka &Zacharias and Marinier & Laird only explore changes. Gratch & Marsella are concernedprimarily with human accuracy and influence, but do not directly explore task performance.Evolutionists argue that emotions developed to help improve behavior (Smith & Lazarus 1990;

Cosmides & Tooby 2000). If this is true, then there should be at least some tasks in which wecan demonstrate an improvement. Certainly, studies of subjects with brain damage that impairsemotion (but little else) indicate that they cannot lead normal lives (Damasio 1994).

There are at least two kinds of performance tests that may work well. One has to do withaccelerating reinforcement learning, and the other has to do with using emotion to help chooseappropriate metacognitive strategies (i.e. coping strategies).

5.2.1 Accelerating reinforcement learning

One feature of reinforcement learning is learning to pick important features out of theenvironment so that operators are proposed under appropriate circumstances. If appraisal theory

picks up on important features of the agent-environment transaction that should be used indecision making and if emotion accurately summarizes those appraisals, then an agent should beable to learn appropriate actions faster by using emotion as a distinguishing feature to proposeoperators. Without emotion, the agent would have to detect several “base” features from theagent-environment transaction which is almost certain to take longer.

To be more specific, I will design a task in which the agent must use Soar’s reinforcementlearning capabilities (Nason & Laird 2004) in order to improve its behavior. This aspect of



43

reinforcement learning in Soar works as follows. Each operator begins with very generalconditions for its proposal. As the agent tries out these different operators and gets rewardfeedback for them, it may discover that the reward varies widely for the operators. That is, thereare multiple situations that result in different values for the same operator, so a single value isinadequate to summarize the expected reward for that operator. To fix this, Soar creates a new

operator proposal that is a specialization of the earlier one. That is, it creates a new proposal thathas additional conditions. It uses the activation of working memory elements to determine whatthe best condition is to add. It can now learn a separate value for the operator in the situationdescribed by the new proposal. This process can repeat, with new proposals with new conditionsgetting created to define the space of possible state-operator values.

The hypothesis is that emotion (or more accurately, feeling) summarizes multiple otherconditions, so reinforcement learning can learn faster since it only needs to add the emotion as anew condition, instead of iteratively adding each of the features that the emotion summarizes.

There is a discrepancy in the amount of processing that must be done between the control agent

(which does not have emotion) and the emotional agent. The control agent can essentially testraw input (or close to it), whereas the emotional agent must go through a comprehension processin order to ultimately evoke an emotional response, which it can then use to make a decision.That is, the control agent may be able to propose an action more or less immediately afterreceiving input, whereas the emotional agent may need to do lots of additional processing.However, it is important to note that the time does not flow more quickly for the control agent,and new inputs will not be available every 50 milliseconds (the Soar decision cycle). Eventsthemselves take time to play out; the exact rate has not been determined yet, but it is likely thatthe agent will have many cycles to utilize between inputs. That is, the world changes slowlyrelative to the rate at which the agent processes information. In fact, it makes evolutionary sensethat the human brain would have evolved to be just fast enough to process the inputs as it needsto. Thus, I expect the control agent to make decisions quickly but also to then waste theremaining time, whereas the emotional agent will more fully utilize the time available to it ingenerating these useful high-level features. Thus, not only will the agent learn faster, it will alsouse its time more efficiently.

This evaluation hinges on being able to use a version of Soar that includes the necessary RL andactivation features. Supposedly a research version exists that may be suitable.

5.2.2 Improving metacognitive strategies

Another area that may benefit from emotion is metacognitive strategies. By this I mean thingslike monitoring progress towards goals, planning, and determining when to give up on a goal.

The coping process supports metacognitive strategies in at least two ways. First, it allows anagent to generate new subgoals to deal with difficulties (problem-focused coping), which is aform of replanning. Second, it allows an agent to give up on unsolvable goals when itdetermines that inadequate progress is being made (emotion-focused coping). Furthermore,coping is a domain-independent framework for supporting these actions.



44

It should be possible to design a task in which an agent needs to engage in some of thesemetacognitive strategies. For example, an agent might generate new subgoals and need to giveup on some goals in order to achieve some overall goal. A control agent will have some domain-specific knowledge that allows it to do these things, but the emotional agent will be able to buildon its domain-independent knowledge; therefore, it should require less domain-specific

knowledge. Furthermore, the rules required by the emotional agent may be simpler since it cantest its feelings, whereas a non-emotional agent may have to test complex aspects of the statedirectly. It may be that, for a single task, the total knowledge required by the emotional agentwill be larger than that required by the control agent, but as the agents are expanded to new tasks,the emotional agent should scale better.

5.3 Checking Predictions

A number of predictions were made in Section 4.1.1.6 regarding properties of the appraisal andcomprehension process. A successful system should have many of these properties regardless of domain; my evaluation will include a check of these predictions against the actualimplementation with explanations of any differences. To summarize:

1) Partial comprehension: an interrupted agent should comprehend what has happened sofar. Whether this property emerges will depend on the evaluation domains.

2) Appraisals are generated in a partially fixed order. This must be the case given thecurrent theory. Exactly what the partial order is will likely evolve over the course of theresearch.

3) Different emotions will require different amounts of processing. Again, this seeminglymust be the case, but whether this property is observed will depend on what emotions aregenerated by the evaluation domains.

4) Time constraints may lead to incorrect or incomplete appraisal, leading to differentemotions or no emotion at all. Observation of this property will depend on the properties

of the evaluation domains.5) Appraisals can happen at different time scales. Again, observing this property dependson having a domain in which comprehension can change over time (i.e. one in whichthere is ambiguity).

5.4 Summary

Unfortunately, it is difficult to design these tasks in detail without first completing more designwork on the system itself (so that I can better understand the constraints). Also, it is somewhatworrisome that the first evaluation relies on reinforcement learning, a system being developed byanother graduate student. However, it is the most developed of the new Soar systems, and thereare plans to make it available internally to the other graduate students. Thus, I expect it to be

ready by the time I am ready to implement this evaluation. Additionally, the metacognitivestrategies have not been worked out in detail, so it is possible that I will discover issues thatmake it unrealistic to pursue that test. Finally, since the implementation of the theory will likelyhave an impact on the theory itself, I expect the predictions of the theory to evolve over time(more likely grow – it seems unlikely to me today that the current predictions will prove invalid).

Given the difficulties in determining what is really feasible for evaluation of my research, I mayneed to follow up with the thesis committee in the future once things become clearer. It may



45

turn out that my research will only allow for one of these evaluations, or that I will come up witha new evaluation to try. In other words, determining appropriate evaluations is research in and of itself.



46

6 PlanMy research plan is essentially to fill in the various parts of the emotion process as depicted inFigure 10. Because of the current limitations of Soar, some parts of the emotion process must befinessed or skipped altogether; most notably, I will pursue the integration of emotion with

architectural level components. However, I should be able to make significant headway in theremainder of the process, from situation comprehension to emotion response. Due to the breathof the research, it will necessarily be a “thin slice” across these various stages. I expect thesituation comprehension to be the deepest, whereas the emotion generation and representationitself may be the thinnest. Once an end-to-end system exists (i.e. an agent that can go throughthe entire cycle repeatedly to achieve goals), I will evaluate it to find whether learning andbehavior are actually enhanced.

Table 10 summarizes the open questions and whether I intend to do further research to answerthose questions. In some cases further research is not intended because it is not possible, and inother cases I have already settled on an answer.



47


Comments

How are theappraisal valuesgenerated?

Yes I will refine the situation comprehension process tosupport an appropriate range of deliberate and automaticappraisals.

What are theproper appraisaldimensions?

Yes This will be strongly influenced by what fits well into thecomprehension process (and the comprehension processwill be strongly influenced by what needs to fit in it).

A p p r a i s a l T

h e o r y

Are the emotionscategories ormodal spaces?

No I have decided to use an n-dimensional representation forthe emotions. Regions within this space correspond toemotions.

How long does anemotion last andhow is itsintensitycalculated?

Yes The intensity aspect will probably be based on some of thenumeric appraisals, whereas how long the emotion lastswill probably be finessed for now (i.e. it will last until thenext one, or some fixed number amount of time).

What does anagent feel?

No I will give the agent the n-dimensional emotion point.

What cognitivechanges should bemodeled?

No Changes at the cognitive level seem to requiremechanisms that Soar does not yet possess or moreextensive research in other areas, and thus I will not modelthese phenomena.

P o s t - a p p r a i s a l

How do actionand thought urgesfit in?

No Soar already supports the simple notion of action andthought urges, but does not currently support themechanisms that would probably be required for a moresophisticated view.

How does an

agent respond toits feelings?

Yes I will develop a feeling comprehension process that allows

an agent to understand its feelings in the context of itsunderstanding of the situation. This will allow it toeffectively cope with the situation by forming new goalsand taking actions.

R e s p o n s e s t o e m o t i o n

What responsesshould beincorporated?

Yes Deciding which responses to include will depend on thedesign of the feeling comprehension process (and viceversa). The tasks that we design for evaluation may alsosuggest an appropriate set of responses.

E v

a l u a t i o n Does emotion

improve theagent’s learning

and behavior?

Yes I will design at least one task that can be used to evaluatethe agent and answer this question. Possible candidatescurrently include a reinforcement learning task and one in

which the agent must use metacognitive strategies toachieve its primary goals.

Table 10: The thesis research plan.

In conclusion, the primary contribution of this thesis is twofold: one, I will establish a framework for future research, and two, I will demonstrate that the framework supports improved learningand behavior.



48

7 References

Bachorowski, J. (1999). Vocal expression and perception of emotion. Current Directions in

Psychological Science, 8(2):53-57.Berenbaum, H. (2002). Varieties of joy-related pleasurable activities and feelings. Cognition and

Emotion. 16 (4), 473-494.Cacioppo, J. T. et al. (2000). The psychophysiology of emotion. In M. Lewis & J. M. Haviland-

Jones (Eds.) Handbook of Emotions, (2nd ed., pp. 173-191). New York: Guilford.Cacioppo, J. T. & Gardner, W. L. (1999). Emotion. Annual Review of Psychology, 50:191-214.Citrin, L, B., Roberts, T. & Fredrickson, B.L. (2005). Objectification Theory and Emotions: A

Feminist Psychological Perspective on Gendered Affect. In L. Z. Teidens & C. W. Leach(Eds.) The social life of emotions. New York: Cambridge University Press.

Cosmides, L. & Tooby, J. (2000). Evolutionary psychology and the emotions. In M. Lewis & J.M. Haviland-Jones (Eds.) Handbook of Emotions, (2nd ed., pp. 91-115). New York: Guilford.

Damasio, A. (1994). Descartes’ error: Emotion, reason, and the human brain. New York: AvonBooks.

Damasio, A. (2003). Looking for spinoza: Joy, sorrow, and the feeling brain. USA: Harcourt.Feldman Barrett, L. (2004). Feelings or words? Understanding the content in self-report ratings

of experienced emotion. Journal of personality and social psychology, 87(2):266-281.Feldman Barrett, L., & Gross, J. J. (2001). Emotional intelligence: A process model of emotion

representation and regulation. In T. J. Mayne & G. A. Bonanno (Eds.) Emotions: Current

Issues and Future Directions (pp. 286-310). New York: Guilford.Flavell, J. H. (1979). Metacognition and cognitive monitoring. American Psychologist ,

34(10):906-911.Flavell, J. H. (1987). Speculations about the nature and development of metacognition. In F. E.

Weinert and R. N. Kluwel (Eds.) Metacognition, Motivation, and Understanding. (pp. 21-29). Hillsdale, NJ: Lawrence Erlbaum Associates.

Forgas, J. P. (1999). Network theories and beyond. In T. Dalgleish & M. Power (Eds.) Handbook of Cognition and Emotion. (pp. 591-611). Chichester, England: Wiley & Sons.

Fredrickson, B. L. & Branigan, C. (2005). Positive emotions broaden the scope of attention andthought-action repertoires. Cognition and Emotion, 19(3):313-332.

Fredrickson, B. L. & Levenson, R. W. (1998). Positive emotions speed recovery fromcardiovascular sequelae of negative emotions. Cognition and Emotion, 12(2):191-220.

Gratch, J. & Marsella, S. (2004). A Domain-independent framework for modeling emotion, Journal of Cognitive Systems Research, Volume 5, Issue 4, pp. 269-306.

Gross, J.J., & John, O.P. (2003). Individual differences in two emotion regulation processes:Implications for affect, relationships, and well-being. Journal of Personality and Social

Psychology, 85, 348-362.

Hudlicka, E. & Zacharias, G. (2004). Approaches for modeling individuals within organizationalsimulations. In R. G. Ingalls, M. D. Rossetti, J. S. Smith, and B. A. Peters (Eds.) Proceedings

of the 2004 Winter Simulation Conference.Isen, A. M. & Daubman, K. A. (1984). The influence of affect on categorization. Journal of

Personality and Social Psychology, 47(6):1206-1217.Kahneman, D. & Tversky, A. (1973). On the psychology of prediction. Psychological Review,

80(4):237-251.



49

Keltner, D. & Ekman, P. (2000). Facial expression of emotion. In M. Lewis & J. M. Haviland-Jones (Eds.) Handbook of Emotions, (2nd ed., pp. 236-249). New York: Guilford.

Laird, J. & Newell, A. (1983). A universal weak method: Summary of results. In IJCAI ’83, (pp.771–773).

Larsen, R. J. & Fredrickson, B. L. (1999). Measurement issues in emotion research. In D.

Kaheneman, E. Diener, & N. Schwarz (Eds.) Well-being: Foundations of hedonic psychology (pp. 40-60). New York: Russell Sage.Lazarus, R. S. (1982). Thoughts on the relations between emotions and cognition. American

Psychologist , 37, 1019-1024. [Reprinted in K. R. Scherer & P. Ekman (Eds). Approaches to

Emotion (pp. 247-257). Hillsdale, NJ: Lawrence Erlbaum.]Lazarus, R.S. (1991). Progress on a cognitive-motivational-relational theory of emotion.

American Psychologist , 46, 819-834.Lerner, J. S. & Keltner, D. (2001). Fear, anger, and risk. Journal of Personality and Social

Psychology, 81, 146-159.Lewis, R. L. (1993). An Architecturally-based theory of Human Sentence Comprehension. PhD

thesis. Carnegie Mellon University.

Marinier, R. and Laird, J. E. (2004). Towards a Comprehensive Computational Model of Emotions and Feelings. 6 th

International Conference on Cognitive Modeling.Nason, S. and Laird, J. E. (2004). Soar-RL: Integrating Reinforcement Learning with Soar. 6

th

International Conference on Cognitive Modeling.Neumann, R. (2001). The causal influences of attributions on emotions: A procedural priming

approach. Psychological Science, 11(3):179-182.Newell, A. 1990. Unified Theories of Cognition. Cambridge, MA: Harvard University Press.Perrez, M. & Reicherts, M. (1992). Stress, Coping, and Health: A situation-behavior approach:

Theory, methods, applications. Seattle: Hogrefe & Huber Publishers.Philippot, P. & Dozier, S. (1996). Subjective and physiological consequences of associating

emotional memories. Proceedings of the 9th

Conference of the International Society for

Research on Emotion, (pp. 155-159).Philippot, P., & Schaefer, A. (2001). Emotion and memory. In T. J. Mayne & G. A. Bonanno

(Eds.) Emotions: Current Issues and Future Directions (pp. 82-122). New York: Guilford.Roseman, I. (2001). A model of appraisal in the emotion system: Integrating theory, research,

and applications. In K.R. Scherer, A. Schorr & T. Johnstone (Eds). Appraisal Processes in

Emotion (pp. 68-91). Oxford: Oxford University Press.Roseman, I. & Smith, C. A. (2001). Appraisal theory: Overview, Assumptions, Varieties,

Controversies. In K. R. Scherer, A. Schorr, & T. Johnstone (Eds.) Appraisal processes in

emotion: Theory, Methods, Research (pp. 3-19). New York and Oxford: Oxford UniversityPress.

Rosenberg, E. L. (1998). Levels of analysis and the organization of affect. Review of General

Psychology, 2, 247-270.Rumelhart, D. E. (1980). Schemata: The building blocks of cognition. In R. J. Spiro, B. C.

Bruce, & W. F. Brewer (Eds.) Theoretical issues in reading comprehension: Perspectives

from cognitive psychology, linguistics, artificial intelligence, and education (pp. 33-58).Hillsdale, NJ: Erlbaum.

Scherer, K. R. (2001). Appraisal considered as a process of multi-level sequential checking. InK. R. Scherer, A. Schorr, & T. Johnstone (Eds.) Appraisal processes in emotion: Theory,

Methods, Research (pp. 92-120). New York and Oxford: Oxford University Press.



Smith, C.A. & Kirby, L.A. (2001). Toward delivering on the promise of appraisal theory. InK.R. Scherer, A. Schorr & T. Johnstone (Eds). Appraisal Processes in Emotion (pp. 121-138). Oxford: Oxford University Press.

Smith, C. A. & Lazarus, R. S. (1990). Emotion and adaptation. In L. A. Pervin (Ed.), Handbook

of personality theory and research (pp. 609-637). New York: Guilford.

Talmy, L. (1975). Semantics and syntax of motion. In J. P. Kimball, editor, Syntax and semanticsVol. 4 (pp. 181-238). New York: Academic Press.Zacks, J. M. & Tversky, B. (2001). Event Structure in Perception and Conception. Psychological

Bulletin, 127(1):3-21.

Date post:	02-Apr-2018
Category:	Documents
Upload:	jaggy286
View:	218 times
Download:	0 times

Marinier_thesisproposal- Unified Theory of Emotions

Documents