CS 124/LINGUIST 180 From Languages to Information 124/LINGUIST 180 From Languages to Information...

CS 124/LINGUIST 180From Languages to Information

DetectingSocialandAffectiveMeaning

DanJurafskyStanfordUniversity

Affectivemeaning

• Drawingonliteraturesin• affectivecomputing(Picard95)• linguisticsubjectivity(Wiebe andcolleagues)• socialpsychology(Pennebaker andcolleagues)

• Canwemodelthewordmeaningrelevantto:• sentiment• emotion• personality• mood• attitudes

2

Whycomputeaffectivemeaning?• Detecting:

• sentimenttowardspoliticians,products,countries,ideas• frustrationofcallerstoahelpline• stressindriversorpilots• depressionandothermedicalconditions• confusioninstudentstalkingtoe-tutors• emotionsinnovels(e.g.,forstudyingattitudestowardgroupsovertime)

• Couldwegenerate:• emotionsormoodsforliteracytutorsinthechildren’sstorybookdomain• emotionsormoodsforcomputergames• personalitiesfordialoguesystemstomatchtheuser

Scherer’stypologyofaffectivestatesEmotion:relativelybriefepisodeofsynchronizedresponseofallormostorganismicsubsystemsinresponsetotheevaluationofaneventasbeingofmajorsignificance

angry,sad,joyful,fearful,ashamed,proud,desperate

Mood:diffuseaffectstate…changeinsubjectivefeeling,oflowintensitybutrelativelylongduration,oftenwithoutapparentcause

cheerful,gloomy,irritable,listless,depressed,buoyant

Interpersonalstance:affectivestancetakentowardanotherpersoninaspecificinteraction,coloringtheinterpersonalexchange

distant,cold,warm,supportive,contemptuous

Attitudes:relativelyenduring,affectivelycoloredbeliefs,preferencespredispositionstowardsobjectsorpersons

liking,loving,hating,valuing,desiring

Personalitytraits:emotionallyladen,stablepersonalitydispositionsandbehaviortendencies,typicalforaperson

nervous,anxious,reckless,morose,hostile,envious,jealous


Reminder:SentimentLexicons











TheGeneralInquirer

• Homepage:http://www.wjh.harvard.edu/~inquirer• ListofCategories: http://www.wjh.harvard.edu/~inquirer/homecat.htm

• Spreadsheet:http://www.wjh.harvard.edu/~inquirer/inquirerbasic.xls• Categories:

• Positiv (1915words)andNegativ (2291words)• Strongvs Weak,Activevs Passive,OverstatedversusUnderstated• Pleasure,Pain,Virtue,Vice,Motivation,CognitiveOrientation,etc

• FreeforResearchUse

PhilipJ.Stone,DexterCDunphy,MarshallS.Smith,DanielM.Ogilvie.1966.TheGeneralInquirer:AComputerApproachtoContentAnalysis.MITPress

LIWC(LinguisticInquiryandWordCount)Pennebaker,J.W.,Booth,R.J.,&Francis,M.E.(2007).LinguisticInquiryandWordCount:LIWC2007.Austin,TX

• Homepage:http://www.liwc.net/• 2300words,>70classes• AffectiveProcesses

• negativeemotion(bad,weird,hate,problem,tough)• positiveemotion(love,nice,sweet)

• CognitiveProcesses• Tentative(maybe,perhaps,guess),Inhibition(block,constraint)

• Pronouns,Negation(no,never),Quantifiers(few,many)• $30or$90fee

MPQASubjectivityCuesLexicon

• Homepage:http://mpqa.cs.pitt.edu/lexicons/• 6885wordsfrom8221lemmas

• 2718positive• 4912negative

• Eachwordannotatedforintensity(strong,weak)• GNUGPL9

Theresa Wilson, Janyce Wiebe, and Paul Hoffmann (2005). Recognizing Contextual Polarity in Phrase-Level Sentiment Analysis. Proc. of HLT-EMNLP-2005.

Riloff and Wiebe (2003). Learning extraction patterns for subjective expressions. EMNLP-2003.

type=weaksubj len=1word1=abandoned pos1=adj stemmed1=npriorpolarity=negativetype=weaksubj len=1word1=abandonment pos1=nounstemmed1=npriorpolarity=negativetype=weaksubj len=1word1=abandon pos1=verbstemmed1=ypriorpolarity=negativetype=strongsubj len=1word1=abase pos1=verbstemmed1=ypriorpolarity=negativetype=strongsubj len=1word1=abasement pos1=anypos stemmed1=ypriorpolarity=negativetype=strongsubj len=1word1=abash pos1=verbstemmed1=ypriorpolarity=negativetype=weaksubj len=1word1=abate pos1=verbstemmed1=ypriorpolarity=negative

10

BingLiuOpinionLexicon

• BingLiu'sPageonOpinionMining• http://www.cs.uic.edu/~liub/FBS/opinion-lexicon-English.rar

• 6786words• 2006positive• 4783negative

11

Minqing HuandBingLiu.MiningandSummarizingCustomerReviews.ACMSIGKDD-2004.


SentimentLexicons


Emotion











Twofamiliesoftheoriesofemotion

• Atomicbasicemotions• Afinitelistof6or8,fromwhichothersaregenerated

• Dimensionsofemotion• Valence(positivenegative)• Arousal(strong,weak)• Control

15

Ekman’s6basicemotions:Surprise,happiness,anger,fear,disgust,sadness

Valence/ArousalDimensions

Higharousal,lowpleasure Higharousal,highpleasureanger excitement

Lowarousal,lowpleasureLowarousal,highpleasuresadness relaxation

arou

sal

valence

Atomicunitsvs.Dimensions

Distinctive• Emotionsareunits.• Limitednumberofbasic

emotions.• Basicemotionsareinnateand

universal

Dimensional• Emotionsaredimensions.• Limited#oflabelsbut

unlimitednumberofemotions.

• Emotionsareculturallylearned.

AdaptedfromJuliaBraverman

Oneemotionlexiconfromeachparadigm!

1. 8basicemotions:• NRCWord-EmotionAssociationLexicon(MohammadandTurney 2011)

2. Dimensionsofvalence/arousal/dominance• Warriner,A.B., Kuperman,V.,andBrysbaert,M.(2013)

• BothbuiltusingAmazonMechanicalTurk

19

Plutchick’s wheelofemotion

20

• 8basicemotions• infouropposingpairs:• joy–sadness• anger–fear• trust–disgust• anticipation–surprise

NRCWord-EmotionAssociationLexicon

21

MohammadandTurney 2011

• 10,000wordschosenmainlyfromearlierlexicons• LabeledbyAmazonMechanicalTurk• 5Turkers perhit• GiveTurkers anideaoftherelevantsenseoftheword• Result:

amazingly anger 0amazingly anticipation 0amazingly disgust 0amazingly fear 0amazingly joy 1amazingly sadness 0amazingly surprise 1amazingly trust 0amazingly negative 0amazingly positive 1

TheAMTHit

22 …

Lexiconofvalence,arousal,anddominance

• Warriner,A.B., Kuperman,V.,andBrysbaert,M.(2013). Normsofvalence,arousal,anddominancefor13,915Englishlemmas. BehaviorResearchMethods45,1191-1207.

• Supplementarydata: Thisworkislicensedundera CreativeCommonsAttribution-NonCommercial-NoDerivs3.0UnportedLicense.

• Ratingsfor14,000wordsforemotionaldimensions:• valence (thepleasantnessofthestimulus)• arousal (theintensityofemotionprovokedbythestimulus)• dominance (thedegreeofcontrolexertedbythestimulus)

23

Lexiconofvalence,arousal,anddominance• valence (thepleasantnessofthestimulus)

9:happy,pleased,satisfied,contented,hopeful1:unhappy,annoyed,unsatisfied,melancholic,despaired,orbored

• arousal (theintensityofemotionprovokedbythestimulus)9:stimulated,excited,frenzied,jittery,wide-awake,oraroused1:relaxed,calm,sluggish,dull,sleepy,orunaroused;

• dominance (thedegreeofcontrolexertedbythestimulus)9:incontrol,influential,important,dominant,autonomous,orcontrolling1:controlled,influenced,cared-for,awed,submissive,orguided

• AgainproducedbyAMT

24

Lexiconofvalence,arousal,anddominance:Examples

Valence Arousal Dominancevacation 8.53 rampage 7.56 self 7.74happy 8.47 tornado 7.45 incredible 7.74whistle 5.7 zucchini 4.18 skillet 5.33conscious 5.53 dressy 4.15 concur 5.29torture 1.4 dull 1.67 earthquake 2.14

25


OtherUsefulLexicons

Concretenessversusabstractness• Thedegreetowhichtheconceptdenotedbyawordreferstoaperceptibleentity.

• Doconcreteandabstractwordsdifferinconnotation?• Storageandretrieval?• Bilingualprocessing?• Relevantforembodiedviewofcognition(Barsalou 1999interalia)

• Doconcretewordsactivatebrainregionsinvolvedinrelevantperception

• Brysbaert,M.,Warriner,A.B.,and Kuperman,V.(2014) Concretenessratingsfor40thousandgenerallyknownEnglishwordlemmas BehaviorResearchMethods46,904-911.

• Supplementarydata: Thisworkislicensedundera CreativeCommonsAttribution-NonCommercial-NoDerivs3.0UnportedLicense.

• 37,058Englishwordsand2,896two-wordexpressions(“zebracrossing”and“zoomin”),

• Ratingfrom1(abstract)to5(concrete)• Calibratorwords:• shirt,infinity,gas,grasshopper,marriage,kick,polite,whistle,theory,andsugar27

Concretenessversusabstractness• Brysbaert,M.,Warriner,A.B.,and Kuperman,V.(2014) Concretenessratingsfor40thousand

generallyknownEnglishwordlemmas BehaviorResearchMethods46,904-911.• Supplementarydata: Thisworkislicensedundera CreativeCommonsAttribution-NonCommercial-NoDerivs3.0UnportedLicense.

• Someexampleratingsfromthefinaldatasetof40,000wordsandphrasesbanana 5bathrobe 5bagel 5brisk 2.5badass 2.5basically 1.32belief 1.19although 1.07

28

PerceptualStrengthNorms

ConnellandLynott norms

29

However, when we examined the original norming instructions used to collect these norms, we found it questionable that participants would have simultaneously considered their sensory experience across all modalities and then managed to aggregate this experience into a single, composite rating per word. Instructions for concreteness ratings, for example, define concrete words as referring to “objects, materials, or persons” and abstract words as referring to something that “ cannot be experienced by the senses” (Paivio, Yuille & Madigan, 1968, p. 5). The resulting ratings, therefore, may reflect different decision criteria at the concrete and abstract ends of the scale, which is consistent with previous observations that the concreteness ratings scale has a bimodal distribution (e.g., Kousta et al., 2011). Imageability ratings are frequently used interchangeably with concreteness ratings (e.g., Binder et al., 2005; Sabsevitz et al., 2005) because of their high correlation and theoretical relationship in dual coding theory. Instructions for imageability ratings repeatedly refer to arousing a “mental image” (Paivio et al., 1968, p. 4), which is likely to lead naïve participants to focus on vision at the expense of other modalities. Both concreteness and imageability ratings could therefore add considerable noise to any dataset that assumed the ratings reflected a smooth continuum of perceptual experience across all modalities.

Our goals in the present paper were twofold. First, we aimed to establish whether concreteness and imageability norms actually reflect the degree with which concepts are perceptually experienced, as is commonly assumed. Second, we examined whether so-called concreteness effects in word processing are better predicted by concreteness/imageability ratings or by strength of perceptual experience. If the former, then forty years of empirical methodology have been validated but the reasons for null and reverse concreteness effects remain unclear. If the latter, then concreteness and imageability ratings are unsuitable for the tasks in which they are employed, and null and reverse concreteness effects are due to the unreliability of perceptual information in these ratings.

Experiment 1

Rather than ask participants to condense their estimations of sensory experience into a single concreteness or imageability rating, modality-specific norming asks people to rate how strongly they experience a variety of concepts using each perceptual modality in turn (i.e., auditory, gustatory, haptic, olfactory or visual: Lynott & Connell, 2009, in prep.; see also Connell & Lynott, 2010; Louwerse

& Connell, 2011).

If concreteness and imageability are a fair reflection of the degree of perceptual information in a concept, then ratings of perceptual strength in all five modalities should be positively related to concreteness and imageability ratings, and these relationships should remain consistent across the rating scale. On the other hand, if we were correct in our hypothesis to the contrary, then we would expect some perceptual modalities to be neglected (i.e., no relationship) or even misinterpreted (i.e., negative relationship) in concreteness and imageability ratings. Specifically, concreteness norming instructions may have led to different decision criteria and therefore distinctly different modality profiles at each end of scale, whereas imageability instructions may have led to a predominantly visual bias.

Method

Materials A total of 592 words were collated that represented the overlap of the relevant sets of norms, so each word had ratings of perceptual strength on five modalities as well as concreteness and imageability (see Table 1 for sample items). Perceptual strength norms came from Lynott and Connell (2009, in prep.), in which participants were asked to rate “to what extent do you experience WORD” (for nouns) or “to what extent do you experience something being WORD” (for adjectives) through each of the five senses (i.e., “by hearing”, “by tasting”, “by feeling through touch”, “by smelling” and “by seeing”), using separate rating scales for each modality. Perceptual strength ratings therefore took the form of a 5-value vector per word, ranging from 0 (low strength) to 5 (high strength). Concreteness ratings were taken from the MRC psycholinguistic database for 522 words, with ratings for the remaining 70 words coming from Nelson, McEvoy and Schreiber (2004). Imageability ratings for 524 words also came from the MRC database, and were supplemented with ratings for a further 68 words from Clark and Paivio (2004). All concreteness and imageability ratings emerged from the same instructions as Paivio et al.'s (1968) original norms, and ranged from 100 (abstract or low-imageability) to 700 (concrete or high-imageability).

Design & Analysis We ran stepwise regression analyses with either concreteness or imageability rating as the dependent variable, and ratings of auditory, gustatory, haptic, olfactory and visual strength as competing predictors. For analysis of consistency across the scales, each dependent variable was split at its midpoint before

Table 1: Sample words, used in Experiments 1 and 2, for which perceptual strength ratings [0-5] match or mismatch ratings

of concreteness and imageability [100-700].

Perceptual strength

Word Auditory Gustatory Haptic Olfactory Visual Concreteness Imageability

soap 0.35 1.29 4.12 4.00 4.06 589 600

noisy 4.95 0.05 0.29 0.05 1.67 293 138

atom 1.00 0.63 0.94 0.50 1.38 481 499

republic 0.53 0.67 0.27 0.07 1.79 376 356

1429

Microsoft Excel Worksheet


Usingthelexiconstodetectaffect

Lexiconsfordetectingdocumentaffect:Simplestunsupervisedmethod

• Sentiment:• Sumtheweightsofeachpositivewordinthedocument• Sumtheweightsofeachnegativewordinthedocument• Choosewhichevervalue(positiveornegative)hashighersum

• Emotion:• Dothesameforeachemotionlexicon

31

Lexiconsfordetectingdocumentaffect:Simplestsupervisedmethod

• Buildaclassifier• Predictsentiment(oremotion,orpersonality)givenfeatures• Use“countsoflexiconcategories”asafeatures• Samplefeatures:• LIWCcategory“cognition”hadcountof7• NRCEmotioncategory“anticipation”hadcountof2

• Baseline• Insteadusecountsofall thewordsandbigramsinthetrainingset• Thisishardtobeat• Butonlyworksifthetrainingandtestsetsareverysimilar32


Prosody

Prosody

• Threecharacteristicsofspeech:• pitch• energy• duration/rate-of-speech

• Thatplayaroleinconveyingmeaning• Andespeciallysocialmeaning

USC’sSAILLabShriNarayananandDaniByrd

Happy

Sad

37

Larynx and Vocal Folds• TheLarynx(voicebox)

• Astructuremadeofcartilageandmuscle• Locatedabovethetrachea(windpipe)andbelowthepharynx(throat)• Containsthevocalfolds• (adjectiveforlarynx:laryngeal)

• VocalFolds(olderterm:vocalcords)• Twobandsofmuscleandtissueinthelarynx• Canbesetinmotiontoproducesound(voicing)

Text from slides by Sharon Rose

The larynx, external structure, from front

Figure thanks to John Coleman!!

Vertical slice through larynx, as seen from back

Figure thnx to John Coleman!!

Voicing:•Aircomesupfromlungs•Forcesitswaythroughvocalcords,pushingopen(2,3,4)•Thiscausesairpressureinglottistofall,since:

• whengasrunsthroughconstrictedpassage,itsvelocityincreases(Venturi tubeeffect)• thisincreaseinvelocityresultsinadropinpressure(Bernoulliprinciple)

•Becauseofdropinpressure,vocalcordssnaptogetheragain(6-10)•Singlecycle:~1/100ofasecond.

Figure & text from John Coleman’s web site

Voicelessness

• Whenvocalcordsareopen,airpassesthroughunobstructed

• Voicelesssounds:p/t/k/s/f/sh/th/ch• Iftheairmovesveryquickly,theturbulencecausesadifferentkindofphonation:whisper

Vocal folds open during breathing

From Mark Liberman’s web site, from Ultimate Visual Dictionary

Vocal Fold Vibration

UCLA Phonetics Lab Demo

Soundwavesarelongitudinalwaves

Dan Russell Figure

particle dispacment

pressure

Dan Russell Figure

RememberHighSchoolPhysicsSimplePeriodWaves(sinewaves)

Time (s)0 0.02

–0.99

0.99

0• Characterized by:• period: T• amplitude A• phase f

• Fundamental frequencyin cycles per second, or Hz

• F0=1/T1 cycle

Simpleperiodicwaves

• Computingthefrequencyofawave:• 5cyclesin.5seconds=10cycles/second=10Hz

• Amplitude:• 1

• Equation:• Y=Asin(2pft)

Speechsoundwaves

• Alittlepiecefromthewaveformofthevowel[iy]• Xaxis:time.• Yaxis:

• Amplitude=airpressureatthattime• +:compression• 0:normalairpressure,• -:rarefaction

Backtowaves:Fundamentalfrequency

• Waveformofthevowel[iy]

• Frequency:10repetitions/.03875seconds=258Hz• Thisisspeedthatvocalfoldsmove,hencevoicing• Eachpeakcorrespondstoanopeningofthevocalfolds• Thelowfrequencyofthecomplexwaveiscalledthe

fundamentalfrequencyofthewaveorF0

.03875 0 Time in seconds

F0(informally:pitch)

•

WecancomputeF0mean,max,minforeachturnAndthestandarddeviationacrossturns

Intensity

Wecancomputeintensitymean,max,minforeachturnAndthestandarddeviationacrossturns


Prosody


ProsodyforEmotionDetection

Actedspeech:EmotionalProsodySpeechandTranscriptsCorpus(EPSaT)

• RecordingsfromLDC• http://www.ldc.upenn.edu/Catalog/LDC2002S28.html

• 8actorsreadshortdatesandnumbersin15emotionalstyles

SlidefromJacksonLiscombe

EPSaTExampleshappysadangryconfidentfrustratedfriendlyinterested


anxiousboredencouraging

Task1

• Binaryclassification• Detecttheemotiontheactorwasinstructedtouse

MajorProblemsforClassification:DifferentValence/DifferentActivation

slide from Julia Hirschberg

But….DifferentValence/SameActivation

slide from Julia Hirschberg

Extractingaudiofeatures

• Mostcommonlyused:OpenSmile• http://www.audeering.com/research/opensmile

“Speech&MusicInterpretationbyLarge-spaceExtraction”

• Praat

Scherersummary:Prosodicfeaturesforemotion


ProsodyforEmotion


Personalitydetection

Sampleaffectivetask:personalitydetection

64











Personality

• Theinternalstructuresandpropensitiesthatexplainaperson’scharacteristicpatternsofthought,emotion,andbehavior.

• Personalitycaptureswhatpeoplearelike.

McGraw-Hill/Irwin Chapter 9

67

TheBigFiveDimensionsofPersonality

Extraversionvs.Introversionsociable,assertive,playfulvs.aloof,reserved,shy

Emotionalstabilityvs.Neuroticismcalm,unemotionalvs.insecure,anxious

Agreeablenessvs.Disagreeablefriendly,cooperativevs.antagonistic,faultfinding

Conscientiousnessvs.Unconscientiousself-disciplined,organised vs.inefficient,careless

Opennesstoexperienceintellectual,insightfulvs.shallow,unimaginative

BigFivePersonality:Agreeableness

warm,kind,cooperative,sympathetic,helpful,andcourteous.• Strongdesiretoobtainacceptanceinpersonalrelationshipsasameansofexpressingpersonality.

• Agreeablepeoplefocuson“gettingalong,”notnecessarily“gettingahead.”


BigFivePersonality:Extraversion

talkative,sociable,passionate,assertive,bold,anddominant• Easiesttojudgeimmediatelyonfirstmeeting• Prioritizedesiretoobtainpowerandinfluencewithinasocial

structureasameansofexpressingpersonality.• Highinpositiveaffectivity— atendencytoexperiencepleasant,

engagingmoodssuchasenthusiasm,excitement,andelation.


BigFivePersonality:Neuroticism• experienceunpleasantmoods:hostility,nervousness,and

annoyance.• morelikelytoappraiseday-to-daysituationsasstressful.• lesslikelytobelievetheycancopewiththestressorsthatthey

experience.• relatedtolocusofcontrol (attributecausesofeventsto

themselvesortotheexternalenvironment)• Neurotics:externallocusofcontrol:believethattheeventsthatoccuraroundthemaredrivenbyluck,chance,orfate.

• lessneuroticpeopleholdinternallocusofcontrol:believethattheirownbehaviordictatesevents. McGraw-Hill/Irwin Chapter 9

ExternalandInternalLocusofControl


BigFivePersonality:OpennesstoExperience

curious,imaginative,creative,complex,sophisticated• Alsocalled“Inquisitiveness”or“Intellectualness”• highlevelsofcreativity,thecapacitytogeneratenoveland

usefulideasandsolutions.• Highlyopenindividualsaremorelikelytomigrateintoartistic

andscientificfields.


ChangesinBigFiveDimensionsOvertheLifeSpan


Aside:DoAnimalsHavePersonalities?

• Gosling(1998)studiedspottedhyenas.• 4humanobserversrated44personalitytraitsofhyenas• RanPCAontheratings• Fivedimensions:Assertiveness,Excitability,Human-DirectedAgreeableness,Sociability,andCuriosity

• Relatedto3humandimensions:neuroticism(excitability),openness(curiosity),agreeableness(sociability+agree)

TaketheBigFiveInventory

http://www.outofservice.com/bigfive/

Varioustextcorporalabeledforpersonalityofauthor

Pennebaker,JamesW.,andLauraA.King.1999."Linguisticstyles:languageuseasanindividualdifference."Journalofpersonalityandsocialpsychology 77,no.6.

• 2,479essaysfrompsychologystudents(1.9millionwords),“writewhatevercomesintoyourmind”for20minutes

Mehl,MatthiasR,SDGosling,JWPennebaker.2006.Personalityinitsnaturalhabitat:manifestationsandimplicitfolktheoriesofpersonalityindailylife.Journalofpersonalityandsocialpsychology90(5),862

• SpeechfromElectronicallyActivatedRecorder(EAR)• Randomsnippetsofconversationrecorded,transcribed• 96participants,totalof97,468wordsand15,269utterances

Schwartz,H.Andrew,JohannesC.Eichstaedt,MargaretL.Kern,LukaszDziurzynski,StephanieM.Ramones,Megha Agrawal,AchalShahetal.2013."Personality,gender,andageinthelanguageofsocialmedia:Theopen-vocabularyapproach."PloS one 8,no.9

• Facebook• 75,000volunteers• 309millionwords• Alltookapersonalitytest

Ears(speech)corpus(Mehl etal.)

Essayscorpus(Pennebaker andKing)

Classifiers

• Mairesse,François,MarilynA.Walker,MatthiasR.Mehl,andRogerK.Moore."Usinglinguisticcuesfortheautomaticrecognitionofpersonalityinconversationandtext."Journalofartificialintelligenceresearch (2007):457-500.• Variousclassifiers,lexicon-basedandprosodicfeatures

• Schwartz,H.Andrew,JohannesC.Eichstaedt,MargaretL.Kern,LukaszDziurzynski,StephanieM.Ramones,Megha Agrawal,Achal Shahetal.2013."Personality,gender,andageinthelanguageofsocialmedia:Theopen-vocabularyapproach."PloS one 8,no.• regressionandSVM,lexicon-basedandall-words

79

SampleLIWCFeaturesLIWC (Linguistic Inquiry and Word Count)

Pennebaker, J.W., Booth, R.J., & Francis, M.E. (2007). Linguistic Inquiry and Word Count: LIWC 2007. Austin, TX

Dialogactofutterance

Labeledbyparsingeachutteranceandthenusingheuristicrulesbasedonparsetree:

Commands:imperatives,“canyou”,etc.Backchannels:yeah,ok,uh-huh,huhQuestionsAssertions (anythingelse)

Prosodicfeatures

ComputedviaPraat

pitch (mean,min,max,sd):intensity (mean,min,max,sd)voicedtimerateofspeech(words/second)

NormalizingLIWCcategoryfeatures(Schwartzetal2013,Facebookstudy)

83

• Mairesse:RawLIWCcounts

• Schwartzetal:Normalizedperwriter:

Sampleresults• Agreeable:

• +Family,+Home,-Anger,-Swear

• Extravert• +Friend,+Religion,+Self

• Conscientiousness:• -Swear,-Anger,-NegEmotion,

• EmotionalStability:• -NegEmotion,+Sports,

• Openness• -Cause,-Space

84

Decisiontreeforpredictingextraversioninessaycorpus(Mairesse etal)Mairesse, Walker, Mehl & Moore

Words per sentence

Familiarity

Up

Positive emotions

Grooming

17.91 > 17.91

> 599.7

> 1.66

> 0.11

> 0.64 0.64

599.7

1.66

0.11

Introvert

Extravert

Introvert

Introvert

Introvert

Extravert

Apostrophes

2.57 > 2.57

Achievement

> 1.52 1.52

ExtravertIntrovert

Sadness

> 1.44 1.44

Extravert Introvert

Parentheses

> 0.64 0.64

Introvert

Sexuality

Articles 7.23 > 7.23

> 0.12

Introvert

2.57 > 2.57

0.12

Figure 1: J48 decision tree for binary classification of extraversion, based on the essayscorpus and self-reports.

Remarkably, we can see that the LIWC features outperform the MRC features for everytrait, and the LIWC features on their own always perform slightly better than the fullfeature set. This clearly suggests that MRC features aren’t as helpful as the LIWC featuresfor classifying personality from written text, however Table 13 shows that they can stilloutperform the baseline for four traits out of five.

Concerning the algorithms, we find that AdaboostM1 performs the best for extraversion(56.3% correct classifications), while SMO produces the best models for all other traits. Itsuggests that support vector machines are promising for modelling personality in general.The easiest trait to model is still openness to experience, with 62.5% accuracy using LIWCfeatures only.

4.2 EAR Corpus

Classification accuracies for the EAR corpus are in Table 14. We find that extraversion isthe easiest trait to model using observer reports, with both Naive Bayes and AdaboostM1

476

85

Featureanalysis:ObservedExtraversion

morewordshigherpitchmoreconcrete,imageable wordsgreatervariationinintensitygreatermeanintensitymorewordrepetitions

M5’RegressionTree

UsingallwordsinsteadoflexiconsFacebookstudy

Schwartzetal.(2013)• Choosingphraseswithpmi >2*length[inwords]

• Onlyusewords/phrasesusedbyatleast1%ofwriters• Normalizecountsofwordsandphrasesbywriter

87

Facebookstudy,Learnedwords,ExtraversionversusIntroversion

Facebookstudy,LearnedwordsNeuroticismversusEmotionalStability

89

EvaluatingSchwartzetal(2013)FacebookClassifier

• Trainonlabeledtrainingdata• LIWCcategorycounts• wordsandphrases(n-gramsofsize1to3,passingacollocationfilter

• Testedonaheld-outset• Correlationswithhumanlabels

• LIWC.21-.29• AllWords.29-.41

90

Whataboutpredictingpersonalityfromwhatsomeonelikes?Youyou Wu,MichalKosinski,andDavidStillwell."Computer-basedpersonalityjudgmentsaremoreaccuratethanthosemadebyhumans."ProceedingsoftheNationalAcademyofSciences (2015)

1. 86,220volunteersfilledina100-itemquestionnaire• InternationalPersonalityItemPool(IPIP)Five-FactorModelofpersonality

2. TheythenusedFacebooklikestopredictpersonalityfor70,520people

3. Andaskedtheirfriendstoanswera10-itemquestionnairefor17,622people

Wuetal.Algorithm:

judged by two friends. A diagram illustrating the methods ispresented in Fig. 1.

ResultsSelf-Other Agreement. The primary criterion of judgmental accuracyis self-other agreement: the extent to which an external judgmentagrees with the target’s self-rating (17), usually operationalized asa Pearson product-moment correlation. Self-other agreement wasdetermined by correlating participants’ scores with the judgmentsmade by humans and computer models (Fig. 1). Since self-otheragreement varies greatly with the length and context of the re-lationship (18, 19), we further compared our results with thosepreviously published in a meta-analysis by Connely and Ones (20),including estimates for different categories of human judges:friends, spouses, family members, cohabitants, and work colleagues.To account for the questionnaires’ measurement error, self-

other agreement estimates were disattenuated using scales’Cronbach’s α reliability coefficients. The measurement error ofthe computer model was assumed to be 0, resulting in the lower(conservative) estimates of self-other agreement for computer-based judgments. Also, disattenuation allowed for direct com-parisons of human self-other agreement with those reported byConnely and Ones (20), which followed the same procedure.The results presented in Fig. 2 show that computers’ average

accuracy across the Big Five traits (red line) steadily grows withthe number of Likes available on the participant’s profile (x axis).Computer models need only 100 Likes to outperform an averagehuman judge in the present sample (r = 0.49; blue point).†

Compared with the accuracy of various human judges reportedin the meta-analysis (20), computer models need 10, 70, 150, and300 Likes, respectively, to outperform an average work col-league, cohabitant or friend, family member, and spouse (graypoints). Detailed results for human judges can be found inTable S1.How accurate is the computer, given an average person? Our

recent estimate of an average number of Likes per individual is227 (95% CI = 224, 230),‡ and the expected computer accuracyfor this number of Likes equals r = 0.56. This accuracy is sig-nificantly better than that of an average human judge (z = 3.68,P < 0.001) and comparable with an average spouse, the best ofhuman judges (r = 0.58, z = −1.68, P = 0.09). The peak computerperformance observed in this study reached r = 0.66 for partic-ipants with more than 500 Likes. The approximately log-linearrelationship between the number of Likes and computer accu-racy, shown in Fig. 2, suggests that increasing the amount ofsignal beyond what was available in this study could further boostthe accuracy, although gains are expected to be diminishing.Why are Likes diagnostic of personality? Exploring the Likes

most predictive of a given trait shows that they represent activ-ities, attitudes, and preferences highly aligned with the Big Fivetheory. For example, participants with high openness to experi-ence tend to like Salvador Dalí, meditation, or TED talks; par-ticipants with high extraversion tend to like partying, Snookie(reality show star), or dancing.Self-other agreement estimates for individual Big Five traits

(Fig. 2) reveal that the Likes-based models are more diagnostic of

Fig. 1. Methodology used to obtain computer-based judgments and estimate the self-other agreement. Participants and their Likes are represented as a matrix,where entries are set to 1 if there exists an association between a participant and a Like and 0 otherwise (second panel). The matrix is used to fit five LASSO linearregression models (16), one for each self-rated Big Five personality trait (third panel). A 10-fold cross-validation is applied to avoid overfitting: the sample israndomly divided into 10 equal-sized subsets; 9 subsets are used to train the model (step 1), which is then applied to the remaining subset to predict the per-sonality score (step 2). This procedure is repeated 10 times to predict personality for the entire sample. The models are built on participants having at least 20Likes. To estimate the accuracy achievable with less than 20 Likes, we applied the regression models to random subsets of 1–19 Likes for all participants.

†This figure is very close to the average human accuracy (r = 0.48) found in Connelly andOnes’s meta-analysis (20).

‡ Estimate based on a 2014 sample of n = 100,001 Facebook users collected for a separateproject. Sample used in this study was recorded in the years 2009–2012.

Youyou et al. PNAS | January 27, 2015 | vol. 112 | no. 4 | 1037

PSYC

HOLO

GICALAND

COGNITIVESC

IENCE

S

Results:

some traits than of others. Especially high accuracy was observedfor openness—a trait known to be otherwise hard to judge due tolow observability (21, 22). This finding is consistent with previousfindings showing that strangers’ personality judgments, based ondigital footprints such as the contents of personal websites (23),are especially accurate in the case of openness. As openness islargely expressed through individuals’ interests, preferences, andvalues, we argue that the digital environment provides a wealth ofrelevant clues presented in a highly observable way.Interestingly, it seems that human and computer judgments

capture distinct components of personality. Table S2 lists cor-relations and partial correlations (all disattenuated) betweenself-ratings, computer judgments, and human judgments, basedon a subsample of participants (n = 1,919) for whom bothcomputer and human judgments were available. The averageconsensus between computer and human judgments (r = 0.37) isrelatively high, but it is mostly driven by their correlations withself-ratings, as represented by the low partial correlations (r =0.07) between computer and human judgments. Substantialpartial correlations between self-ratings and both computer (r =0.38) and human judgments (r = 0.42) suggest that computer andhuman judgments each provide unique information.

Interjudge Agreement. Another indication of the judgment accu-racy, interjudge agreement, builds on the notion that two judgesthat agree with each other are more likely to be accurate thanthose that do not (3, 24–26).The interjudge agreement for humans was computed using

a subsample of 14,410 participants judged by two friends. As thejudgments were aggregated (averaged) on collection (i.e., we didnot store judgments separately for the judges), a formula was usedto compute their intercorrelation (SI Text). Interjudge agreement

for computer models was estimated by randomly splitting theLikes into two halves and developing two separate models fol-lowing the procedure described in the previous section.The average consensus between computer models, expressed

as the Pearson product-moment correlation across the Big Fivetraits (r = 0.62), was much higher than the estimate for humanjudges observed in this study (r = 0.38, z = 36.8, P < 0.001) or inthe meta-analysis (20) (r = 0.41, z = 41.99, P < 0.001). All resultswere corrected for attenuation.

External Validity. The third measure of judgment accuracy, ex-ternal validity, focuses on how well a judgment predicts externalcriteria, such as real-life behavior, behaviorally related traits, andlife outcomes (3). Participants’ self-rated personality scores, aswell as humans’ and computers’ judgments, were entered intoregression models (linear or logistic for continuous and di-chotomous variables respectively) to predict 13 life outcomesand traits previously shown to be related to personality: lifesatisfaction, depression, political orientation, self-monitoring,impulsivity, values, sensational interests, field of study, substanceuse, physical health, social network characteristics, and Face-book activities (see Table S3 for detailed descriptions). The ac-curacy of those predictions, or external validity, is expressed asPearson product-moment correlations for continuous variables,or area under the receiver-operating characteristic curve (AUC)for dichotomous variables.§As shown in Fig. 3, the external validity of the computer

judgments was higher than that of human judges in 12 of the 13

Fig. 2. Computer-based personality judgment accuracy (y axis), plotted against the number of Likes available for prediction (x axis). The red line representsthe average accuracy (correlation) of computers’ judgment across the five personality traits. The five-trait average accuracy of human judgments is positionedonto the computer accuracy curve. For example, the accuracy of an average human individual (r = 0.49) is matched by that of the computer models based onaround 90–100 Likes. The computer accuracy curves are smoothed using a LOWESS approach. The gray ribbon represents the 95% CI. Accuracy was averagedusing Fisher’s r-to-z transformation.

§AUC is an equivalent of the probability of correctly classifying two randomly selectedparticipants, one from each class, such as liberal vs. conservative political views. Note thatfor dichotomous variables, the random guessing baseline corresponds to an AUC = 0.50.

1038 | www.pnas.org/cgi/doi/10.1073/pnas.1418680112 Youyou et al.

SummaryonPersonalityDetection

• Fromtextandspeech• Notasolvedtask• Textandprosodicfeaturesbothsomewhatuseful• Especiallyhardtoextractself-labeledpersonality• Especiallyhardtodetectopenness

• Fromlikes:• Seemstobeaneasiertask• Computer-basedjudgmentofpersonality(r=0.56)correlatesmorestronglywithself-ratingsthanaveragehumanjudgmentsdo(r=0.49)


DetectingPersonality


Othertasks:Detectingstudentuncertaintyanddisinterestforonlineeducation

Detectingstudentconfusioninatutoringsystem

ITSpoke:IntelligentTutoringSpokenDialogueSystemDianeLitman,KatherineForbes-Riley,ScottSilliman,MihaiRotaru• JacksonLiscombe,JuliaHirschberg,JenniferJ.Venditti.2005.Detecting

CertainnessinSpokenTutorialDialogues• Forbes-Riley,Kate,andDianeLitman."Benefitsandchallengesofreal-time

uncertaintydetectionandadaptationinaspokendialoguecomputertutor."SpeechCommunication53,no.9(2011):1115-1136.

Tutorialcorpus:howcertainisthestudent

• 151dialoguesfrom17subjects• studentfirstwritesanessay,thendiscusseswithtutor

• botharerecordedwithmicrophones• manuallytranscribedandsegmentedintoturns• 6778studentutterances(average2.3seconds)• eachutterancehand-labeledforcertainty

JacksonLiscombe,JuliaHirschberg,JenniferJ.Venditti.2005.DetectingCertainnessinSpokenTutorialDialogues

UncertaintyinITSpoke


[71-67-1:92-113]

um <sigh> I don’t even think I have anidea here ...... now .. mass isn’t weight...... mass is ................ the ..........space that an object takes up ........ isthat mass?





Doesithelp?AtutorialsystemthatadaptstouncertaintyForbes-Riley,Kate,andDianeLitman."Benefitsandchallengesofreal-timeuncertaintydetectionandadaptationinaspokendialoguecomputertutor."SpeechCommunication53,no.9(2011):1115-1136.

ager, where states represent tutor questions and responsesand transitions between states represent student answervalues. The dialogue manager determines the semantic clas-sification and correctness value for each student answerusing Levenshtein distance between the answer string anda set of strings corresponding to semantic concepts repre-senting anticipated correct and incorrect answers to thecurrent tutor question. For instance, the semantic conceptdownward is the semantic classification for answer stringssuch as “down”, “towards earth”, etc. There are 51 differ-ent semantic concepts shared among the 142 different tutorstates; the set of answer strings which represent eachsemantic concept were assembled from prior ITSPOKEcorpora. If an answer string is successfully classified as ananticipated semantic concept, it receives the correctnessvalue associated with that concept, otherwise, the answerstring is labeled as incorrect.

In non-adaptive ITSPOKE, the next tutor state dependsonly on the correctness value of the student answer. Inuncertainty-adaptive ITSPOKE, the next tutor statedepends on both the correctness and uncertainty value ofthe student answer, where the uncertainty value is deter-mined by prosodic features and other features of the stu-dent answer (see Sections 4 and 5).

Finally, the tutor response produced by the dialoguemanager is sent to the Cepstral text-to-speech synthesis sys-tem8 and played to the student through the headphone.The tutor text is also displayed on the web interface, asshown in Fig. 2.

Earlier implementations of the non-adaptive ITSPOKEsystem have been used in prior experiments as a platformto compare human-computer versus human-human spokendialogue tutoring, spoken versus typed dialogue computertutoring, and synthesized versus pre-recorded tutor speech(Litman et al., 2006; Forbes-Riley et al., 2006). In theseprior experiments, significant overall learning has beenassociated with ITSPOKE tutoring (as measured by pretestand posttest before and after system use, respectively).

However, there is still significant room for improvement;for example, the average student posttest score after usingthe ITSPOKE system has never exceeded 75% (out of100%).

4. UNC-ITSPOKE: automatically adapting to uncertainty

The automatic uncertainty adaptation used in thisexperiment was based on the hypothesis that student learn-ing could be significantly improved by detecting and adapt-ing to student uncertainty during the computer tutoring. Inparticular, the adaptation was motivated by research thatviews uncertainty as well as incorrectness as a signal of a“learning impasse”, i.e., as an opportunity for the studentto better learn the material about which s/he is uncertainor incorrect; this research further suggests that experienc-ing uncertainty can motivate the student to take an activerole in constructing a better understanding of the materialbeing tutored (e.g. VanLehn et al., 2003; Kort et al., 2001).

We further observed that to be motivated to resolve alearning impasse, the student must first perceive that itexists. Incorrectness and uncertainty differ in this percep-tion. Incorrectness simply signals the student has reachedan impasse, while uncertainty signals the student perceivess/he has reached an impasse. Based on this, we associatedeach combination of binary uncertainty (UNC,CER) andcorrectness (INC,COR) with an “impasse severity”, as inFig. 6. COR + CER corresponds to a state where a studentis not experiencing an impasse, since s/he is correct and notuncertain about. INC + CER corresponds to a state wherea student is experiencing the most severe type of impasse,since s/he is incorrect and not aware of it. INC + UNCand COR + UNC answers indicate impasses of lesserseverity: the student is incorrect but aware s/he may be,and the student is correct but uncertain if s/he is, respec-tively. In (Forbes-Riley et al., 2008) we show empirical sup-port for distinguishing impasse severities.

We hypothesized that student learning would increase ifUNC-ITSPOKE would provide additional substantivecontent to remediate all learning impasse states. As dis-cussed in Section 3, both types of incorrectness impasse

Fig. 5. ITSPOKE architecture.

8 The Cepstral system is a commercial outgrowth of the Festival system(Black, 1997).

1120 K. Forbes-Riley, D. Litman / Speech Communication 53 (2011) 1115–1136

GivemoreinformationifthestudentisuncertainForbes-Riley,Kate,andDianeLitman."Benefitsandchallengesofreal-timeuncertaintydetectionandadaptationinaspokendialoguecomputertutor."SpeechCommunication53,no.9(2011):1115-1136.

Featuresforuncertainty

Conclusions

• Uncertaintyisveryhardtodetect• F-scoreof.27

• Evenso,usingtheuncertaintydetectorimprovedlearneroutcomesabitovernotusingit.

• Butneedbetterdetectionofuncertainty,andalsobetterdetectionofcorrectanswers.

DetectingdisengagementKateForbes-Riley,DianeLitman,HeatherFriedberg,JoannaDrummond.2012.IntrinsicandExtrinsicEvaluationofanAutomaticUserDisengagementDetectorforanUncertainty-AdaptiveSpokenDialogueSystem.NAACL2012.

DisengagementFeatures

Mostimportantfeature:Pausepriortostartofturn<250msmeansdisengagement

Doesdisengagementdetectionhelpthesystem?

YesButnotforallstudents

Onlyformalestudents(improvedtheirperformancesignificantly)

Litman,Diane,andKateForbes-Riley."EvaluatingaSpokenDialogueSystemthatDetectsandAdaptstoUserAffectiveStates."In15thAnnualMeetingoftheSpecialInterestGrouponDiscourseandDialogue,p.181.2014


InterpersonalStanceDetection











AutomaticallyExtractingSocialMeaningfromSpeedDates

RajeshRanganath,DanJurafsky,andDanielA.McFarland.2013.Detectingfriendly,flirtatious,awkward,andassertivespeechinspeed-dates.ComputerSpeechandLanguage27:1,89-115.

McFarland,Daniel,DanJurafsky,andCraigM.Rawlings.2013."MakingtheConnection:SocialBondinginCourtshipSituations.”AmericanJournalofSociology.

Detectingstance

10004-minutespeeddatesSubjectslabeledselves andeachotherfor

• friendly(eachonascaleof1-10)• awkward• flirtatious• assertive

Linguisticfeaturesweexamined

• Words:• HEDGES:kindof,sortof,alittle,Idon’tknow,Iguess• NEGEMOTION:bad,weird,crazy,problem,tough,awkward,boring• LOVE:love,loved,loving,passion,passions,passionate• WORK:research,advisor,lab,work,finish,PhD,department• I: I,me,mine,my,myself,you,your,yours,etc.

• Prosody• pitchceiling,pitchfloor,energy,rateofspeech

Dialogactfeatures

• ClarificationquestionsWhat?Excuseme?

• Laughter[Beginningofturn][Endofturn]

• AppreciationsAwesome!That’samazing!Oh,great

• SympathyThatsoundsterrible!That’sawful!Thatsucks!

Positiveandnegativeassessments

Sympathy(that’s|thatis|thatseems|itis|thatsounds)(very|really|alittle|sortof)?(terrible|awful|weird|sucks|aproblem|tough|toobad)

Appreciations(“Positivefeedback”)

(Oh)?(Awesome|Great|Allright|Man|Nokidding|wow|mygod)

That(‘s|is|sounds|wouldbe)(so|really)?(great|funny|good|interesting|neat|amazing|nice|notbad|fun)

(Goodwin, 1996; Goodwin and Goodwin, 1987; Jurafsky et al., 1998)

Interruptions

A:Notnecessarily.Imeanithappenstonotnecessarilybemything,butthereareplentyof--B:No,no,Iunderstandyourpoint.

Model

• Multinomiallogisticregressionclassifier• Predictwhetheraconversationsideislabeled

flirt/friendly/assertive/awkward• Givenlinguisticfeatures

• Thenlookatthefeatureweights

Linguisticsignsofawkwardness

• Awkwardmenandwomenusemorehedges• kindof,sortof,alittle

• Peoplewhoaresouncomfortableinthedate• Soinneedofdistancingthemselves

• Thattheycan’tevencommittotheirsentence.

Whatmakessomeoneseemfriendly?“Collaborativeconversationalstyle”

• Friendlypeople:• laughatthemselves• don’tusenegativeemotions

• Friendlymen• aresympatheticandagreemoreoften• don’tinterrupt• don’tusehedges

• Friendlywomen:• highermaxpitch• laughattheirdate

Whatdoflirtersdo?

Men:� raisepitchfloor� laughturn-initially(at

theirdate,teasing?)� say“you”� don’tusewords

relatedtoacademics

Women:� raisepitchceiling� laughturn-finally

(atthemselves?)� say“I”� usenegation(don’t,

no,not)

Unlikelywordsformaleflirting

• academia• interview• teacher• phd• advisor• lab• research• management• finish

Howconsistentlydopeoplereadeachotherslinguisticcues?

Eachdaterlabeledthemselves• Iflirted(from1-10)• Iwasfriendly(from1-10)• Iwasawkward(from1-10)• Iwasassertive(from1-10)

Andlabeledtheirpartner• Mydateflirted(from1-10)• …

Howmuchdotheselabelsagree?• IfIthoughtIwasfriendly,doesmydateagree?

Notatall

Notatall!

Peopledisagreewiththeirdateaboutstance:

Male is flirting (1-10)Male101 says: 8

Female127 says: 1

Why?

Peopleassumetheirdateisbehavinglikethemselves:

Maleisflirting

Femaleisflirting

Male101 says: 8 7

Female127 says: 1 1

CorrelationsbetweenmybehaviorandwhatIsayaboutmydate

I labelmyselfx Ilabeldate

Ilabeldatex datelabelsthemselves

Flirting .73 .15

Friendly .77 .05

Awkward .58 .07

Assertive .58 .09

Wearenotverygoodatmodelingeachother’sintentions

• Atleastnotin4minutes• Speakersinsteadbasetheirjudgmentsontheirownbehaviororintentions

FunctionofInterruptions?

Theory1:Control:Usedbymentotakethefloor(ZimmermanandWest1975;West1985)Theory2:Sharedmeaning,alignment,engagement:(Tannen 1994;Coates1996,1997),collaborativefloor(Edelsky 1981)

Wefound:interruptionsarejointconstruction(“collaborativecompletions”)

• aturnwhereaspeakercompletestheutterancebegunbythealter(Lerner,1991;Lerner,1996).

Soareyoualmost--

Onmywayout,yeah.

Orshowingsharedunderstanding

Female:Ididn’tusedtolikeitbutnowI’m—Male:Ohsameforme.…

Otherfactorsthatmightmatter• Height• BMI(Bodymassindex)• Foreign-born• Datingexperience• Lookingforrelationship?• Orderofdateinevening• Metbefore• Agedifference• Hobbysimilarities

Thesefactorsdoinfluencestance

• Morelikelytoflirt:• high-BMImenorwomen• tallermenandshorterwomen• menlaterintheevening

• Morelikelytobeflirtedwith• low-BMIwomen• high-BMImen

• Bigger(taller,heavier)mensaytheyaremoreassertive

Butlanguagestillmatters

Language Traits Language+Traits

Maleflirt

66% 64 72

Femaleflirt

74 55 76

accuracy(baselineis50%)

ConclusionsfromDating

• Wearenotverygoodatreadingintentions

• Howfriendlypeoplesound• besympathetic,askclarificationquestions,agree,accommodate

• Howtodate:• Don’ttalkaboutyouradvisor• Wecandetectthesewithfairaccuracywithrelativelysimplelinguisticfeatures


InterpersonalStanceDetection


Summary











Date post:	13-Apr-2018
Category:	Documents
Upload:	lamtuyen
View:	214 times
Download:	1 times

CS 124/LINGUIST 180 From Languages to Information 124/LINGUIST 180 From Languages to Information...

Documents