Post on 13-Jul-2020
transcript
Social Networks:Analyzing Social Information in Deep Convolutional Neural Networks Trained for Face Identification
SocialTraits• Humansmakesocialtraitinferencesfromfacesreadily[1]andrapidly[2]• Traitinferencespredictimportantdecisions(e.g.,votingpreferences)[3,4]• Socialtraitscanbegeneratedfrommodelsoffacestructureandreflectance[5,6]
IdentityDescriptors
Introduction&Goals
Results
0.344
0.245
0.253
0.261
0.253
0.394
0.107
0.19
0.125
0.165
0.303Anxious
Artistic
Assertive
Careless
Efficient
Impulsive
Lazy
Quiet
Shy
Talkative
Warm
0.00 0.25 0.50 0.75 1.00Coefficient of Determination
WarmTalkative
ShyQuietLazy
ImpulsiveEfficientCarelessAssertiveArtisticAnxious
DCNNsmodeledafterprimatevisualcortex
• Networkusedinthisstudycontains6convolutionallayers,3fullyconnectedlayers[9]
• State-of-the-artperformanceonchallenging,unconstrainedIJB-Adataset[10]
ConnorJ.Parde1,YingHu1,CarlosCastillo2,SwamiSankaranarayanan2,andAliceJ.O’Toole11TheUniversityofTexasatDallas,2UniversityofMaryland
TalkativeEnergeticWarmShyQuiet
SympatheticAssertiveHelpfulArtistic
AnxiousTrusting
Soft-heartedEfficientThoroughCarelessImpulsiveReliableLazy
Collectedratingsfor18SocialTraits
11UniqueDimensions
Averagedhighlycorrelatedtraits:• talkative,energetic• warm,sympathetic,soft-
hearted,trusting,helpful,reliable
• efficient,thorough
SocialTraitRatings
Humanratingsofsocialtraitsforfaces• 280faceimages• 18traitsfromBigFiveFactorsofPersonality[9]• 20setsofratingsperface• responsesaveragedacrossparticipants
AcknowledgementsThisresearchisbaseduponworksupportedbytheOfficeoftheDirectorofNationalIntelligence(ODNI),IntelligenceAdvancedResearchProjectsActivity(IARPA),viaIARPAR&DContractNo.2014-14071600012.Theviewsandconclusionscontainedhereinarethoseoftheauthorsandshouldnotbeinterpretedasnecessarilyrepresentingtheofficialpoliciesorendorsements,eitherexpressedorimplied,oftheODNI,IARPA,ortheU.S.Government.TheU.S.GovernmentisauthorizedtoreproduceanddistributereprintsforGovernmentalpurposesnotwithstandinganycopyrightannotationthereon.
Goal1:Measuresimilaritybetweenhumanand
computertraitpredictionsmade
fromidentity-trainedDCNNs
Goal3:Predictindividual
socialtraitinferencesfromtop-levelDCNNfeatures
Participants:• n =80(60female)• Meanage=21
Stimuli:• 280images,194identities• 204female,76male• Caucasian• neutralexpression• Ratingscollectedforfront-facing images
• N xK “featurematrix”obtainedfromDCNN• N xM “traitmatrix”obtainedfromaveraged
participantresponses• Predicttraitmatrixfromfeaturematrixusing
linearregression
<K features>
<Nfaces>
<M traits>
<Nfaces>
Learne
dWeigh
tMatrix
W
W
• RemoveL features(n =140)withlowlearnedweights,re-trainmodel• Keeponlyfeaturesimportantfortraitprediction
• Columnsinthefinaltraitmatrixarecomputerpredictionsofcolumnsfromoriginaldatamatrix
<K- L features>
<Nfaces>
<M traits>
<Nfaces>
New
Weigh
tMatrix
W’ ’
W’
VerifyStructureofFaceTraitSpace(e.g.[5])
• principalcomponentanalysisofhumantraitratings• created“traitspace”
• 2significantprincipalcomponents:• 1st componentinterpretedasapproachability• 2nd componentinterpretedasdominance
• Predictionsmadefromnon-frontalDCNNfeatures
DCNNsforFaceIdentification• State-of-the-artforfaceidentification[7]andgeneralizeoverviewpoint,illumination,etc.• “Top-level”DCNNsfeaturesretainnon-identityinformation(e.g.,pose,imagequality)[8]• Doface-identificationfeaturesalso retainsocialtraitinformation?
DCNNsmodeledafterprimatevisualcortex• EarlylayersmodelV1-V4,finallayersmodelITcortex• Forfaceidentification,finalDCNNlayerstores
abstractidentitycode<- facerepresentation
Trait-ProfilePredictions
IndividualTraitPredictions
PredictSocialTraitInferences
• Similaritybetweenhuman-generatedandcomputer-predictedtraitvectorsmeasuredusingcosinedistance
• AccuracyofindividualtraitpredictionsmeasuredusingR2 betweenhuman-generatedandcomputer-predictedvalues
• Errorbetweenhumanratingsandpredictedtraits,plottedagainstanulldistribution• Alltraitspredictedsignificantlyabovechance• Blueline:α=0.002• Redline:predictedvalue
• Cosinesimilaritybetweenhuman-generatedtraitprofilesandcomputerpredictions:
Goal2:Measureaccuracyoftraitpredictions
usingDCNNfeaturesfromnon-frontal
images
0
10
20
30
0.00 0.25 0.50 0.75 1.00Prediction Error
coun
t
"Talkative"
0
10
20
30
0.00 0.25 0.50 0.75 1.00Prediction Error
coun
t
"Warm"
0
10
20
30
0.00 0.25 0.50 0.75 1.00Prediction Error
coun
t
"Shy"
0
10
20
30
40
0.00 0.25 0.50 0.75 1.00Prediction Error
coun
t
"Quiet"
0
10
20
30
0.00 0.25 0.50 0.75 1.00Prediction Error
coun
t
"Assertive"
0
10
20
30
40
0.00 0.25 0.50 0.75 1.00Prediction Error
coun
t
"Artistic"
0
10
20
30
40
0.00 0.25 0.50 0.75 1.00Prediction Error
coun
t
"Anxious"
0
10
20
30
0.00 0.25 0.50 0.75 1.00Prediction Error
coun
t
"Efficient"
0
10
20
30
0.00 0.25 0.50 0.75 1.00Prediction Error
coun
t
"Careless"
0
10
20
30
0.00 0.25 0.50 0.75 1.00Prediction Error
coun
t
"Impulsive"
0
10
20
30
40
0.00 0.25 0.50 0.75 1.00Prediction Error
coun
t
"Lazy"
Trait-PredictionError
R2 BetweenHumanInferencesandComputerPredictions
• Differenttraitspredictedtodifferentextents
• Alltraitinferencespredictedabovechance
References[1]Bruce,V.,&Young,A.(1986).Understandingfacerecognition.Britishjournalofpsychology,77(3),305-327.[2]Bar,M.,Neta,M.,&Linz,H.(2006).Veryfirstimpressions.Emotion,6(2),269.[3]Todorov,A.,Mandisodza,A.N.,Goren,A.,&Hall,C.C.(2005).Inferencesofcompetencefromfacespredictelectionoutcomes.Science,308(5728),1623-1626.[4]Rule,N.O.Ambady,N.(2008).Thefaceofsuccess: Inferencesfromchiefexecutiveofficers'appearancepredictcompanyprofits.PsychologicalScience:AJournaloftheAmericanPsychologicalSociety/APS,19,109–111.[5]Oosterhof,N.N.,&Todorov,A.(2008).Thefunctionalbasisoffaceevaluation.ProceedingsoftheNationalAcademyofSciences,105(32),11087-11092.[6]Walker,M.,&Vetter,T.(2009).Portraitsmadetomeasure:Manipulatingsocialjudgmentsabout individualswithastatisticalfacemodel.JournalofVision,9(11),12-12.[7]Taigman,Y.,Yang,M.,Ranzato,M.A.,&Wolf,L.(2014).Deepface:Closingthegaptohuman-levelperformanceinfaceverification.InProceedingsoftheIEEEconferenceoncomputervisionandpatternrecognition (pp.1701-1708).[8]Parde,C.J.,Castillo,C.,Hill,M.Q.,Colon,Y.I.,Sankaranarayanan,S.,Chen,J.C.,&O’Toole,A.J.(2017,May).FaceandImageRepresentationinDeepCNNFeatures.InAutomaticFace&GestureRecognition(FG2017),201712thIEEEInternationalConferenceon (pp.673-680).IEEE.[9]Gosling,S.D.,Rentfrow,P.J.,&SwannJr,W.B.(2003).AverybriefmeasureoftheBig-Fivepersonalitydomains.JournalofResearchinpersonality,37(6),504-528.[10]Sankaranarayanan,S.,Alavi,A.,Castillo,C.D.,&Chellappa,R.(2016,September).Tripletprobabilisticembeddingforfaceverificationandclustering.InBiometricsTheory,ApplicationsandSystems(BTAS),2016IEEE8thInternationalConferenceon (pp.1-8).IEEE.
Conclusions
Humantraitinferencescanbepredictedfromthe
top-levelfeaturesofaDCNNtrainedforface
identification
Conclusion1
Traitinferencesassignedtofrontal
facescanbepredictedfromDCNNfeaturesgeneratedforbothfrontalandnon-frontalfaces
Conclusion2
Top-levelDCNNfeaturesforface
identificationretainrobusttrait
representation– eachindividualtraitpredictedabove
chance
Conclusion3
Nulldistribution:𝝰 =arccos(0.078)
UsingKfeatures:𝝰 =arccos(0.353)
UsingK– L features:𝝰 =arccos(0.533)
• DCNNrepresentationallowsforstate-of-the-artidentification• Notindependentofimageinformation,socialtraits