Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy’s National Nuclear Security Administration under contract DE-AC04-94AL85000. Brett Bader*, Richard Harshman** & Tamara Kolda* *Sandia National Laboratories **University of Western Ontario Workshop for Algorithms on Modern Massive Data Sets June 24, 2006 Analysis of Latent Relationships in Semantic Graphs using DEDICOM
Transcript
Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy’s National Nuclear Security Administration
under contract DE-AC04-94AL85000.
Brett Bader*, Richard Harshman** & Tamara Kolda**Sandia National Laboratories**University of Western Ontario
Workshop for Algorithms on Modern Massive Data SetsJune 24, 2006
Analysis of Latent Relationships in Semantic Graphs using DEDICOM
Common Graph Analysis Technique
Uk
Vk
Adjacencymatrix
ΣkT
Best rank-k matrix filters out noise and captures “latent” information, which improves
certain data mining tasks
But we may have ignored critical informationby not considering edge metadata!
+ + ...Third dimension offers more explanatory power: uncovers new
latent information and reveals subtle relationships
Build an “adjacency tensor” such that there is an adjacency matrix for each edge type.
DEDICOM
PARAFAC
Multilinearalgebra
Adjacencymatrix
Adjacencytensor
Objective
Use DEDICOM to analyze a semantic graph of email communications
changing over time
David
Ellen
Bob
Frank
Alice Carl
IngridHenk
Gary
role
s
time patterns
3-way DEDICOM=
DEDICOM
• DEcomposition into DIrectional COMponents
• Introduced in 1978 by Harshman
• Past applications- Study asymmetries in telephone calls among cities- Marketing research• car switching: car owners and what they buy next • free associations of words- words to describe hair in advertising shampoo:
“body” evokes “fullness” more often than “fullness” evokes “body”
- Asymmetric measures of world trade (import/export)
• A (n x p) is an orthogonal matrix of loadings or weights• R (p x p) is a dense matrix that captures asymmetric relationships
• Decomposition is not unique- A can be transformed with no loss of fit to the data- Nonsingular transformation Q:
- Usually “fix” A with some standard rotation (e.g., VARIMAX)
X ! ARAT
=X AR AT
s.t. A orthogonal
ARAT = (AQ)(Q!1RQ!T )(AQ)T
n
n p
n
Single domain model
New Algorithm
Anew !
!
X XT"
#
!
R RT"
#
AT 0
0 AT
$$†
Anew =!
XART + XTAR
" !
R(ATA)RT + RT (ATA)R"
!1.
Solving for A:
or
Solving for R:
Stack data and model “side by side” in a single equation!
X XT"
=!
ARAT
ARTAT
"
= A
#
!
R RT"
#
AT 0
0 AT
$$
...and solve least-squares problem:
Rnew = A†X(AT )†
= AY ZT
minA
!
!
!
Y ! AZT
!
!
!
2
F
Three-way DEDICOM
Xi = ADiRDiAT + Ei for i = 1, . . . ,m,
=
minA,R,D
m!
i=1
"
"Xi ! ADiRDiAT
"
"
2
F
• A (n x p) is a matrix of loadings or weights (not necessarily orthogonal)• R (p x p) is a dense matrix that captures asymmetric relationships• D (p x p x m) is a tensor with diagonal frontal slices giving the weights
of the columns of A for each slice in third mode
• *Unique* solution with enough slices of X with sufficient variation- i.e., no rotation of A possible- greater confidence in interpretation of results
n
nm
A
ATRD D
n
p
New Algorithm - Updating A
!
X1 XT1 · · · Xm XT
m
"
= A!
D1RD1 D1RTD1 · · · DmRDm DmRTDm
" !
I2m ! AT"
A =
!
m"
i=1
#
XiADiRTDi + XT
i ADiRDi
$
% !
m"
i=1
(Bi + Ci)
%
!1
Bi ! DiRDi(ATA)DiR
TDi,
Ci ! DiRTDi(A
TA)DiRDi.
Solving for A:
where
= AY
minA,R,D
m!
i=1
"
"Xi ! ADiRDiAT
"
"
2
F
ZT
A = YZ(ZTZ)!1
New Algorithm - Updating D
minDi
!
!Xi ! ADiRDiAT
!
!
2
F
A = QA,
minDi
!
!
!
QTXiQ ! ADiRDiAT
!
!
!
2
F
gk = !
!
i,j
"
2(X ! ADRDAT ) " (ADrka
Tk + akrk,:DA
T )#
i,j
Solving for D:
Use compressionQR factorization:
Use Newton’s method to solve the optimization problem for
Smaller problem (p x p)
hst = !2!
i,j
"
(X ! ADRDAT ) " (asrsta
Tt + atrtsa
Ts )
! (ADrsaTs + asrs:DA
T ) " (ADrtaTt + atrt:DA
T )#
i,j
dnew = d ! H!1g
d = diag(Di)
Gradient:
Hessian:
Our Algorithm - Updating R
Solving for R:
f(R) =
!
!
!
!
!
!
!
"
#
$
Vec(X1)...
Vec(Xm)
%
&
'!
"
#
$
AD1 " AD1
...ADm " A Dm
%
&
'Vec(R)
!
!
!
!
!
!
!
Vec(R) =
!
m"
i=1
(DiATADi) ! (DiA
TADi)
#
!1 m"
i=1
Vec(DiATXiADi)
minimize:
Use the approach in (Kiers, 1993)
minR
m!
i=1
"
"Xi ! ADiR DiAT
"
"
2
F
Algorithm Costs
ATA
XiART
XT
i AR
QTXiQ
QR factorization of AO(p2
n)
Xi
Dominant costs:
Updating A is most expensive part
linear in nnz of
Application: Enron Email Analysis
• Links consist of email communications
• What can we learn about this network strictly from their communication patterns? (Social network analysis)
• U.S. corporation involved with creating energy markets- 7th largest by revenue• EnronOnline: e-trading business- natural gas- electric power
• Investigations- U.S. Federal Energy Regulatory Commission (FERC)• energy market manipulation• involved energy traders- U.S. Securities and Exchange Commission (SEC)• accounting fraud• insider trading
Enron Email Data
• FERC collected email of ~150 employees as evidence- Included emails saved in inbox, sent items, deleted
items, and all other folders
• Released to the public in 2002 by FERC as part of their investigation- To/from, date, subject, body- Attachments and some names/emails removed- Approx. 500,000 email messages
Smaller Enron Data Set
N D 99 F M A M J J A S O N D 00 F M A M J J A S O N D 01 F M A M J J A S O N D 02 F M A M J0
500
1000
1500
2000
2500
3000
3500
Month
Messages
Figure 1: Number of emails per month in the Enron email graph.
biasing from prolific emailers. Other weightings are possibleas well.
An obvious di!culty in dealing with the Enron corpusis the lack of information regarding the former employees.Without access to a corporate directory or organizationalchart at Enron at the time of these emails, it is di!cult toascertain the validity of our results and assess the perfor-mance of the DEDICOM model. Other researchers usingthe Enron corpus have had this same problem, and informa-tion on the participants has been collected and slowly madeavailable.
The Priebe data set [32] provided partial information onthe 184 employees of the small Enron network, which ap-pears to be based largely on information collected by Shettyand Adibi [36]. It provides most employees’ position andbusiness unit. To facilitate a better analysis of the DEDI-COM results, we collected extra information on the partic-ipants from the email messages themselves. We searchedfor corroborating information of the preexisting data or fornew identification information, such as title, business unit,or manager to help analyze our results. We also collectedsome relevant information posted on the FERC website [9].
5. EXPERIMENTAL RESULTSIn this section we summarize our findings of applying two-
way and three-way DEDICOM on the Enron email network.Our algorithms were written in MATLAB, using sparse ex-tensions of the Tensor Toolbox [2].
Table 1 shows the A and R matrices for a single decompo-sition (p = 3) of the two-way DEDICOM model. The largeadjacency matrix X, showing nonsymmetric relations amongemployees at Enron, related by flows of email, is condensedinto a smaller matrix R giving the same kind of asymmetricrelations but among “types” or abstract idealized individ-uals. In this case, the relations among elements in R areexchanges of email. The latent components are patterns ofthe same kind of flow as among the surface objects, justabstracted into a “higher level” summary of patterns.
DEDICOM does not actually identify clusters, except inspecial circumstances when such clusters happen to exist inthe data as we are partially seeing in the Enron data. Thecomponents or patterns of asymmetric relationships that itidentifies have loadings in A that are continuously-valued,like factor loadings, rather than discrete cluster membershipassignments.
Here, DEDICOM describes the employees by the di"erentlatent dimensions. The first factor (a1) describes an execu-
tive role that fits many of the top executives. The secondfactor (a2) describes a legal role, and the third factor (a3)describes a pipeline employee.
The R matrices show that most of the communication isamong employees that share the same role, as evidenced bythe large diagonal values in R. We do see some asymmetriccommunication. The entries in the lower triangular por-tion are typically larger than the corresponding transposeentry in the upper triangular. This suggests that slightlymore communication “flows up” the management chain than“down.”
As a point of reference, we compute the singular valuedecomposition X = U#V T . Table 1 shows the first threecolumns of the left singular vectors (U matrix) and rightsingular vectors (V matrix). Because X is nearly symmetric,the left and right singular vectors are nearly the same. Anydi"erences between U and V indicate whether the person ismore likely to send mail (U) or receive mail (V).
The SVD solution is somewhat similar to the DEDICOMmodel. Many of the same people are identified and weightedsimilarly by DEDICOM and SVD. However, there are manymore negative entries in SVD than in DEDICOM. The DEDI-COM model also provides directional information betweenthe latent groups in the R matrix that the SVD does notshow.
Table 2 shows the A and R matrices for three instances(p = 2, 3, 4) of the three-way DEDICOM model. The 2-dimensional solution groups the employees largely from thelegal department and those executives dealing with govern-ment and regulatory a"airs. The 3-dimensional solutionadds a another role of top executives, and the 4-dimensionalsolution includes those from the pipeline business in a fourthrole.
The aggregate communication patterns over the 44 monthsamong these 2-4 groups is summarized in the R matrix. Inthe 2-dimensional solution we see that most of the com-munication is within each group as evidenced by the largediagonal elements and small o"-diagonal elements. The 3-dimensional solution shows some communication betweenthe government/regulatory a"airs people and other seniorVP’s (dimensions 2 and 3, respectively). However, the com-munication is substantially asymmetric in that the r2,3 ele-ment is larger than r3,2. This indicates that the VP’s weremostly recipients of messages while the government/regulatorya"airs employees were senders. With the addition of thepipeline employees in the 4-dimensional solution, we see thatthey interact almost exclusively with themselves due to the
Email communications at Enron (1998-2002)
34,427 emails among 184 employees over 44 months
• Limited information on the 184 employees
• No org chart
We used a smaller data set prepared by Priebe et al.
DEDICOM Experiment
• Aggregate communications- Sparse matrix of size 184 x 184 (3007 nonzeros)
• Time series of communication graphs - Sparse tensor of size 184 x 184 x 44 (9838 nonzeros)
• Weighted adjacency matrix - scaling: x number of messages scaled by log(x)+1- other common choices give similar results
• Models:- SVD- 2-way DEDICOM- 3-way DEDICOM
Social Network Analysis
Communication graph among employees over
all times
patterns
• Description of employees by their roles• Aggregate communication patterns among roles
DEDICOM
Adjacencymatrix
role
s
DEDICOM Results
LegalExecutives
Pipeline
patternsro
les
Some employees have dual rolesPattern of communications in R matrix
Table 2: Three-way DEDICOM results on the Enron email graph for three di!erent decompositions, p = 2, 3, 4.The top 10 entries from all reported columns of A are listed in the table. Entries exceeding a threshold of0.06 are highlighted.
Table 3: DtRDt matrices showing communicationpatterns for October, 2000 and October, 2001.
that it identifies some people who were pretty much purelyof a certain type and other people who had mixed charac-teristics. For example, a given person might “load” on bothan executive and a lawyer component or aspect, and thusshow email exchanges resembling each of these two roles tosome extent.
The entries in matrix R describe the communication pat-terns between groups of the same and di!erent type. They
show how a particular person’s combination of roles or at-tributes influences the pattern of messages he/she exchangeswith particular other employees given the other employee’sroles or attributes. The R matrix is asymmetric and of-fers an idealized version of a directed graph involving thecomponents identified in A.
In addition, three-way DEDICOM shows the associatedcommunication patterns over time in the tensor D. Thescales in each Dt show the strength of participation of aparticular group for time period t.
In the present study, we investigated a semantic graphwith edges labeled by time. As an alternative to time, wepoint out that our semantic graph could have incorporateddi!erent types of communication media (e.g., email, phone,and mail communications) instead of time in the third mode.Then an analysis with three-way DEDICOM would repre-sent information about the vertices across all forms of com-munication (appropriately scaled by slices of D) in the Aand R matrices.
Furthermore, DEDICOM is not limited to the analysis ofsociometric and intercommunication data; DEDICOM may
Table 2: Three-way DEDICOM results on the Enron email graph for three di!erent decompositions, p = 2, 3, 4.The top 10 entries from all reported columns of A are listed in the table. Entries exceeding a threshold of0.06 are highlighted.
Table 3: DtRDt matrices showing communicationpatterns for October, 2000 and October, 2001.
that it identifies some people who were pretty much purelyof a certain type and other people who had mixed charac-teristics. For example, a given person might “load” on bothan executive and a lawyer component or aspect, and thusshow email exchanges resembling each of these two roles tosome extent.
The entries in matrix R describe the communication pat-terns between groups of the same and di!erent type. They
show how a particular person’s combination of roles or at-tributes influences the pattern of messages he/she exchangeswith particular other employees given the other employee’sroles or attributes. The R matrix is asymmetric and of-fers an idealized version of a directed graph involving thecomponents identified in A.
In addition, three-way DEDICOM shows the associatedcommunication patterns over time in the tensor D. Thescales in each Dt show the strength of participation of aparticular group for time period t.
In the present study, we investigated a semantic graphwith edges labeled by time. As an alternative to time, wepoint out that our semantic graph could have incorporateddi!erent types of communication media (e.g., email, phone,and mail communications) instead of time in the third mode.Then an analysis with three-way DEDICOM would repre-sent information about the vertices across all forms of com-munication (appropriately scaled by slices of D) in the Aand R matrices.
Furthermore, DEDICOM is not limited to the analysis ofsociometric and intercommunication data; DEDICOM may
Table 2: Three-way DEDICOM results on the Enron email graph for three di!erent decompositions, p = 2, 3, 4.The top 10 entries from all reported columns of A are listed in the table. Entries exceeding a threshold of0.06 are highlighted.
Table 3: DtRDt matrices showing communicationpatterns for October, 2000 and October, 2001.
that it identifies some people who were pretty much purelyof a certain type and other people who had mixed charac-teristics. For example, a given person might “load” on bothan executive and a lawyer component or aspect, and thusshow email exchanges resembling each of these two roles tosome extent.
The entries in matrix R describe the communication pat-terns between groups of the same and di!erent type. They
show how a particular person’s combination of roles or at-tributes influences the pattern of messages he/she exchangeswith particular other employees given the other employee’sroles or attributes. The R matrix is asymmetric and of-fers an idealized version of a directed graph involving thecomponents identified in A.
In addition, three-way DEDICOM shows the associatedcommunication patterns over time in the tensor D. Thescales in each Dt show the strength of participation of aparticular group for time period t.
In the present study, we investigated a semantic graphwith edges labeled by time. As an alternative to time, wepoint out that our semantic graph could have incorporateddi!erent types of communication media (e.g., email, phone,and mail communications) instead of time in the third mode.Then an analysis with three-way DEDICOM would repre-sent information about the vertices across all forms of com-munication (appropriately scaled by slices of D) in the Aand R matrices.
Furthermore, DEDICOM is not limited to the analysis ofsociometric and intercommunication data; DEDICOM may
LegalGovernment & regulatory affairs
Trade executivesPipeline employees
135.470.7
• Mostly communication within roles• Some large exchanges• Negative values complicates interpretation- Non-negative factorization being investigated
Temporal Patterns
N D 99 F M A M J J A S O N D 00 F M A M J J A S O N D 01 F M A M J J A S O N D 02 F M A M J0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
Month
No
rma
lize
d s
ca
le
Group 1
Group 2
Group 3
Group 4
Figure 2: Scales in D indicate the strength of participation of each group’s communication over time.
derive useful information from any directed graph. New pos-sibilities include analyzing a network of web tra!c betweenservers over time or perhaps a web/citation graph, whereedges convey authority among vertices. A third mode enterswhen the 2-way data are categorized by time, demographic,click number, or some other feature of the data.
Finally, we suggest a few extensions to the DEDICOMmodel and its application in data mining that we intend topursue. First, constrained DEDICOM [23] is an extensionof DEDICOM that has been suggested in the 90’s and pur-sued more recently. The idea is to put constraints on theA factors themselves so that the columns of A lie in a pre-scribed column space. For example, in the email graph, onemight want to impose a constraint on the first column ofA so that it contains only the top executives. Many othervariations are possible. This procedure allows for includingdomain knowledge or incorporating human understandinginto the problem. Kiers and Takane [23] o"ered an algorithmfor handling di"erent subspace constraints on A. More re-cently, Rocci [33] proposed a new algorithm for fitting anyconstrained DEDICOM model.
Second, a nonnegative factorization of DEDICOM, whereA and/or R are nonnegative, would preserve the non-negativityof the data, which could be desirable in some domains andapplications.
Finally, DEDICOM has been applied to skew-symmetricdata [17] and has yielded some benefits. There might beways to apply this technique to semantic graphs as well.
7. REFERENCES[1] E. Acar, S. A. Camtepe, M. S. Krishnamoorthy, and
B. Yener. Modeling and multiway analysis ofchatroom tensors. In Intelligence and SecurityInformatics: IEEE Intl. Conf. on Intelligence andSecurity Informatics, ISI 2005, volume 3495 of LectureNotes in Computer Science, pages 256–268. SpringerVerlag, 2005.
[2] B. W. Bader and T. G. Kolda. MATLAB tensorclasses for fast algorithm prototyping. TechnicalReport SAND2004-5187, Sandia NationalLaboratories, Albquerque, NM 87185 and Livermore,CA 94550, Oct. 2004. Submitted to ACM Trans.Math. Software.
[3] M. W. Berry and M. Browne. Email surveillance usingnonnegative matrix factorization. In Workshop on
Link Analysis, Counterterrorism and Security, SIAMConf. on Data Mining, Newport Beach, CA, 2005.
[4] J. D. Carroll and J. J. Chang. Analysis of individualdi"erences in multidimensional scaling via an N-waygeneralization of ‘Eckart-Young’ decomposition.Psychometrika, 35:283–319, 1970.
[5] A. Chapanond, M. S. Krishnamoorthy, and B. Yener.Graph theoretic and spectral analysis of Enron emaildata. In Workshop on Link Analysis,Counterterrorism and Security, SIAM Conf. on DataMining, Newport Beach, CA, 2005.
[6] W. W. Cohen. Enron email dataset. Webpage.http://www.cs.cmu.edu/!enron/.
[7] J. E. Dennis, Jr. and R. B. Schnabel. NumericalMethods for Unconstrained Optimization andNonlinear Equations. Prentice-Hall, Englewood Cli"s,NJ, 1983.
[8] J. Diesner and K. M. Carley. Exploration ofcommunication networks from the Enron emailcorpus. In Workshop on Link Analysis,Counterterrorism and Security, SIAM Conf. on DataMining, Newport Beach, CA, 2005.
[9] Federal Energy Regulatory Commision. Ferc:Information released in Enron investigation.http://www.ferc.gov/industries/electric/indus-act/wec/enron/info-release.asp.
[10] C. W. Harris and H. F. Kaiser. Oblique factor analyticsolutions by orthogonal transformations.Psychometrika, 29(4):347–362, 1964.
[11] R. A. Harshman. Foundations of the PARAFACprocedure: models and conditions for an“explanatory” multi-modal factor analysis. UCLAworking papers in phonetics, 16:1–84, 1970.
[12] R. A. Harshman. Models for analysis of asymmetricalrelationships among n objects or stimuli. In FirstJoint Meeting of the Psychometric Society and theSociety for Mathematical Psychology, McMasterUniversity, Hamilton, Ontario, August 1978.http://publish.uwo.ca/!harshman/asym1978.pdf.
[13] R. A. Harshman. Alternating least squares estimationfor the single domain DEDICOM model, 1981.Unpublished technical memorandum, BellLaboratories, Murray Hill, NJhttp://publish.uwo.ca/!harshman/asym1981.pdf.
• DEDICOM paper on Social Network Analysis:- Tech report SAND2006-2161 available
• MATLAB Tensor Toolbox:- http://csmr.ca.sandia.gov/~tgkolda/TensorToolbox- Tech report SAND2004-5189 available on website- Paper to appear in ACM Trans. Math. Softw.- sparse_tensor class to be released soon
