ABSTRACTInformation networks are ubiquitous in many applications. A pop-ular way to facilitate the information in a network is to embedthe network structure into low-dimension spaces where each nodeis represented as a vector. The learned representations have beenproven to advance various network analysis tasks such as linkprediction and node classification. The majority of existing em-bedding algorithms are designed for the networks with one typeof nodes and one dimension of relations among nodes. However,many networks in the real-world complex systems have multipletypes of nodes and multiple dimensions of relations. For example,an e-commerce network can have users and items, and items canbe viewed or purchased by users, corresponding to two dimensionsof relations. In addition, some types of nodes can present hier-archical structure. For example, authors in publication networksare associated to affiliations; and items in e-commerce networksbelong to categories. Most of existing methods cannot be natu-rally applicable to these networks. In this paper, we aim to learnrepresentations for networks with multiple dimensions and hierar-chical structure. In particular, we provide an approach to captureindependent information from each dimension and dependent infor-mation across dimensions and propose a framework MINES, whichperforms Multi-dImension Network Embedding with hierarchicalStructure. Experimental results on a network from a real-worlde-commerce website demonstrate the effectiveness of the proposedframework.
∗Work performed during an internship at Data Science Lab, JD.com.†These two authors contributed equally.‡Corresponding author
KEYWORDSNetwork Embedding, Multi-dimensional Networks, HierarchicalStructureACM Reference Format:Yao Ma, Zhaochun Ren, Ziheng Jiang, Jiliang Tang, and Dawei Yin. 2018.Multi-Dimensional Network Embedding with Hierarchical Structure. InWSDM 2018: WSDM 2018: The Eleventh ACM International Conference onWebSearch and Data Mining , February 5–9, 2018, Marina Del Rey, CA, USA. ACM,New York, NY, USA, 9 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn
1 INTRODUCTIONWe are living in a connected world where information networksare ubiquitous. Some examples of information networks includesocial networks, publication networks, the World Wide Web ande-commerce networks. Network embedding, aiming to learn vectorrepresentations for nodes, has attracted increasing attention inrecent years. Many advanced network embedding algorithms haveemerged such as Deepwalk [26], LINE [29] and Metapath2vec [11],which have been proven to help numerous network analysis taskssuch as link prediction [18], node classification [5][33] and networkvisualization [21][28].
Most of existing embedding algorithms are designed for net-works with one type of nodes and one dimension of relationsamong nodes. However, many networks in real-world complexsystems contain multiple dimensions of relations among nodes. Forexample, in social networking sites such as Facebook, two userscould be connected by friend relations, and via various social in-teractions; in the transportation network [3], two cities could beconnected via various means of transportations such as train, high-way and airplane; while in e-commerce networks, items can beviewed and purchased by users, corresponding to two dimensionsof relations between users and items. In addition, some of the nodescan present certain hierarchical structure. For example, in publi-cation networks, authors are associated to affiliations; while ine-commerce networks, items are organized by categories. A typicalexample of multi-dimensional networks with hierarchical struc-ture is illustrated in Figure 1 where there are two types of nodesU = {u1,u2,u3,u4} and T = {t1, t2, t3, t4}, and C = {c1, c2, c3}is the set of parent nodes. The relations of nodes in U, nodes inT and nodes betweenU and T are two-dimensional; while eachnode inU is associated to one parent in C. The vast majority of
Figure 1: An illustrative example of amulti-dimensional net-work with hierarchical structure
existing embedding algorithms cannot be naturally applicable tomulti-dimensional networks with hierarchical structure as shownin Figure 1.
In this paper, we aim to learn representations of nodes in net-works with multiple dimensions and hierarchical structure. In par-ticular, we study approaches (1) to mathematically capture multi-dimensional information and hierarchical structure; and (2) to in-corporate such information simultaneously for embedding. Conse-quently, we propose a framework MINES for Multi-dImensionalNetwork Embedding with hierarchical Structure. Our major con-tributions are summarized as follows:
• Providing a principled approach to model multi-dimensionalnetworks, which can capture independent information fromeach dimension and dependent information across dimen-sions;• Proposing a framework MINES, which incorporates multi-dimensional relations and hierarchical structure into a co-herent model for node representation learning; and• Validating the effectiveness of the proposed framework in areal-world e-commerce network.
The rest of this paper is arranged as follows. In Section 2, wereview some works that are related to our problem. The problemof embedding networks with multiple dimensions and hierarchi-cal structure to vector space is formally defined in Section 3. Theapproach to model networks with multiple dimensions and hierar-chical structure and the proposed framework with an optimizationmethod are introduced in Section 4. The experiments on a real-world e-commerce network with discussions are presented in Sec-tion 5. The conclusion and future work are presented in Section 6.
2 RELATEDWORKOur work is related to multi-dimensional network analysis andnetwork embedding. In this section, we briefly review them.
2.1 Multi-dimensional Network AnalysisNetwork analysis has been extensively studied for many years[34][35][6][14][2][7]. Multidimensional networks, which are quiteubiquitous in the real-world applications, have attracted increasingattention. In [3], the authors introduced a few examples of real-world multidimensional networks, and they also defined measuressuch as degree, neighbors for the multidimensional networks. Moremeasures for the multidimensional networks are introduced in [22].The classic link prediction problem has been extended to multidi-mensional networks with the new problem “what is the probabilitythat a new link between two nodes will form in a specific dimen-sion?” [27]. Multidimensional versions of the Common Neighborsand Adamic-Adar have been introduced to solve this problem [27].In [4], the authors studied the community discovery problem in themultidimensional network setting. In [16], the authors investigatedfriendship maintenance and prediction in multidimensional socialnetworks.
2.2 Network EmbeddingNetworks can be represented by adjacency matrices; however, theserepresentations are too sparse and high-dimensional. Many classicmethods such as Laplacian eigenmap [1] and IsoMap [30] havebeen proposed to learn low-dimensional representations. Thesemethods work fine on small size networks but cannot be scaledto very large networks. Inspired by word2vec [23][25], DeepWalkand LINE are proposed recently which can be applied to very largescale networks. DeepWalk regards the nodes in the network as the“words” of an artificial language and uses random walk to generatethe “sentences” for this language. Then, following the procedure ofword2vec, the representations for the nodes can be learned. LINEtries to capture both the first order and second order proximity inthe representations. node2vec [13] extends DeepWalk by addingparameters to introduce the biased random walk. These networkembedding methods have shown effectiveness in various taskson many homogeneous networks. In [11], the authors extendedDeepWalk method to heterogeneous networks by introducing meta-path based random walks. In [9], the authors also facilitate themeta-path to learn the heterogeneous network embedding, whilethey focus on the selection of the meta-path. In [8], the authorsfacilitate deep architectures to perform heterogeneous networkembedding. In [32], a signed network embedding algorithm SiNE isproposed based on the notion that a user should be closer to their“friend” than their “enemy”. In [20], the authors try to preserve bothlocal and global information in the network for network embedding.There are also works on attributed network embedding [17][31].Two recent surveys [10][12] give a comprehensive overview ofnetwork embedding algorithms. However, most of the existingmethods cannot naturally be applicable to networks with multipledimensions of relations and hierarchical structure. In this paper,we aim to model the multi-dimensional relations and hierarchicalstructure and propose a framework to embed these networks tovector space.
3 PROBLEM STATEMENTIn the multi-dimensional networks, we have different types of nodesand multiple dimensions of relations. Assume that there areK types
of nodes in total and letVi = {v(i)1 ,v
(i)2 , . . . ,v
(i)Ni} be the set of the
i-th type with Ni nodes. Let V denote the set of all the nodes
V =K⋃i=1Vi . Some types of nodes in the network might present
hierarchical structure. In other words, these nodes are associatedwith categories. For simplicity, we assume all the types of nodeshave hierarchical structures with a depth of 2, and we name theparent nodes as categories in this case. Note that though in thiswork, we focus on the hierarchical structures with a depth of 2,it is straightforward to apply the proposed framework for deeperhierarchical structures. We set Ci = {c
(i)1 , c
(i)2 , . . . , c
(i)Mi} as the set
ofMi categories for the i-th type of nodes, and set T(i) ∈ RNi×Mi
as the matrix that describes the category information, for the i-thtype of nodes.
Two nodes could be connected via multiple relations, and weregard each type of relations as a dimension. Thus, nodes from thesame type or different types can be connected in the same dimen-sion. These connections can be described by adjacency matrices(for the same type nodes) and the interaction matrices (for differenttypes of nodes). Let A(i)d ∈ R
Ni×Ni be the adjacency matrix of the i-
th node type and H(i, j)d ∈ RNi×Nj be the interaction matrix betweenthe i-th and j-th types of node in the d-th dimension. We targetto learn representations for each node in each dimension of thenetwork. LetUd
(i) = {ud(i)1 , ud
(i)2 , . . . , ud
(i)Ni} denote the represen-
tations of i-th type of nodes in the dimension d (d = 1, . . .D) whereD is the number of dimensions.
With the aforementioned notations and definitions, our problemcan be formally defined as follow:
Given
• K different sets of nodes, i.e., Vi = {v(i)1 ,v
(i)2 , . . . ,v
(i)Ni} (i =
1, . . . ,K);• multi-dimensional relations among the nodes, i.e., A(i)d (i =
1, . . . ,K) and H(i, j)d (i, j = 1, . . . ,K ; i , j;d = 1, . . . ,D);• the hierarchical structure information, i.e., Ti (i = 1, . . . ,K).
We aim to learn a set of representations for all nodes, i.e.,
Ud(i) = {ud
(i)1 , ud
(i)2 , . . . , ud
(i)Ni} (i = 1, . . . ,K)
in each dimension d (d = 1, . . .D).
4 THE MULTI-DIMENSIONAL EMBEDDINGFRAMEWORKWITH HIERARCHICALSTRUCTURE
In this section, wewill first introduce how tomodelmulti-dimensionalrelations and hierarchical structure; and then discuss the proposedframework with an optimization method.
4.1 Capturing Multi-Dimensional RelationsIn a multi-dimensional network, all dimensions share the sameset of nodes, while having their own network structures in eachdimension. A straightforward way to learn representations for eachdimension is to perform the network embedding for each dimen-sion, separately. This strategy treats each dimension independently
and completely ignores information across dimensions. Hence thelearned representations for different dimensions are not related.However, dimensions are inherently related since they share thesame set of nodes. Thus, in this subsection, we study how to modelmulti-dimensional relations.
Intuitively, each dimension should have its independent infor-mation individually; while all dimensions should share dependentinformation across dimensions. Therefore, the learned represen-tations for each dimension should not only preserve independentinformation from the dimension but also keep dependent informa-tion across dimensions. To achieve this goal, for a given dimensiond , the representation ud for a node contains two components – (1)one component u for the information shared across dimensions;and (2) one component ed specific to the dimension d . With thesetwo components, we can rewrite ud as:
ud = f (u, ed ), (1)
where f is a function to combine the shared component u and thespecific component ed . The shared component u not only capturesdependent information across dimensions but also helps the learnedrepresentations of all dimensions to be related. The specific com-ponent ed preserves independent information from the dimensiond .
4.2 Capturing Hierarchical StructureFor these nodes which have the hierarchical structure, we also needto model its category information (or parents). The category infor-mation is actually shared by all the dimensions, hence it should beindicated in the shared component of representations. For the nodesin the same category, they should also share the similar character-istics. Therefore, to model the hierarchical structure, the sharedcomponent of the node representation should further contain twocomponents – (1) one component cu indicates category informa-tion which is shared by all the nodes in the category, and (2) onecomponent su is specific to the node. With the defined components,we can further rewrite u in Eq. (1) as:
u = д(cu , su ) (2)
where д is the function to combine the category shared informationcu and the node specific information su . Note that Eq. (2) can beeasily extended to deeper hierarchical structure by further decom-posing the category shared information cu .
4.3 The Proposed FrameworkWith approaches to capture multi-dimensional relations and hier-archical structure, in this subsection, we introduce the embeddingframework MINES.
To learn the embeddings for the nodes in each dimension, wefollow the idea of skip-gram model [24], which is an effective andefficient way to learn distributed representations of words. Theskip-gram model predicts surrounding context given a center word,which can be formulated as follows:
p(N (wc )|wc ), (3)
where N (wc ) is the set of words that surround wordwc .Similarly, we can use the skip-gram to model the network in a
given dimension d . For a node v , we define all nodes connected to
v as the “context” of v , which is formally defined as:
Nd (v) =K⋃i=1
N(i)d (v); (4)
where N(i)d is the set of i-th type of nodes that are connected to
node v in the dimension d . Note that the “context” of v consists ofdifferent types of nodes, and we treat them differently, which willbe further explained later.
Then, given a center node v , we need to predict its “context” as:
pd (Nd (v)|v) =K∏i=1
p(N(i)d (v)|v) =
K∏i=1
∏v (i )j ∈N
(i )d (v)
pd (v(i)j |v); (5)
where p(v(i)j |v) can be modeled using a softmax function as:
pd (v(i)j |v) =
exp(uTd ud(i)j )∑
v (i )∈Viexp(uTd ud
(i)). (6)
In (6), the softmax function is over the i-th type nodesVi insteadof the whole nodes setV .
To learn the representations for the dimension d , we model thisproblem as a maximum likelihood problem. In other words, weneed to maximize the probability that Nd (v) is the “context” ofnode v for all the nodes v ∈ V . Hence, we need to maximize:
Pd =∏v ∈V
pd (Nd (v)|v). (7)
With allD dimensions, we need to jointlymaximize the followingterm:
P =D∏d=1
Pd . (8)
Instead of maximizing Eq. (8), we equivalently minimize its neg-ative logarithm with respect to the representationsU(i)d as:
min{U(i )d }
d=1, . . .,Di=1, . . .,K
− log P
⇔ min{U(i )d }
d=1, . . .,Di=1, . . .,K
−
D∑d=1
log Pd (9)
⇔ min{U(i )d }
d=1, . . .,Di=1, . . .,K
−
D∑d=1
∑v ∈V
K∑i=1
∑v (i )j ∈N
(i )d (v)
logpd (v(i)j |v).
4.4 An Optimization MethodThere are two challenges to address when optimizing Eq. (9). First,the minimization of Eq. (9) is computationally expensive due tosummation over the whole set of nodesVi when calculating eachterm logpd (v
(i)j |v). Second, how to choose the functions of f in
Eq. (1) and д in Eq. (2).To solve computational challenge, we adopt the negative sam-
pling approach proposed in [25]. By using the negative sampling
method, we replace each logpd (v(i)j |v) with
Od (v,v(i)j ) = logσ (uTd ud
(i)j ) +
Ne∑n=1
logσ (−uTd ud(i)(n)); (10)
where σ (x) = 1/(1 + exp(x)) is the sigmoid function, and Ne isthe number of negative samples. The negative samples are ran-domly sampled from some noise distribution. For logpd (v
(i)j |v),
the negative samples are sampled from node setVi according toP(i)i(v)(v
(i)) ∼ d3/4v (i )
, as proposed in [25], where dv (i ) is the in-degree
of v(i) corresponding to the i(v)-th type of nodes and i(v) indicatesthe type of nodev . Note again, for i-th type of nodev(i)j , we samplethe negative samples from the i-th type of nodes setVi instead ofthe whole nodes setV .
We adopt mini-batch Stochastic Gradient Descent (SGD) to opti-mize the problem. In each step, a mini-batch of edges of the sametype are sampled according to their weights. Here, by “same” typeof edge, we mean, these edges have same types of nodes for sourceand target nodes respectively and also the relations between themis in the same dimension. For each sampled edge, the source nodeis treated as v and the target node is treated as v(i)j in (10). The
derivatives for v , v(i)j and v(i)(n) are
∂Od (v,v(i)j )
∂ud= (1 − σ (uTd ud
(i)j ))ud
(i)j −
Ne∑n=1(1 − σ (−uTd ud
(i)(n)))ud
(i)(n);
∂Od (v,v(i)j )
∂ud(i)j
= (1 − σ (uTd ud(i)j ))ud ; (11)
∂Od (v,v(i)j )
∂ud(i)(n)
= −(1 − σ (−uTd ud(i)(n)))ud ,n = 1, . . . ,Ne .
Next we discuss how to choose f and д functions. In fact, f isused to combine the dimension shared component u and dimension-specific component cd . It can be a linear function, a non-linearfunction (e.g., exponential functions) or even can be automaticallylearned (e.g., neural networks). In this work, we choose a linearfunction f . In other words, we define as – f (u, ed ) = u+cd . We alsouse a similar function for д. We would like to leave the investigationof other choices of f and д as one future direction. With choices off and д, the representations for nodes can be rewritten as:
ud = cu + su + ed ; (12)
ud(i)j = cu
(i)j + su
(i)j + ed
(i)j ; (13)
ud(i)(n) = cu
(i)(n) + su
(i)(n) + ed
(i)(n). (14)
We need to update cu , su , ed , cu j , su j , ed j , cu (n), su (n) and ed (n).We update these representations using Gradient Decent (GD).
To update the representations for v , we need to update its threecomponents cu , su and cd according to (15).
cu ← cu + ρ ·∂Od (v,v
(i)h )
∂ud;
su ← su + ρ ·∂Od (v,v
(i)h )
∂ud; (15)
ed ← ed + ρ ·∂Od (v,v
(i)h )
∂ud.
Similarly, to update the representations for v(i)j , we need to up-
date its three components cu(i)j , su
(i)j and cd
(i)j according to (16).
cu(i)j ← cu
(i)j + ρ ·
∂Od (v,v(i)j )
∂ud(i)j
;
su(i)j ← su
(i)j + ρ ·
∂Od (v,v(i)j )
∂ud(i)j
; (16)
ed(i)j ← ed
(i)j + ρ ·
∂Od (v,v(i)j )
∂ud(i)j
.
Finally, to update the representations for v(i)(n), n = 1, . . . ,Ne , we
need to update their three components cu(i)(n), su
(i)(n) and cd
(i)(n) ac-
cording to (17) respectively.
cu(i)(n) ← cu
(i)(n) + ρ ·
∂Od (v,v(i)j )
∂ud(i)(n)
,n = 1, . . .Ne;
su(i)(n) ← su
(i)(n) + ρ ·
∂Od (v,v(i)j )
∂ud(i)(n)
,n = 1, . . .Ne; (17)
ed(i)(n) ← ed
(i)(n) + ρ ·
∂Od (v,v(i)j )
∂ud(i)(n)
,n = 1, . . .Ne .
We summarize the optimization procedure in Algorithm 1. Inthe algorithm, the input includes the number of mini-batch sizem, the training size S , the dimension of representations dim, thenumber of negative samples Ne , the learning rate ρ and the setof all the edges E in the network. In line 1, we initialize all thecomponents for all the representations. Then, we sample a set ofsame type edges SE from E in line 4. In line 6, for each edge, wesample Ne negative samples. We calculate the gradients and updatethe components in lines 7 and 8, respectively. Finally, we combinethe components to form the representations for each node in eachdimension in line 12.
To efficiently sample the edges and negatives samples, we adoptthe alias methods proposed in [15], which can generate a randomvariable from a discrete distribution in constant time O(1). Theoptimization with negative sampling takes O(dim · (Ne + 2) + Ne)time, where Ne is the number of negative samples. hence, each stepof MINES takesO(dim · Ne) operations. If the training size is S , theoverall time complexity of MINES is O(S · dim · Ne).
Algorithm 1: Optimization procedureInput: Ne ,m, S , ρ, dim, EOutput: {U(i)d }
d=1, ...,Di=1, ...,K
1 Initialize cu(i)j , su
(i)j and ed
(i)j , as dim dimension vectors
randomly, for d = 1, . . . ,D, i = 1, . . . ,K and j = 1, . . . ,Ni ;2 s = 0;3 while s < S do4 Sample a set ofm edges of the same type SE from E;5 for e = (v,v(i)j ) ∈ SE do
6 Sample a set of Ne negative samples {v(i)(n)}n=1, ...,Ne ;
7 Calculate the gradients according to (11);8 Update the corresponding vectors according to (15),
(16) and (17);9 end
10 s ← s +m.11 end12 ud
(i)j = cu
(i)j + su
(i)j + ed
(i)j ; for d = 1, . . . ,D, i = 1, . . . ,K and
j = 1, . . . ,Ni ;13 return {U(i)d }
d=1, ...,Di=1, ...,K .
5 EXPERIMENTSIn this section, we present the experimental details to verify theeffectiveness of the proposed framework. We first introduce thedataset we will use in the evaluation. Then, we describe the experi-mental settings. Finally, we present the experimental results withdiscussions and study the key parameter in the proposed frame-work.
5.1 DatasetIn our experiments, we sample data from JD.com, which is one of thelargest e-commerce companies. In our dataset, we have two typesof nodes: users and items. The items have hierarchical structure andeach item belongs to some predefined categories. Users can performvarious behaviors on items such as “view", “save", and “purchase".In this work, we collect two behaviors, i.e., “view” and “purchase”,to construct two-dimensional relations between users and items.In addition, we collect two other relations: one is the “view session”of a user, while another is the “purchase basket” of the user.
A view session is a sequence of items that are viewed by a userwithin a period of time. It is intuitively to understand the items thatare viewed within a short period by the same user should be similar.To incorporate these relations into the network, we construct anitem-item view network by connecting the items that are viewedwitems before or after a given item in a session with this item, wherew is the window size. In this work, we set the window size to 5.These edges are in the “view” dimension and they are weightedwhere the weight is the co-occurrence frequency.
A purchase basket is a set of items that are purchased by a userat the same time. Items that are purchased in the same basket aresupposed to be related to each other. To incorporate these relations,we construct an item-item purchase network. In particular, weconnect two items if they are purchased in the same basket. These
edges are weighted where the weight is the frequency of the twoitems presenting in the same basket. In the other way around,users that have “viewed” or “purchased” the same item also showssimilarity. In each dimension, we connected users that have “viewed”or “purchased” the same items.
To sum up, in the constructed multidimensional e-commercenetwork, we have two types of nodes, i.e., the users and the items,and the items have hierarchical structure. There are two dimensions,i.e., the “view” dimension and the “purchase” dimension. We canconclude that the constructed network has all the characteristicsof networks we want to study in this work; hence it is suitable forus to use the dataset to evaluate the proposed framework. Somestatistics of the network are shown in Table 1.
5.2 Experimental SettingFollowing the common way to assess network embedding algo-rithms [13], we choose link prediction as the evaluation task. Theintuition is that a better embedding algorithm should learn bet-ter node representations, which will lead to better link predictionperformance.
In the link prediction task, a certain fraction of edges are removed,and we would like to predict whether these “missing” edges exist.
In our evaluation, we perform the link prediction task on thetwo dimensions, separately. For each dimension, we remove theuser-item edges and use them as parts of the testing set. We set up3 groups of experiments, where 10%, 30% and 50% of the user-itemedges are removed, respectively. To form the training set, we firstput all the remaining user-item edges into the training set, and then,for each user-item edge in the training set, we randomly samplean item that is not connected to this user and use this user andnon-connected item pair as the negative sample in the training set.We form the testing set in the same way.
After removing the edges, we use the remaining network tolearn the representations for all the nodes. Then, to perform thelink prediction task, the representations for the edges (or the useritem pairs) should be learned.We use two different ways to combinethe representations of two nodes as the representation of the edge(or user item pair) as used in [13].
• Element-wise addition Given two dim dimension repre-sentations of two nodes, we add them element-wisely andget a new dim dimension vector as the representation forthis pair of nodes.
• Element-wise multiplication Given two dim dimensionrepresentations of two nodes, we multiply them element-wisely and get a new dim dimension vector as the represen-tation for this pair of nodes.
For all themethods, we use bothways to form the representationsfor the pairs of nodes and report the results for each method.
After we form the representations for the pairs of the nodes inthe training set and the testing set, we train a binary classifier usinglogistic regression on the training set and perform link predictionon the testing set. In this work, we will use Micro-F1, Macro-F1 andAUC as the metric to evaluate the link prediction performance.
5.3 Performance ComparisonTo evaluate the performance of our algorithm, we compare theperformance of our algorithm with the following representativebaselines:• LINE [29]: As LINE can only work for one-dimensional net-work, we apply LINE to the two dimensions separately andlearn one set of representations for each dimension, respec-tively. We treat categories as nodes, and add item-categoryedges into the networks for LINE.• DeepWalk [26]: We apply DeepWalk to the two dimensionsseparately and learn two sets of representations. We treatcategories as nodes, and add item-category edges into thenetworks for DeepWalk. DeepWalk can only work for un-weighted networks, hence, we convert our network to un-weighted network by ignoring the weights.• Non-negative Matrix Factorization (NMF) [19]: We ap-ply it to the user-item interaction matrix and use the factor-ized two matrices as the embeddings for the users and items.NMF is also applied to the two dimensions, separately.• Co-NMF: In Co-NMF, we perform a co-factorization on themulti-dimensional networks and learn unified user repre-sentations for all dimensions. Basically Co-NMF assumes alldimensions share the same embeddings, which completelyignores independent information from each dimension.• MINES(S): This is a variant of our framework MINES. In-stead of using all three components cu , su and ed , we onlyuse the shared components cu and su to form the represen-tation for a node v .
We summarize the experiments results for the “view” dimensionand “purchase” dimension in Table 2, and Table 3, respectively. Wemake the following observations from Table 2:• For all methods, using the Element-wise multiplicationis better than Element-wise addition, which is consistentwith the observation in [13].• The performance of Co-NMF is worse than that of NMF,which indicates that the independent information from eachdimension is very important to accurately predict links inthat dimension.• MINES shows better performance than MINES(S), whichfurther shows the importance of the dimension specific in-formation.• As we remove more percent of edges, the performance ofall methods decrease in all three measures when using theElement-wise multiplication.
Table 3: Link Prediction Performance Comparison: The Purchase Dimension
• The performance of DeepWalk is worse than LINE and NMF.This is mainly because DeepWalk can only work for un-weighted networks and cannot take advantage of the edgeweights.• The proposed framework MINES obtains the best perfor-mance. For example, MINES boosts the performance 3%− 5%compared to the best baseline when 10% − 50% edges are
moved. The major reason is the proposed framework hastwo components to capture the multi-dimensional relationsand the hierarchical structure.
We have similar observations for the “purchase” dimension –(1) using the Element-wise multiplication is better than using
Figure 2: Parameter Analysis: The View Dimension
Figure 3: Parameter Analysis: The Purchase Dimension
Element-wise addition and (2) MINES outperforms all the base-lines; for example, MINES obtains over 2% improvement in termsof all the measures compared to the best baseline.
5.4 Parameter AnalysisIn this section, we analyze how the dimension of the learned rep-resentations in our method affects the performance of the linkprediction task. In particular, we set the dimension of the represen-tations to {16, 32, 64, 128} with the setting of 50% edges removed.The results are reported in in Figure 2 and Figure 3 for view andpurchase dimensions, separately. Note that we ignore the resultswith other settings since we can make similar observations.
As shown in Figure 2 and Figure 3, in both view and purchasedimensions, the Micro-F1 and Macro-F1 first increase as the dimen-sion of the learned representations gets large, and then decrease.Both the Micro-F1 and Macro-F1 reach the maximum when thedimension of the representations is 32 in both view and purchasedimension. AUC also increases first and then decrease as the dimen-sion of the representations gets larger. However, the AUC scorereaches the maximum when the dimension of the representation is64.
In summary, the performance first increases and then decreasesas the dimension of the representations gets larger. The dimensionof the representations affects the different measures differently.
6 CONCLUSIONIn this paper, we propose an approach to model multi-dimensionalnetworks, which can capture independent information from eachdimension and dependent information across dimensions. Basedon this approach, we propose the MINES framework which canembed multi-dimensional network with hierarchical structure tolow-dimensional vector spaces. We can learn a set of node repre-sentations for each dimension using this framework. The learnedrepresentations for each dimension will contain the hierarchical in-formation, the independent information from the specific dimensionand also dependent information across dimensions. We evaluate theeffectiveness of our framework on a multi-dimensional e-commercenetwork. The results of our experiments show the advancement ofour framework.
In this work, we utilize linear functions to model the across di-mension information and the hierarchical structure information.In our future work, more complicated non-linear functions suchas exponential functions or even the neural networks can be used.Meanwhile, as a limitation in our work, we only focus on hierarchi-cal structures with depth of 2 in this paper. As another direction inour future work, we would like to investigate the proposed frame-work with deeper hierarchical structures. Real-world networkstypically evolve such as addition of new nodes and links, and dele-tion of old nodes and links. Therefore, multi-dimensional networkembedding with dynamics should provide new insights in future.
ACKNOWLEDGEMENTSThe authors wish to thank the anonymous reviewers for their help-ful comments. Yao Ma and Jiliang Tang are supported by the Na-tional Science Foundation (NSF) under grant number IIS-1714741and IIS-1715940.
REFERENCES[1] Mikhail Belkin and Partha Niyogi. 2002. Laplacian eigenmaps and spectral
techniques for embedding and clustering. In Advances in neural informationprocessing systems. 585–591.
[2] Michael GH Bell, Yasunori Iida, et al. 1997. Transportation network analysis.(1997).
[3] Michele Berlingerio, Michele Coscia, Fosca Giannotti, Anna Monreale, and DinoPedreschi. 2013. Multidimensional networks: foundations of structural analysis.World Wide Web 16, 5-6 (2013), 567–593.
[4] Michele Berlingerio, Fabio Pinelli, and Francesco Calabrese. 2013. Abacus: fre-quent pattern mining-based community discovery in multidimensional networks.Data Mining and Knowledge Discovery 27, 3 (2013), 294–320.
[5] Smriti Bhagat, Graham Cormode, and S Muthukrishnan. 2011. Node classificationin social networks. In Social network data analytics. Springer, 115–148.
[6] Ronald L Breiger, Scott A Boorman, and Phipps Arabie. 1975. An algorithmfor clustering relational data with applications to social network analysis andcomparison with multidimensional scaling. Journal of mathematical psychology12, 3 (1975), 328–383.
[7] Carter T Butts. 2009. Revisiting the foundations of network analysis. science 325,5939 (2009), 414–416.
[8] Shiyu Chang, Wei Han, Jiliang Tang, Guo-Jun Qi, Charu C Aggarwal, andThomas S Huang. 2015. Heterogeneous network embedding via deep archi-tectures. In Proceedings of the 21th ACM SIGKDD International Conference onKnowledge Discovery and Data Mining. ACM, 119–128.
[9] Ting Chen and Yizhou Sun. 2017. Task-Guided and Path-Augmented Hetero-geneous Network Embedding for Author Identification. In Proceedings of theTenth ACM International Conference on Web Search and Data Mining, WSDM2017, Cambridge, United Kingdom, February 6-10, 2017, Maarten de Rijke, MiladShokouhi, Andrew Tomkins, and Min Zhang (Eds.). ACM, 295–304.
[10] Peng Cui, Xiao Wang, Jian Pei, and Wenwu Zhu. 2017. A Survey on NetworkEmbedding. arXiv preprint arXiv:1711.08752 (2017).
[11] Yuxiao Dong, Nitesh V Chawla, and Ananthram Swami. 2017. metapath2vec:Scalable Representation Learning for Heterogeneous Networks. (2017).
[12] Palash Goyal and Emilio Ferrara. 2017. Graph Embedding Techniques, Applica-tions, and Performance: A Survey. arXiv preprint arXiv:1705.02801 (2017).
[13] Aditya Grover and Jure Leskovec. 2016. node2vec: Scalable feature learning fornetworks. In Proceedings of the 22nd ACM SIGKDD international conference onKnowledge discovery and data mining. ACM, 855–864.
[14] Gueorgi Kossinets and Duncan J Watts. 2006. Empirical analysis of an evolvingsocial network. science 311, 5757 (2006), 88–90.
[15] Richard A Kronmal and Arthur V Peterson Jr. 1979. On the alias method for gen-erating random variables from a discrete distribution. The American Statistician33, 4 (1979), 214–218.
[16] Ka-Wei Roy Lee and Ee-Peng Lim. 2016. Friendshipmaintenance and prediction inmultiple social networks. In Proceedings of the 27th ACM Conference on Hypertext
and Social Media. ACM, 83–92.[17] Jundong Li, Harsh Dani, Xia Hu, Jiliang Tang, Yi Chang, and Huan Liu. 2017.
Attributed Network Embedding for Learning in a Dynamic Environment. arXivpreprint arXiv:1706.01860 (2017).
[18] David Liben-Nowell and Jon M. Kleinberg. 2003. The link prediction problem forsocial networks. In Proceedings of the 2003 ACM CIKM International Conference onInformation and Knowledge Management, New Orleans, Louisiana, USA, November2-8, 2003. ACM, 556–559.
[20] Yao Ma, Suhang Wang, ZhaoChun Ren, Dawei Yin, and Jiliang Tang. 2017. Pre-serving Local and Global Information for Network Embedding. arXiv preprintarXiv:1710.07266 (2017).
[21] Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE.Journal of Machine Learning Research 9, Nov (2008), 2579–2605.
[22] MARGGF Magnani, Anna Monreale, Giulio Rossetti, and Fosca Giannotti. 2013.On multidimensional network measures. In Italian conference on Sistemi Evolutiper le Basi di Dati (SEBD).
[23] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficientestimation of word representations in vector space. arXiv preprint arXiv:1301.3781(2013).
[24] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. EfficientEstimation of Word Representations in Vector Space. CoRR abs/1301.3781 (2013).
[25] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.Distributed representations of words and phrases and their compositionality. InAdvances in neural information processing systems. 3111–3119.
[26] Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014. DeepWalk: online learningof social representations. In The 20th ACM SIGKDD International Conference onKnowledge Discovery and Data Mining, KDD ’14, New York, NY, USA - August 24- 27, 2014, Sofus A. Macskassy, Claudia Perlich, Jure Leskovec, Wei Wang, andRayid Ghani (Eds.). ACM, 701–710.
[27] Giulio Rossetti, Michele Berlingerio, and Fosca Giannotti. 2011. Scalable linkprediction on multidimensional networks. In Data Mining Workshops (ICDMW),2011 IEEE 11th International Conference on. IEEE, 979–986.
[28] Jian Tang, Jingzhou Liu, Ming Zhang, and Qiaozhu Mei. 2016. VisualizationLarge-scale and High-dimensional Data. CoRR abs/1602.00370 (2016).
[29] Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei.2015. LINE: Large-scale Information Network Embedding. In Proceedings of the24th International Conference on World Wide Web, WWW 2015, Florence, Italy,May 18-22, 2015, Aldo Gangemi, Stefano Leonardi, and Alessandro Panconesi(Eds.). ACM, 1067–1077.
[30] Joshua B Tenenbaum, Vin De Silva, and John C Langford. 2000. A global geometricframework for nonlinear dimensionality reduction. science 290, 5500 (2000), 2319–2323.
[31] Suhang Wang, Charu C. Aggarwal, Jiliang Tang, and Huan Liu. [n. d.]. AttributedSigned Network Embedding. In Proceedings of CIKM.
[32] Suhang Wang, Jiliang Tang, Charu C. Aggarwal, Yi Chang, and Huan Liu. 2017.Signed Network Embedding in Social Media. In Proceedings of SDM. 327–335.
[33] Suhang Wang, Jiliang Tang, Charu C. Aggarwal, and Huan Liu. 2016. LinkedDocument Embedding for Classification. In Proceedings of CIKM. 115–124.
[34] Stanley Wasserman and Katherine Faust. 1994. Social network analysis: Methodsand applications. Vol. 8. Cambridge university press.
[35] Barry Wellman. 1983. Network analysis: Some basic principles. Sociologicaltheory (1983), 155–200.
