Graph Representation Learning: A SurveyFENXIAO CHEN, YUNCHENG WANG, BIN WANG AND C.-C. JAY KUO
Research on graph representation learning has received a lot of attention in recent years since many data in real-worldapplications come in form of graphs. High-dimensional graph data are often in irregular form, which makes them moredifficult to analyze than image/video/audio data defined on regular lattices. Various graph embedding techniques havebeen developed to convert the raw graph data into a low-dimensional vector representation while preserving the intrinsicgraph properties. In this review, we first explain the graph embedding task and its challenges. Next, we review a wide rangeof graph embedding techniques with insights. Then, we evaluate several stat-of-the-art methods against small and largedatasets and compare their performance. Finally, potential applications and future directions are presented.
I. INTRODUCTION
Research on graph representation learning has gained moreand more attention in recent years since many real worlddata can be represented by graphs conveniently. Examplesinclude social networks, linguistic (word co-occurrence)networks, biological [87] networks and many other mul-timedia domain-specific data. Graph representation allowsthe relational knowledge of interacting entities to be storedand accessed efficiently [4]. Analysis of graph data canprovide significant insights into community detection [37],behavior analysis and other useful applications such asnode classification [7], link prediction [65] and cluster-ing [44]. Various graph embedding techniques have beendeveloped to convert the raw graph data into a high-dimensional vector while preserving intrinsic graph prop-erties. This process is also known as graph representationlearning. With a learned graph representation, one canadopt machine learning tools to perform downstream tasksconveniently. Obtaining an accurate representation of agraph is challenging in three aspects. First, finding the opti-mal embedding dimension of a representation [99], [100]is not an easy task. A representation of a higher dimen-sion tends to preserve more information of the originalgraph at the cost of storage and computation. A represen-tation of a lower dimension is more resource efficient. Itmay reduce noise in the original graph as well. However,there is a risk in losing some critical information of theoriginal graph. The dimension choice depends on the inputgraph type as well as the application domain [79]. Second,choosing the proper graph property to embed is an issueof concern if a graph has a plethora of properties. Third,many graph embedding methods have been developed inthe past. It is desired to have some guideline in selectinga good embedding method for a target application. In thiswork, we intend to provide an extensive survey on graphembedding methods with the following three contributionsin mind.
University of Southern California, Los Angeles, CA 90089, USA
• We would like to offer new comers in this field a globalperspective with insightful discussion and an extensivereference list. Thus, a wide range of graph embeddingtechniques, including the most recent graph representa-tion models, are reviewed.
• To shed light on the performance of different embeddingmethods, we conduct extensive performance evaluationon both small and large data sets in various applica-tion domains. To the best of our knowledge, this is thefirst survey paper that provides systematic evaluation of arich set of graph embedding methods in domain-specificapplications.
• We provide an open-source Python library, called theGraph Representation Learning Library (GRLL), to read-ers. It offers a unified interface for all graph embeddingmethods discussed in this paper. This library covers thelargest number of graph embedding techniques up to now.
The rest of this paper is organized as follows. We firststate the problem as well as several definitions in Sec. II.Then, traditional and emerging graph embedding methodsare reviewed in Sec. III and IV, respectively. Next, we con-duct extensive performance evaluation on a large numberof embedding methods against different datasets in differ-ent application domains in Sec. V. The application of thelearned graph representation and the future research direc-tions are discussed in Sec. VI and Sec. VII, respectively.Finally, concluding remarks are given in Sec. VIII.
II. DEFINITION AND PRELIMINARIES
A) NotationsA graph, denoted by G = (V,E), consists of vertices, V ={v1, v2, ..., vn}, and edges, E = {ei,j}, where an edge ei,jconnects vertex vi to vertex vj . Graphs are usually repre-sented by an adjacency matrix or a derived vector spacerepresentation [26]. The adjacency matrix, A, of graph Gcontains non-negative weights associated with each edge,aij ≥ 0. If vi and vj are not directly connected to oneanother, aij = 0. For undirected graphs, aij = aji for all1 ≤ i ≤ j ≤ n.
1
arX
iv:1
909.
0095
8v1
[cs
.LG
] 3
Sep
201
9
2 FENXIAO CHEN, YUNCHENG WANG, BIN WANG AND C.-C. JAY KUO
Graph representation learning (or graph embedding)aims to map each node to a vector where the distance char-acteristics among nodes is preserved. Mathematically, forgraph G = (V,E), we would like to find a mapping:
f : vi → xi ∈ Rd,
where d� |V |, andXi = {x1, x2, ..., xd} is the embedded(or learned) vector that captures the structural properties ofvertex vi.
The first-order proximity [14] in a network is the pair-wise proximity between vertices. For example, in weightednetworks, the weights of the edges are the first-orderproximity between vertices. If there is no edge observedbetween two vertices, the first-order proximity betweenthem is 0. If two vertices are linked by an edge withhigh weight, they should be close in the embedding space.This objective can be obtained by minimizing the dis-tance between the joint probability distribution in the vec-tor space and the empirical probability distribution of thegraph. If we use the KL-divergence [41] to calculate thedistance, the objective function is given by
O1 = −∑
(i,j)∈E
wij log p1(vi, vj), (1)
where
p1(vi, vj) =1
1 + exp( ~−uiT · ~uj)
and where ~ui ∈ Rd is the low-dimensional vector represen-tation of vertex vi and wij is the weight. The second-orderproximity [115] is used to capture the 2-step relationshipbetween two vertices. Although there is no direct edgebetween two vertices of the second-order proximity, theirrepresentation vectors should be close in the embeddedspace if they share similar neighborhood structures.
The objective function of the second-order proximitycan be defined as
O2 = −∑
(i,j)∈E
wij log p2(vi|vj), (2)
where wi,j is the edge weight between node i and j and
p2(vj |vi =exp( ~u
′j
T· ~ui)∑|V |
k=1 exp( ~u′k
T· ~ui)
, (3)
and where ~ui ∈ Rd is the representation of vertex vi whenit is treated as a vertex and ~u
′j ∈ Rd is the vector represen-
tation of vertex vj when it is treated as a specific contextfor vertex vi.
Graph sampling is used to simplify graphs [5]. They canbe categorized into two types.
• Negative Sampling [68],[96]Negative sampling is proposed as an alternative to thehierarchical computation of the softmax, which reducesthe runtime of the softmax computation on a large scale
dataset. Graph optimization requires the summation overthe entire set of vertices. It is computationally expensivefor large-scale networks. Negative sampling is developedto address this problem. It helps distinguish the neighborsfrom other nodes by sampling multiple negative samplesaccording to the noise distribution. In the training pro-cess, correct surrounding neighbors positive examples incontrast to a set of sampled negative examples (usuallynoise).
• Edge Sampling [61], [74]In the training stage, it is difficult to choose an appropri-ate learning rate in graph optimization when the differ-ence between edge weights is large. To address this prob-lem, one solution is to use edge sampling that unfoldsweighted edges into several binary edges at the cost ofincreased memory. An alternative is treating weightededges as binary ones with their sampling probabilitiesproportional to the weights. This treatment would notmodify the objective function.
B) Graph InputGraph embedding methods take a graph as the input, wherethe graph can be homogeneous graphs, heterogeneousgraphs, with/without auxiliary information or constructedgraphs [11]. They are detailed below.
• Homogeneous graphs refer to graphs whose nodes andedges belong to the same type. All nodes and edges ofhomogeneous graphs are treated equally.
• Heterogeneous graphs contain different edge types torepresent different relations among different entities orcategories. For example, their edges can be directedor undirected. Heterogeneous graphs typically exist incommunity-based question answering (cQA) sites, mul-timedia networks and knowledge graphs. Most socialnetwork graphs are directed graphs [85]. Only the basicstructural information of input graphs is provided in realworld applications.
• Graphs with auxiliary information [38], [86] are thosethat have labels, attributes, node features, informationpropagation, etc. A label indicates node’s category.Nodes with different labels should be embsedded fur-ther away than those with the same label. An attributeis a discrete or continuous value that contains additionalinformation about the graph rather than just the struc-tural information. Node features are shown in form oftext information for each node. Information propagationindicates dynamic interaction among nodes such as postsharing or "retweet" while Wikipedia [104], DBpedia [8],Freebase [9], etc. provide popular knowledge bases.
• Graphs constructed from non-relational data are assumedto lie in a low dimensional manifold.
The input feature matrix can be represented as X ∈R|V |×N where each row Xi is a N -dimensional featurevector for the ith training instance. A similarity matrix,denoted by S, can be constructed by computing the sim-ilarity between Xi and Xj for graph classifications.
GRAPH REPRESENTATION LEARNING: A SURVEY 3
C) Graph OutputThe output of a graph embedding method is a set ofvectors representing the input graph. It could be nodeembedding, edge embedding, hybrid embedding or whole-graph embedding. The preferred output form is application-oriented and task-driven. We elaborate them below.
• Node embedding represents each node as a vector, whichwould be useful for node clustering and classification.For node embedding, nodes that are close in the graphare embedded closer together in the vector representa-tions. Closeness can be first-order proximity, second-order proximity or other similarity calculation.• Edge embedding aims to map each edge into a vector. It
is useful for predicting whether a link exists between twonodes in a graph. For example, knowledge graph embed-ding can be used for knowledge graph entity/relationprediction.• Hybrid embedding is the combination of different types
of graph components such as node and edge embed-ding. Hybrid embedding is useful for semantic proximitysearch and subgraphs learning. It can also be used forgraph classification based on graph kernels. Substructureor community embedding can also be done by aggregat-ing individual node and edge embedding inside it. Some-times, better node embedding is learned by incorporatinghybrid embedding methods.• Whole graph embedding is usually done for small graphs
such as proteins and molecules. These smaller graphs arerepresented as one vector, and two similar graphs areembedded to be closer. Whole-graph embedding facili-tates graph classification tasks by providing a straightfor-ward and efficient solution in computing graph similari-ties.
D) History of Graph EmbeddingThe study of graph embedding can be traced back to 1900swhen people questioned whether all planar graphs with nvertices have a straight line embedding in an nk × nk grid.This problem was solved in [21] and [29]. The same resultfor convex maps was proved in [83]. More analytic workon the embedding method and time/space complexity ofsuch a method were studied in [18] and [22]. However,a more general approach is needed since most real worldgraphs are not planer. A large number of methods havebeen proposed since then.
E) Overview of Graph Embedding IdeasWe provide an overview on various graph embedding ideasbelow.
• Dimensionality ReductionIn early 2000s, graph embedding is achieved by dimen-sionality reduction. For a graph with n nodes, each ofwhich is of dimension D, these embedding methods aimto embed nodes into a d-dimensional vector space, where
d� D. They are called classical methods and reviewedin Section A. Dimensionality reduction is less scalable.
• Random WalkOne can trace a graph by starting random walks from ran-dom initial nodes so as to create multiple paths. Thesepaths reveal the context of connected vertices. The ran-domness of these walks gives the ability to explore thegraph, capture the global and local structural informa-tion by walking through neighboring vertices. Later on,probability models like skip-gram and bag-of-word areperformed on the random sampled paths to learn noderepresentations. The random walk based methods will bediscussed in Section B.
• Matrix FactorizationBy leverage the sparsity of real-world networks, one canapply the matrix factorization technique that finds anapproximation matrix for the original graph. This idea iselaborated in Section C.
• Neural NetworksNeural network models such as convolution neural net-work (CNN) [57], recursive neural networks (RNN) [67]and their variants have been widely adopted in graphembedding. This topic will be described in Section A.
• Large GraphsSome large graphs are difficult to embed since CNN andRNN models do not scale well with the numbers of edgesand nodes. New embedding methods are designed target-ing at large graphs. They become popular due to theirefficiency. This topic is reviewed in Section B
• HypergraphsMost social networks are hypergraphs. As social net-works get more attention in recent years, hypergraphembedding becomes a hot topic, which will be presentedin Section C.
• Attention MechanismAttention mechanism can be added to existing embeddingmodels to increase embedding accuracy, which will beexamined in Section B.
An extensive survey on graph embedding methods will beconducted in the next section.
III. CLASSICAL METHODS
A) Dimension-Reduction-Based MethodsClassical graph embedding methods aim to reduce thedimension of high-dimensional graph data into a lowerdimensional representation while preserving the desiredproperties of the original data. They can be categorized intolinear and nonlinear two types. The linear methods includethe following.
• Principal Component Analysis (PCA) [51]The basic assumption for PCA is that that principalcomponents that are associated with larger variances rep-resent the important structure information while thosesmaller variances represent noise. Thus, PCA computes
4 FENXIAO CHEN, YUNCHENG WANG, BIN WANG AND C.-C. JAY KUO
the low-dimensional representation that maximizes thedata variance. Mathematically, it first finds a linear trans-formation matrix W ∈ RD×d by solving
W = argmaxTr(WTCov(X)W ), d = 1, 2, · · · , D,(4)
where Cov(X) denotes the covariance of data matrix X .It is well known that the principal components are orthog-onal and they can be solved by eigen decomposition ofthe covariance of data matrix [90].• Linear Discriminant Analysis (LDA) [105]
The basic assumption for LDA is that each class is Gaus-sian distributed. Then, the linear projection matrix, W ∈RD×d, can be obtained by maximizing the ratio betweenthe inter-class scatter and intra-class scatters. The maxi-mization problem can be solved by eigen decompositionand the number of low dimension d can be obtained bydetecting a prominent gap in the eigen-value spectrum.• Multidimensional Scaling (MDS) [75]
MDS is a distance-preserving manifold learning method.It preserves spatial distances. MDS derives a dissimi-larity matrix D, where Di,j represents the dissimilaritybetween points i and j, and produces a mapping in alower dimension to preserve dissimilarities as much aspossible.
The above three-mentioned methods are referred to as“subspace learning" [100] under the linear assumption.However, linear methods might fail if the underlying dataare highly non-linear [78]. Then, non-linear dimension-ality reduction (NLDR) [24] can be used for manifoldlearning. The objective is to learn the nonlinear topologyautomatically. The NLDR methods include the following.
• Isometric Feature Mapping (Isomap) [77]The Isomap finds low-dimensional representation thatmost accurately preserves the pairwise geodesic dis-tances between feature vectors in all scales as measuredalong the submanifold from which they were sampled.Isomap first constructs neighborhood graph on the mani-fold, then it computes the shortest path between pairwisepoints. Finally it constructs low-dimensional embeddingby applying MDS.• Locally Linear Embedding (LLE) [76]
LLE preserves the local linear structure of nearby fea-ture vectors. LLE first assign neighbors to each datapoint. Then it compute the weightsW i,j that best linearlyreconstruct Xi from its neighbors. Finally it compute thelow-dimensional embedding that best reconstructed byW i,j . Besides NLDR, kernel PCA is another dimensionreduction technique that is comparable to Isomap, LLE.• Kernel Methods [49]
Kernel extension can be applied to algorithms that onlyneed to compute the inner product of data pairs. Afterreplacing the inner product with kernel function, data ismapped implicitly from the original input space to higherdimensional space and then apply linear algorithm in thenew feature space. The benefit of kernel trick is data thatare not linearly separable in the original space could be
separable in new high dimensional space. Kernel PCAis often used for NLDR with polynomial or Gaussiankernels.
B) Random-Walk-Based MethodsRandom-walk-based methods sample a graph with a largenumber of paths by starting walks from random initialnodes. These paths indicate the context of connected ver-tices. The randomness of walks gives the ability to explorethe graph and capture global as well as local structuralinformation by walking through neighboring vertices. Afterthat, probability models such as skip-gram and bag-of-word can be performed on these randomly sampled pathsto learn the node representation.
• DeepWalk [73]DeepWalk is the most popular random-walk-based graphembedding method. In DeepWalk, a target vertex, vi, issaid to belong to a sequence S = {v1, · · · , v|s|} sam-pled by random walks if vi can reach any vertex in Swithin a certain number of steps. The set of vertices,Vs = {vi−t, · · · , vi−1, vi+1, · · · , vi+t}, is the context ofcenter vertex vi with a window size of t. DeepWalk aimsto maximize the average logarithmic probability of allvertex context pairs in random walk sequence S. It canbe written as
1
|S|
|S|∑i=1
∑−t≤j≤t,j 6=0
log p(vi+j |vi), (5)
where p(vj |vi) is calculated using the softmax function.It is proven in [101] that DeepWalk is equivalent tofactoring a matrix
M = WT ×H, (6)
where M ∈ R|V |×|V | whose entry, Mij , is the logarithmof the average probability that vertex vi can reach ver-tex vj in a fixed number of steps and W ∈ Rk×|V | is thevertex representation. The information in H ∈ Rk×|V | israrely utilized in the classical DeepWalk model.
• node2vec [46]node2vec is a modified version of DeepWalk. In Deep-Walk, sampled sequences are based on DFS (Depth-firstSampling) strategy. They consist of neighboring nodessampled at increasing distances from the source nodesequentially. However, if the contextual sequences aresampled by the DFS strategy alone, only few verticesclose to the source node will be sampled. Consequently,the local structure will be easily overlooked. In contrastwith the DFS strategy, the BFS (Breadth-first Sampling)strategy will explore neighboring nodes with a restrictedmaximum distance to the source node while the globalstructure may be neglected.As a result, node2vec proposes a probability model inwhich the random walk has a certain probability, 1
p , torevisit nodes being traveled before. Furthermore, it usesan in-out parameter q to control the ability of exploring
GRAPH REPRESENTATION LEARNING: A SURVEY 5
the global structure. When return parameter p is small,the random walk may get stuck in a loop and capture thelocal structure only. When in-out parameter q is small, therandom walk is more similar to a DFS strategy and capa-ble of preserving the global structure in the embeddingspace.
C) Matrix-Factorization-Based MethodsTo obtain node embedding, matrix-factorization-basedembedding methods, also called Graph Factorization (GF)[3], was the first one to achieve graph embedding inO(|E|)time. To obtain the embedding, GF factorizes the adjacencymatrix of a graph. It corresponds to a structure-preservingdimensionality reduction process. There are several varia-tions as summarized below.
• Graph Laplacian Eigenmaps [6]This technique minimizes a cost function to ensure thatpoints close to each other on the manifold are mappedclose to each other in the low-dimensional space topreserve local distances.• Node Proximity Matrix Factorization [81]
This method approximates node proximity in a low-dimensional space via matrix factorization by minimizingthe following objective function:
Y = argmax min |W − Y Y T |, (7)
where W is the node proximity matrix, which can bederived by several methods. One way to obtain W is touse Eq. (6).• Text-Associated DeepWalk (TADW) [101]
TADW is an improved DeepWalk method for text data. Itincorporates the text features of vertices in network rep-resentation learning via matrix factorization. Recall thatthe entry, mij , of matrix M ∈ R|V |×|V | denotes the log-arithm of the average probability that vertex vi randomlywalks to vertex vj . Then, TADW factorizes M into threematrices
M = WT ×H × T, (8)
where W ∈ Rk×|V |, H ∈ Rk×ft and T ∈ Rft×|V | is thetext feature matrix. In TADW, W and HT are concate-nated as the representation for vertices.• Homophily, Structure, and Content Augmented (HSCA)
Network [111]The HSCA model is an improvement upon the TADWmodel. It uses Skip-Gram and hierarchical Softmax tolearn a distributed word representation. The objectivefunction for HSCA can be written as
minW,H
(||M −WTHT ||2F +λ
2(||W ||2F + ||H||2F )
+µ(R1(W ) +R2(H))),
(9)
where ||.||2 is the matrix l2 norm and ||.||F is the matrixFrobenius form. In Eq. (9), the first term aims to mini-mize the matrix factorization error of TADW. The second
term imposes the low-rank constraint on W and H anduses λ to control the trade-off. The last regularizationterm enforces the structural homophily between con-nected nodes in the network. The conjugate gradient (CG)[69] optimization technique can be used to update Wand H . We may consider another regularization term toreplace the third term; namely,
R(W,H) =1
4
|V |∑i=1,j=1
Ai,j ||[wiHti
]−[wjHtj
]||22. (10)
This term will make connected nodes close to each otherin the learned network representation [17].
• GraRep [13]GraRep aims to preserve the high order proximity ofgraphs in the embedding space. While the random-walkbased methods have a similar objective, their probabilitymodel and objective functions used are difficult to explainhow the high order proximity are preserved. GraRepderives a k-th order transition matrix,Ak, by multiply theadjacency matrix to itself k times. The transition proba-bility from vertex w to vertex c is the entry in the w-throw and c-th column of the k-th order transition matrix.Mathematically, it can be written as
pk(c|w) = Akw,c. (11)
With the transition probability defined in (11), the lossfunction is defined by the skip-gram model and negativesampling. To minimize the loss function, the embeddingmatrix can be expressed as
Y ki,j = W ki · Ckj = log(
Aki,j∑tA
kt,j
)− log(β), (12)
where β is a constant λN , λ is the negative sampling
parameter, and N is the number of vertices. The embed-ding matrix, W , can be obtained by factorizing matrix Yin (12).
• HOPE [72]HOPE preserves asymmetric transitivity in approximat-ing the high order proximity, where asymmetric transitiv-ity indicates a certain correlation among directed graphs.Generally speaking, if there is a directed edge from uto v, it is likely that there is a directed edge from v tou as well. Several high order proximities such as theKatz Index [53], the Rooted PageRank, the CommonNeighbors, and the Adamic-Adar were experimented in[72]. The embedding, ~vi, for node i can be obtained byfactorizing the proximity matrix, S, derived from theseproximities. To factorize S, SVD is adopted and only thetop-k eigenvalues are chosen.
6 FENXIAO CHEN, YUNCHENG WANG, BIN WANG AND C.-C. JAY KUO
IV. EMERGING METHODS
A) Neural-Network-Based MethodsNeural network models become popular again since 2010.Being inspired by the success of recurrent neural net-works (RNNs) and convolutional neural networks (CNNs),researchers attempt to generalize and apply them to graphs.Natural language processing (NLP) models often usethe RNN to find a vector representation for words. TheWord2Vec [66] and the skip-gram models [68] aim to learnthe continuous feature representation of words by opti-mizing a neighborhood preserving likelihood function. Byfollowing this idea, one can adopt a similar approach forgraph embedding, leading to the Node2Vec method [46].Node2Vec utilizes random walks [82] with a bias to samplethe neighborhood of a target node and optimizes its repre-sentation using stochastic gradient descent (SGD). Anotherfamily of neural-network-based embedding methods adoptCNN models. The input is either paths sampled from agraph or the whole graph itself. Some use the original CNNmodel designed for the Euclidean domain and reformatthe input graph to fit it. Others generalize the deep neuralmodel to non-Euclidean graphs.
Several neural-network-based Methods based graphembedding methods are presented below.
• Graph Convolutional Network (GCN) [56]GCN allows end-to-end learning of the graph with arbi-trary size and shape. This model uses convolution opera-tor on the graph and iteratively aggregates the embeddingof neighbors for nodes. This approach is widely used forsemi-supervised learning on graph-structured data thatis based on an efficient variant of convolutional neuralnetworks that operates directly on graphs and it learnshidden layer representations that encode both local graphstructure and features of nodes. In the first step of theGCN, a node sequence will be selected. The neighbor-hood nodes will be assembled, then the neighborhoodmight be normalized to impose order of the graph, thenconvolutional layers will be used to learn representationof nodes and edges. The propagation rule used is
f(H(l), A) = σ(D−12AD−
12H(l)W (l)), (13)
whereA is the identity matrix, with enforced self loops toinclude the node features of itself we get ˆA= A+ I , I isthe identity matrix and ˆD is the diagonal node degreematrix of ˆA to normalize the neighbors of A. Underspectral graph theory of CNNs on graphs, GCN is equiv-alent to Graph Laplacian in the non-euclidean domain[23]. The decomposition of eigenvalues for the Normal-ized Graph Laplacian data can also be used for tasks suchas classification and clustering. However, GCN usuallyonly uses two convolutional layer and why it works its notwell explained. One recent work showed that GCN modelis a special from of Laplacian smoothing [63] [32]. Thisis the reason that GCN works, and also more than two
convolutional layers will lead to over-smoothing, there-fore making making the features of nodes similar to eachother and more difficult to separate from each other.
• Signed Graph Convolutional Network (SGCN) [25]Most GCNs operate on unsigned graphs, however, manylinks in real world have negative links. To solve thisproblem, signed GCNs aims to learn graph representa-tion with the additional signed link information. Negativelinks usually contains semantic information that is dif-ferent from positive links, also the principles are inher-ently different from positive links. The signed networkwill have a different representation as G = (V,E+, E−)where the signs of the edges are differentiated. The aggre-gation for positive and negative links are different. Eachlayer will have two representations, one for the balanceduser where the number of negative links is even and onefor the unbalanced user where the number of negativelinks is odd. The hidden states are
hB(1)i = σ(WB(l)[
∑j∈N(i)+
h0j
|N+i |, h
(0)i ]), (14)
hU(1)i = σ(WU(l)[
∑j∈N(i)−
h0j
|N−i |, h
(0)i ]), (15)
where σ() is the non-linear activation function, (WB(l)
and (WU(l) are the linear transformation matrices forbalanced and unbalanced sets.
• Variational Graph Auto-Encoders (VGAE) [55]An autoencoder minimizes the reconstruction error of theinput and the output using an encoder and a decoder. Theencoder maps input data to a representation space. Then,it is further mapped to a reconstruction space that pre-serves the neighborhood information. VGAE uses GCNas the encoder and an inner product decoder to embedgraphs.
• GraphSAGE [47]GraphSAGE uses a sample and aggregate method toconduct inductive node embedding with node featuressuch as text attributes, node profiles, etc. It trains a setof aggregation functions that integrate features of localneighborhood and pass it to the target node i. Then, thehidden state of node i is updated by
h(k+1)i = ReLU(W (k)h
(k)i ,∑
n∈N(i)(ReLU(Q(k)h(k)n ))),
(16)where h(0)i = Xi is the initial node attributes and
∑(.)
denotes a certain aggregator function, e.g., average,LSTM, max-pooling, etc.
• Structural Deep Network Embedding (SDNE) [92]SDNE learns a low-dimensional network-structure-preserving representation by considering both the first-order and the second-order proximities between vertexesusing CNNs. To achieve this objective, it adopts a semi-supervised model to minimize the following objective
GRAPH REPRESENTATION LEARNING: A SURVEY 7
function:
||(X −X)�B||2
F + α
n∑i,j=1
si,j ||yi − yj ||22 + vLreg,
(17)where Lreg is an L2-norm regularizing term to avoidoverfitting, S is the adjacency matrix, and B is the biasmatrix.
B) Large Graph Embedding MethodsTo address the scalability issue, several embedding meth-ods targeting at large graphs have been proposed recently.They are examined in this subsection.
Fig. 1. Illustration of a learnable graph convolutional layer (LGCL) method[35].
Fig. 2. Illustration of the sub-graph selection process [35].
• Learnable graph convolutional layer (LGCL) [35]For each feature dimension, every node in the LGCLmethod selects a fixed number of features from its neigh-boring nodes with value ranking. Fig. 1 serves as anexample. Each node in this figure has a feature vectorof dimension n = 3. For the target node (in orange), thefirst feature component of its six neighbors takes the val-ues of 9, 6, 5, 3, 0, 0. If we set the window size to k = 4,
then the four largest values (i.e., 9, 6, 5, 3) are selected.The same process is repeated for the two remaining fea-tures. By including the feature vector of the target nodeitself, we obtain a data matrix of dimension (k + 1)× n.This results in a grid-like structure. Then, the traditionalCNN can be conveniently applied so as to generate thefinal feature vector. To embed large-scale graphs, a sub-graph selection method is used to reduce the memoryand resource requirements. As shown in Fig. 2, it beginswithNinit = 3 randomly sampled nodes (in red) that arelocated in the center of the figure. At the first iteration,the breadth-first-search (BFS) is used to find all first-order neighboring nodes of initial nodes. Among them,Nm = 5 nodes (in blue) are randomly selected. At thenext iteration, Nm = 7 nodes (in green) are randomlyselected. After two iterations, 15 nodes are selected asa sub-graph that serves as the input to LGCL.
• Graph partition neural networks (GPNN) [70]GPNN extends graph neural networks (GNNs) to embedextremely large graphs. It alternates between local (prop-agate information among nodes) and global informa-tion propagation (messages among subgraphs). Thisscheduling method can avoid deep computational graphsrequired by sequential schedules. The graph partition isdone using a multi-seed flood fill algorithm, where nodeswith large out-degrees are sampled randomly as seeds.The subgraphs grow from seeds using flood fill, whichreaches out unassigned nodes that are direct neighbors ofthe current subgraph.
• LINE [85]LINE is used to embed graphs of an arbitrary type such asundirected, directed and weighted graphs. It utilizes neg-ative sampling to reduce optimization complexity. Thisis especially useful in embedding networks containingmillions of nodes and billions of edges. It is trained topreserve the first- and second-order proximities, sepa-rately. Then, the two embeddings are merged to generatea vector space to better represent the input graph. Oneway to merge two embeddings is to concatenate embed-ding vectors trained by two different objective functionsat each vertex.
C) Hypergraph EmbeddingResearch on social network embedding grows quickly. Asimple graph is not powerful enough in representing theinformation of social networks. The relation of vertices insocial networks is far more complicated than the vertex-to-vertex edge relationship. Being different from traditionalgraphs, edges in hypergraphs may have a degree larger thantwo. All related nodes are connected by a hyperedge toform a supernode. Mathematically, an unweighted hyper-graph is defined as follows. A hypergraph, denoted byG = (V,E), consists of a vertex set
V = {v1, v2, ..., vn},
8 FENXIAO CHEN, YUNCHENG WANG, BIN WANG AND C.-C. JAY KUO
and a hyperedge set,
E = {e1, e2, ..., em}.
A hyperedge, e, is said to be incident with a vertex vif v ∈ e. When v ∈ e, the incidence function h(v, e) = 1.Otherwise, h(v, e) = 0. The degree of a vertex v is definedas
d(v) =∑
e∈E,v∈eh(v, e).
Similarly, the degree of a hyperedge e is defined as
d(e) =∑v∈V
h(v, e).
A hypergraph can be represented by an incidence matrixHof dimension |V | × |E| with entries h(v, e).
Hyperedges possess the properties of edges and nodesat the same time. As an edge, hyperedges connect multi-ple nodes that are closely related. A hyperedge can alsobe seen as a supernode. For each pair of two supernodes,their connection is established by shared incident vertices.As a result, hypergraphs can better indicate the communitystructure in the network data. These unique characteris-tics of hyperedges make hypergraphs more challenging.An illustration of graph and hypergraph structures is givenin Fig. 3. It shows how to express a hypergraph in tableform. The hyperedges, which are indecomposable [89], canexpress the community structure of networks. Furthermore,properties of graphs and hypergraphs are summarized andcompared in Table 1. Graphs and hypergraphs conversiontechniques have been developed. Examples include cliqueexpansion and star expansion. Due to indecomposibility ofhyperedges, conversion from a hypergraph to a graph willresult in information loss.
Fig. 3. Illustration of graph and hypergraph structures [31].
Since hypergraphs provide a good tool for social net-work modeling, and hypergraph embedding is a hotresearch topic nowadays. On one hand, hypergraph model-ing has a lot of applications that are difficult to achieve bygraph modeling such as multi-modal data representation.On the other hand, hypergraphs can be viewed as a vari-ant of simple graphs and many graph embedding methodscould be applied onto the hypergraphs with minor modifi-cations. There are embedding methods proposed for simple
optimization optimizationSpectral Embedding Matrix Project to
factorization eigenspaceTable 1. Comparison of properties of graphs and hypergraphs.
graphs and they can be applied to hypergraphs as well asreviewed below.
• Spectral Hypergraph Embedding [117]Hypergraph embedding can be treated as a k-way parti-tioning problem and solved by optimizing a combinato-rial function. It can be further converted to a real-valuedminimization problem by normalizing the hypergraphLaplacian. Its solution is any lower dimension embeddingspace spanned by orthogonal eigenvectors of the hyper-graph Laplacian, ∆, with the k smallest eigenvalues.
• Hyper-Graph Neural Network (HGNN) [31]Being inspired by the spectral convolution on graphs inGCN [56], HGNN applies the spectral convolution tohypergraphs. By training the network through a semi-supervised node classification task, one can obtain thenode representation at the end of convolutional layers.The architecture is depicted in Fig. 4. The hypergraphconvolution is derived from the hypergraph Laplacian, ∆,which is a positive semi-definite matrix. Its eigenvectorsprovide certain basis functions while its associated eigen-values are the corresponding frequencies. The spectralconvolution in each layer is carried out via
f(X,W,Θ) = σ(D−12v HWD−1e HTD
−12v XΘ), (18)
where X is the hidden embedding in each layer, Θ isthe filter response, and Dv and De are diagonal matri-ces with entries being the degree of the vertices and thehyperedges, respectively.
Fig. 4. The architecture of HGNN [31].
• Deep Hyper-Network Embedding (DHNE) [89]DHNE aims to preserve the structural information ofhyperedges by a deep neural auto-encoder. The auto-encoder first embed each vertex to a vector in a lowerdimensional latent space and then reconstruct it to theoriginal incidence vector afterwards. In the process ofencoding and decoding, the second order proximity ispreserved to learn the global structural information. The
GRAPH REPRESENTATION LEARNING: A SURVEY 9
first order proximity is preserved in the embedding spaceby defining an N -tuplewise similarity function. That is,if N nodes are in the same hyperedge, the similarityof these nodes in the embedding space should be high.Based on similarity, one can predict whether N nodesare connected by a single hyperedge. However, the N -tuplewise similarity function should be non-linear; oth-erwise, it will lead to contradicted predictions. The localinformation of a hypergraph can be preserved by shorten-ing the distance of connected vertices in the embeddingspace.
D) Attention Graph EmbeddingAn attention mechanism can be used in machine learning toallow the learning process to focus on parts of a graph thatare relevant to a specific task. One advantage of applyingattention to graphs is to avoid the noisy part of a graph soas to increase the signal-to-noise (SNR) ratio [60] in infor-mation processing. Attention-based node embedding aimsto assign an attention weight, αi ∈ [0, 1], to neighborhoodnodes of a target node t, where
∑i∈N(t) αi = 1 and N(t)
denotes the set of neighboring nodes of t.
• Graph Attention Networks (GAT) [91]GAT utilizes masked self-attentional layers to limit theshortcomings of prior graph convolutional based meth-ods. They aim to compute the attention coefficients
αij =exp(LeakyReLU(−→a T [W
−→hi ||W
−→hj ]))∑
k∈Niexp(LeakyReLU(−→a T [W
−→hi ||W
−→hj ]
,
(19)where W is the weight matrix for the initial linear trans-formation, then the transformed information on eachneighbor’s feature are concatenated to obtain the new hid-den state, which will be passed through a LeakyReLuactivation. The above attention mechanism is a single-layer feedforward neural network parameterized by theabove weight vector.• AttentionWalks [1],[2]
Generally speaking, one can use the random walk tofind the context of the node. For a graph, G, with cor-responding transition matrix T and window size c theparameterized conditional expectation after a k-step walkcan be expressed as
E[D|q1, q2, ..., qc] = In
c∑k=1
qkTk, (20)
where In is the size-n identity matrix, qk, 1 ≤ i ≤ c, arethe trainable weights, D is the walk distribution matrixwhose entry Duv encodes the number of times node u isexpected to visit node v. The trainable weights are used tosteer the walk towards a broader neighborhood or restrictit within a smaller neighborhood. Following this idea,AttentionWalks adopts an attention mechanism to guidethe learning procedure. This mechanism suggests whichpart of the data to focus on during the training process.
The weight parameters are called the attention parametersin this case.
• Attentive Graph-based Recursive Neural Network (AGRNN)[97]AGRNN applies attention to a graph-based recursiveneural network (GRNN) [98] to make the model focuson vertices with more relevant semantic information. Itbuilds subgraphs to construct recursive neural networksby sampling a number of k-step neighboring vertices.AGRNN finds a soft attention, αr, to control how neigh-bor information should be passed to the target node.Mathematically, we have
αr = Softmax(xTW (a)hr), (21)
where xk is the input, W (a) is the weight to learn andhr is the hidden state of the neighbors. The aggregatedrepresentation from all neighbors is used as the hiddenstate of the target vertex
hk =∑
vr∈N(vk)αrhr, (22)
whereN(vk) denotes the set of neighboring nodes of ver-tex vk. Although attention has been shown to be usefulin improving some neural network models, it does notalways increase the accuracy of graph embedding [16].
E) Others• GraphGAN [93]
GraphGAN employs both generative and discriminativemodels for graph representation learning. It adopts adver-sarial training and formulates the embedding problem asa minimax game by borrowing the idea from the Genera-tive Adversarial Network (GAN) [43]. To fit the true con-nectivity distribution ptrue(v|vc) of vertices connectedto target vertex vc, GraphGAN models the connectivityprobability among vertices in a graph with a generator,G(v|vc; θG), to generate vertices that are most likely con-nected to vc. A discriminator D(v, vc; θD) outputs theedge probability between v and vc to differentiate the ver-tex pair generated by the generator from the ground truth.The final vertex representation is determined by alter-nately maximizing and minimizing the value functionV (G,D) as
minθG
maxθD
V (G,D)
V (G,D) =
V∑c=1
(Ev∼ptrue(·|vc)[logD(v,vc;θD)])
+Ev∼G(·|vc;θG)[log(1−D(v, vc; θD)]).
(23)
• GenVector [104]GenVector leverages large-scale unlabeled data to learnlarge social knowledge graphs. It is cast as a weaklysupervised problem and solved by unsupervised tech-niques with a multi-modal Bayesian embedding model.
10 FENXIAO CHEN, YUNCHENG WANG, BIN WANG AND C.-C. JAY KUO
GenVector can serve as a generative model in applica-tions. For example, it uses latent discrete topic variablesto generate continuous word embeddings and graph-based user embeddings and integrates the advantages oftopic models and word embeddings.
V. EVALUATION
We study the evaluation of various graph representationmethods in this section. Evaluation tasks and datasetswill be discussed in Secs. A and B, respectively. Then,evaluation results will be presented and analyzed in Sec.C.
A) Evaluation TasksThe two most common evaluation tasks are vertex classi-fication and link prediction. We use vertex classificationto compare different graph embedding methods and drawinsights from the obtained results.
• Vertex ClassificationVertex classification aims to assign a class label to eachnode in a graph based on the information learned fromother labeled nodes. Intuitively, similar nodes shouldhave the same label. For example, closely-related pub-lication may be labeled as the same topic in the citationgraph while individuals of the same gender, similar ageand common interests may have the same preference insocial networks. Graph embedding methods embed eachnode into a low-dimensional vector. Given an embed-ded vector, a trained classifier can predict the label ofa vertex of interest, where the classifier can be SVM(Support Vector Machine) [42], logistic regression [94],kNN (k nearest neighbors) [59], etc. The vertex labelcan be obtained in an unsupervised or semi-supervisedway. Node clustering is an unsupervised method thatgroups similar nodes together. It is useful when labelsare unavailable. The semi-supervised method can be usedwhen part of the data are labeled. The F1 score is usedfor evaluation in binary-class classification while theMicro-F1 score is used in multi-class classification. Sinceaccurate vertex representations contribute to high clas-sification accuracy, vertex classification can be used tomeasure the performance of different graph embeddingmethods.• Link Prediction [36]
Link prediction aims to infer the existence of relation-ship or interaction among pairs of vertices in a graph.The learned representation should help infer the graphstructure, especially when some links are missing. Forexample, links might be missing between two users andlink prediction can be used to recommend friends insocial networks. The learned representation should pre-serve the network proximity and the structural similarityamong vertices. The information encoded in the vectorrepresentation for each vertex can be used to predict miss-ing links in incomplete networks. The link prediction
performance can be measured by the area under curve(AUC) or the receiver operating characteristic (ROC)curve. A better representation should be able to capturethe connections among vertices better.
We describe the benchmark graph datasets and conductexperiments in vertex classification on both small and largedatasets in the following subsections.
B) Evaluation DatasetsCitation datasets such as Citeseer [39], Cora [10] andPubMed [12] are examples of small datasets. They canbe represented as directed graphs in which edges indicatesauthor-to-author or paper-to-paper citation relationship andtext attributes of paper content at nodes.
First, we describe several representative citation datasetsbelow.
• Citeseer [39]It is a citation index dataset containing academic papersof six categories. It has 3,312 documents and 4,723 links.Each document is represented by a 0/1-valued word vec-tor indicating the absence/presence of the correspondingword from a dictionary of 3,703 words. Thus, the textattributes of a document is a binary-valued vector of3,703 dimensions.
• Cora [10]It consists of 2,708 scientific publications of sevenclasses. The graph has 5,429 links that indicate citationrelations between documents. Each document has textattributes that are expressed by a binary-valued vector of1,433 dimensions.
• WebKB [88]It contains seven classes of web pages collected fromcomputer science departments, including student, fac-ulty, course, project, department, staff, etc. It has 877web pages and 1,608 hyper-links between web pages.Each page is represented by a binary vector of 1,703dimensions.
• KARATE [109]Zachary’s karate network is a well-known social networkof a university karate club. It has been widely studied insocial network analysis. The network has 34 nodes, 78edges and 2 communities.
• Wikipedia [19]The Wikipedia is an online encyclopedia created andedited by volunteers around the world. The dataset is aword co-occurrence network constructed from the entireset of English Wikipedia pages. This data contains 2405nodes, 17981 edges and 19 labels.
Next, we present several commonly used large graphdatasets below.
• Blogcatalog [84]It is a network of social relationships of bloggers listedin the BlogCatalog website. The labels indicate blogger’s
GRAPH REPRESENTATION LEARNING: A SURVEY 11
interests inferred from the meta-data provided by blog-gers. The network has 10,312 nodes, 333,983 edges and39 labels.• Youtube [95]
It is a social network of Youtube users. This graph con-tains 1,157,827 nodes, 4,945,382 edges and 47 labels.The labels represent groups of users who enjoy commonvideo genres.• Facebook [62]
It is a set of postings collected from the Facebook web-site. This data contains 4039 nodes, 88243 edges and nolabels. It is used for link prediction.• Flickr [80]
It is an online photo management and sharing dataset. Itcontains 80513 nodes, 5899882 edges and 195 labels.
Finally, parameters of the above-mentioned datasets aresummarized in Table 2.
C) Evaluation Results and AnalysisSince evaluations were often performed independently ondifferent datasets under different settings in the past, it isdifficult to draw a concrete conclusion on the performanceof various graph embedding methods. Here, we comparethe performance of graph embedding methods using acouple of metrics under the common setting and analyzeobtained results. In addition, we provide an open-sourcePython library, called the Graph Representation LearningLibrary (GRLL), to readers in the Github. It offers a uni-fied interface for all graph embedding methods that wereexperimented in this work. To the best of our knowledge,this library covers the largest number of graph embeddingtechniques up to now.
1) Vertex ClassificationWe compare vertex classification accuracy of seven graphembedding methods on Cora and Wiki. We used the defaulthyper-parameter setting provided by each graph embed-ding method. For the classifier, we adopted linear regres-sion for all methods. We split samples equally into thetraining and the testing sets (i.e. 50% and 50%). The ver-tex classification results are shown in Table 3. DeepWalkand node2vec offer the highest accuracy for Cora and Wiki,
respectively. The random-walk-based methods (e.g., Deep-Walk, node2vec and GraRep) are the top three performersfor both Cora and Wiki. DeepWalk and node2vec are pre-ferred among them since GraRep usually demands muchmore memory and it is difficult to apply GraRep to largergraphs.
Table 3. Performance comparison of seven graph embedding methodsin vertex classification on Cora and Wiki.
2) Clustering QualityWe compare various graph embedding methods by exam-ining their clustering quality in terms of the Macro- andMicro-F1 scores. The K-means++ algorithm is adopted forthe clustering task. Since the results of K-means++ clus-tering are dependent upon seed initialization, we perform10 consecutive runs and report the best result. We testedthem on three large graph datasets (i.e., YouTube, Flickrand BlogCatalog). The experimental results are shown inTable 4. YouTube and Flickr contain more than millions ofnodes and edges and we can only run DeepWalk, node2vecand LINE on them with the 24G RAM limit as reportedin the table. We see that DeepWalk and node2vec providethe best results. They are both random-walk based methodswith different sampling scheme. Also, they demand lessmemory as compared with others. In general, random walkwith the skip-gram model is a good baseline for unsuper-vised graph embedding. GraRep offers comparable graphembedding quality for BlogCatalog. However, its mem-ory requirement is huge so that it is not suitable for largegraphs.
3) Time ComplexityTime complexity is an important factor to consider, whichis especially true for large graphs. The time complexity ofthree embedding methods against three datasets is com-pared in Table 5. We see that the training time of DeepWalkis significantly lower than node2vec and LINE for largergraph datasets such as YouTube and Flickr. DeepWalkis an efficient graph embedding method with high accu-racy by considering embedding quality as well as trainingcomplexity.
4) Influence of Embedding DimensionsAs the embedding dimension decreases, less information ofthe input graph is preserved so that the performance drops.However, some drops faster than others. We show thenode classification accuracy as a function of the embedding
Micro-F1 0.393 0.400 0.356 0.393 0.236 0.308Table 4. Comparison of clustering quality of six graph embedding methods in terms of Macro- and Micro-F1 scores against three large graph datasets.
Table 5. Comparison of time used in training (seconds).
dimension for the Wiki dataset in Fig. 5. We compare sixgraph embedding methods (node2vec, DeepWalk, LINE,GraRep, SDNE and GF) and their embedding dimensionsvary from 4, 8, 16, 32, 64 to 128. We see that the per-formance of the random-walk based embedding methods(node2vec and Deep-Walk) degrades slowly. Only about20% drop in performance when the embedding dimensionsize goes from 128 to 4. In contrast, The performance ofthe structural preserving methods (LINE and GreRep) andthe graph factorization method (GF) drops significantly (asmuch as 45%) when the embedding size goes from 128 to4. One explanation is that the structural preserving methodsoptimize the representation vectors in the embedding spaceso that a small information loss will result in substantial dif-ference. Random-walk based methods obtain embeddingvectors by selecting paths from the input graph randomly.Yet, the relationship between nodes is still preserved whenthe embedding dimension is small. SDNE adopts the auto-encoder architecture to preserve the information of theinput graph so that its performance remains about the sameregardless of the embedding dimension.
Fig. 5. The node classification accuracy as a function of the embeddingdimension for the Wiki dataset.
Fig. 6. The node classification accuracy as a function of the training sampleratio for the Cora dataset.
5) Influence of Training Sample RatioBy the training sample ratio, we mean the percentages oftotal graph samples that are used for the training purpose.When the ratio is high, the classifier could be overfit. On theother hand, if the ratio is too low, the offered informationmay not be sufficient for the training purpose. Such anal-ysis is classifier dependent, and we adopt a simple linearregression classifier from the python sklearn toolkit in theexperiment. The node classification accuracy as a functionof the training sample ratio for the Cora dataset is shownin Fig. 6. Most methods have consistent performance forthe training data ratio between 0.2 and 0.8 except for themachine learning based methods (SDNE and GCN). Theiraccuracy drops when the training data ratio is low. Theyneed a larger number of training data.
VI. EMERGING APPLICATIONS
Graphs offer a powerful modeling tool and find a widerange of applications. Since many real-world data havecertain relationship between entities, they can be conve-niently modeled by graphs. Multi-modal data can also beembedded into the same space through graph representa-tion learning and, as a result, the information from differentdomains can be represented and analyzed in one commonsetting.
In this section, we examine three emerging areas thatbenefit from graph embedding techniques.
GRAPH REPRESENTATION LEARNING: A SURVEY 13
• Community DetectionGraph embedding can be used to predict the label of anode given a fraction of labeled node [15], [26], [108],[113]. Thus, it has been widely used for communitydetection [27], [71]. In social networks, node labels mightbe gender, demography or religion. In language networks,documents might be labeled with categories or keywords.Missing labels can be inferred from labeled nodes andlinks in the network. Graph embedding can be used toextract node features automatically based on the networkstructure and predict the community that a node belongsto. Both vertex classification [40] and link prediction [20][114] can facilitate community detection [33], [102].• Recommendation System
Recommendation is an important function in socialnetworks and advertising platforms [48], [106], [112].Besides the structure, content and label data [54], somenetworks contain spatial and temporal information. Forexample, Yelp may recommend restaurants based onuser’s location and preference. Spatial-temporal embed-ding [110] is an emerging topic in mobile applications.• Graph Compression and Coarsening
By graph compression (graph simplification), we referto a process of converting one graph to another, wherethe latter has a smaller number of edges. It aims to storea graph more efficiently and run graph analysis algo-rithms faster. For example, a graph is partitioned intobipartite cliques and replaced by trees to reduce the edgenumber in [30]. Along this line, one can also aggregatenodes or edges for faster processing with graph coarsen-ing [64], where a graph is converted into smaller onesrepeatedly using a hybrid matching technique to main-tain its backbone structure. The Structural EquivalenceMatching (SEM) method [45] and the Normalized HeavyEdge Matching method (NHEM) [52] are two examples.
VII. FUTURE RESEARCH DIRECTIONS
Several future research opportunities in graph embeddingare discussed below.
• Deep Graph EmbeddingGCN [56] has drawn a lot of attention due to its superiorperformance. However, the number of graph convolu-tional layers in typically not greater than two. Whenthere are more graph convolutional layers in cascade, itis surprising that the performance drops significantly. Itwas argued in [63] that each GCN layer corresponds tograph Laplacian smoothing since node features are prop-agated in the spectral domain. When a GCN is deeper, thegraph Laplacian is over-smoothed and the correspondingnode features become obscure. Yet, each layer of GCNonly learns one-hop information, and two GCN layerslearn the first and second-order proximity in the graph.It is difficult for a shallow structure to learn the globalinformation. The receptive field of each filter in GCNis global since graph convolution is conducted in the
spectral domain. One solution to fix this problem is toconduct the convolution in the spatial domain. For exam-ple, one can convert graph data into grid-structure dataas proposed in [35]. Then, the graph representation canbe learned using multiple CNN layers. Another way toaddress the problem is to down-sample graphs and mergesimilar nodes together. Then, we can build a hierachicalnetwork structure, which allows to learn both local andglobal graph data in a hierarchical manner. Such a graphcoarsening idea was adopted by [34] [50], [107] to builddeep GCNs.
• Semi-supervised Graph EmbeddingClassical graph embedding methods such as PCA, Deep-Walk and matrix factorization are unsupervised learn-ing methods. They use topological structures and nodeattributes to generate graph representations. No labelsare required in the training process. However, labels areuseful since they provide more information about thegraph. In most real world applications, labels are avail-able in a subset of nodes, leading to a semi-supervisedlearning problem. The feasibility for semi-supervisedlearning on graphs was discussed in [103], [116]. Alarge number of graph embedding problems belong to thesemi-supervised learning paradigm, and it deserves ourattention.
• Dynamic Graph EmbeddingSocial graphs such as the twitter are constantly chang-ing. Another example is graphs of mobile users whoselocation information is changing along with time. Tolearn the representation of dynamic graphs is an impor-tant research topic and it finds applications in real-timeand interactive processes such as the optimal travel pathplanning in a city at traffic hours.
• Scalability of Graph EmbeddingWe expect to see graphs of a larger scale and higher diver-sity because of the rapid growth of social networks, whichcontain millions and billions of nodes and edges. It isstill an open problem how to embed very large graph dataefficiently and accurately.
• Interpretability of Graph EmbeddingMost state-of-the-art graph embedding methods are builtupon CNNs, which are trained with backpropagation(BP) to determine their model parameters. However, thetraining complexity is very high. Besides, CNNs aremathematically intractable. Research was done to lowerthe training complexity such as quickprop [28]. Further-more, there is new work [58] that attempts to explainCNNs using an interpretable and feedforward (FF) designwithout any BP. The work in [58] adopts a data-centricapproach to network parameters of the current layerbased on data statistics from the output of the previouslayer in a one-pass manner.
VIII. CONCLUSION
A comprehensive survey of the literature on graph repre-sentation learning techniques was conducted in this paper.
14 FENXIAO CHEN, YUNCHENG WANG, BIN WANG AND C.-C. JAY KUO
We examined various graph embedding techniques thatconvert the input graph data into a low-dimensional vec-tor representation while preserving intrinsic graph prop-erties. Besides classical graph embedding methods, wecovered several new topics such as neural-network-basedembedding methods, hypergraph embedding and attentiongraph embedding methods. Furthermore, we conducted anextensive performance evaluation of several stat-of-the-art methods against small and large datasets. For experi-ments conducted in our evaluation, an open source Pythonlibrary, called the Graph Representation Learning Library(GRLL), was provided to readers. Finally, we presentedsome emerging applications and future research directions.
R E F E R E N C E S
[1] Abu-El-Haija, S., Perozzi, B., Al-Rfou, R., and Alemi, A. (2017).Watch your step: Learning graph embeddings through attention. arXivpreprint arXiv:1710.09599.
[2] Abu-El-Haija, S., Perozzi, B., Al-Rfou, R., and Alemi, A. A. (2018).Watch your step: Learning node embeddings via graph attention. InAdvances in Neural Information Processing Systems, pages 9180–9190.
[3] Ahmed, A., Shervashidze, N., Narayanamurthy, S., Josifovski, V., andSmola, A. J. (2013). Distributed large-scale natural graph factorization.In Proceedings of the 22nd international conference on World WideWeb, pages 37–48. ACM.
[4] Angles, R. and Gutierrez, C. (2008). Survey of graph databasemodels. ACM Computing Surveys (CSUR), 40(1):1.
[5] Anis, A., Gadde, A., and Ortega, A. (2016). Efficient sampling setselection for bandlimited graph signals using graph spectral proxies.IEEE Trans. Signal Processing, 64(14):3775–3789.
[6] Belkin, M. and Niyogi, P. (2003). Laplacian eigenmaps for dimen-sionality reduction and data representation. Neural computation,15(6):1373–1396.
[7] Bhagat, S., Cormode, G., and Muthukrishnan, S. (2011). Node clas-sification in social networks. In Social network data analytics, pages115–148. Springer.
[8] Bizer, C., Lehmann, J., Kobilarov, G., Auer, S., Becker, C., Cyganiak,R., and Hellmann, S. (2009). Dbpedia-a crystallization point for theweb of data. Web Semantics: science, services and agents on the worldwide web, 7(3):154–165.
[9] Bollacker, K., Evans, C., Paritosh, P., Sturge, T., and Taylor, J.(2008). Freebase: a collaboratively created graph database for struc-turing human knowledge. In Proceedings of the 2008 ACM SIGMODinternational conference on Management of data, pages 1247–1250.AcM.
[10] Cabanes, C., Grouazel, A., Schuckmann, K. v., Hamon, M., Turpin,V., Coatanoan, C., Paris, F., Guinehut, S., Boone, C., and Ferry, N.(2013). The cora dataset: validation and diagnostics of in-situ oceantemperature and salinity measurements. Ocean Science, 9(1):1–18.
[11] Cai, H., Zheng, V. W., and Chang, K. C.-C. (2018). A comprehensivesurvey of graph embedding: Problems, techniques, and applications.IEEE Transactions on Knowledge and Data Engineering, 30(9):1616–1637.
[12] Canese, K. and Weis, S. (2013). Pubmed: the bibliographic database.In The NCBI Handbook [Internet]. 2nd edition. National Center forBiotechnology Information (US).
[13] Cao, S., Lu, W., and Xu, Q. (2015). Grarep: Learning graph rep-resentations with global structural information. In Proceedings of the
24th ACM international on conference on information and knowledgemanagement, pages 891–900. ACM.
[14] Cavallari, S., Zheng, V. W., Cai, H., Chang, K. C.-C., and Cambria,E. (2017). Learning community embedding with community detectionand node embedding on graphs. In Proceedings of the 2017 ACM onConference on Information and Knowledge Management, pages 377–386. ACM.
[15] Chen, D. (2009). A novel clustering algorithm for graphs. In 2009International Conference on Artificial Intelligence and ComputationalIntelligence, volume 4, pages 279–283. IEEE.
[16] Chen, F., Wang, B., and Kuo, C.-C. J. (2018). Graph-based deep-treerecursive neural network (dtrnn) for text classification. arXiv preprintarXiv:1809.01219.
[17] Chen, F., Wang, B., and Kuo, C.-C. J. (2019). Deepwalk-assistedgraph pca (dgpca) for language networks. In ICASSP 2019-2019 IEEEInternational Conference on Acoustics, Speech and Signal Processing(ICASSP), pages 2957–2961. IEEE.
[18] Chrobak, M. and Payne, T. H. (1995). A linear-time algorithm fordrawing a planar graph on a grid. Information Processing Letters,54(4):241–246.
[19] Cucerzan, S. (2007). Large-scale named entity disambiguation basedon wikipedia data. In Proceedings of the 2007 Joint Conference onEmpirical Methods in Natural Language Processing and Computa-tional Natural Language Learning (EMNLP-CoNLL), pages 708–716.
[20] De Bacco, C., Power, E. A., Larremore, D. B., and Moore, C. (2017).Community detection, link prediction, and layer interdependence inmultilayer networks. Physical Review E, 95(4):042317.
[21] de Fraysseix, H., Pach, J., and Pollack, R. (1988). Small sets support-ing fary embeddings of planar graphs. In Proceedings of the twentiethannual ACM symposium on Theory of computing, pages 426–433.ACM.
[22] De Fraysseix, H., Pach, J., and Pollack, R. (1990). How to draw aplanar graph on a grid. Combinatorica, 10(1):41–51.
[23] Defferrard, M., Bresson, X., and Vandergheynst, P. (2016). Con-volutional neural networks on graphs with fast localized spectralfiltering. In Advances in neural information processing systems, pages3844–3852.
[24] DeMers, D. and Cottrell, G. W. (1993). Non-linear dimensional-ity reduction. In Advances in neural information processing systems,pages 580–587.
[25] Derr, T., Ma, Y., and Tang, J. (2018). Signed graph convolutionalnetworks. In 2018 IEEE International Conference on Data Mining(ICDM), pages 929–934. IEEE.
[26] Ding, C. H., He, X., Zha, H., Gu, M., and Simon, H. D. (2001).A min-max cut algorithm for graph partitioning and data clustering.In Proceedings 2001 IEEE International Conference on Data Mining,pages 107–114. IEEE.
[27] Dourisboure, Y., Geraci, F., and Pellegrini, M. (2007). Extractionand classification of dense communities in the web. In Proceedings ofthe 16th international conference on World Wide Web, pages 461–470.ACM.
[28] Fahlman, S. E. (1988). An empirical study of learning speed in back-propagation networks.
[29] Fáry, I. (1948). On straight-line representation of planar graphs. ActaSci. Math., 11:229–233.
[30] Feder, T. and Motwani, R. (1995). Clique partitions, graph com-pression and speeding-up algorithms. Journal of Computer and SystemSciences, 51(2):261–272.
[31] Feng, Y., You, H., Zhang, Z., Ji, R., and Gao, Y. (2018). Hypergraphneural networks. arXiv preprint arXiv:1809.09401.
GRAPH REPRESENTATION LEARNING: A SURVEY 15
[32] Field, D. A. (1988). Laplacian smoothing and delaunay triangula-tions. Communications in applied numerical methods, 4(6):709–712.
[33] Fortunato, S. (2010). Community detection in graphs. Physicsreports, 486(3-5):75–174.
[34] Gao, H., Chen, Y., and Ji, S. (2019). Learning graph pooling andhybrid convolutional operations for text representations. arXiv preprintarXiv:1901.06965.
[35] Gao, H., Wang, Z., and Ji, S. (2018). Large-scale learnable graphconvolutional networks. In Proceedings of the 24th ACM SIGKDDInternational Conference on Knowledge Discovery & Data Mining,pages 1416–1424. ACM.
[36] Gao, S., Denoyer, L., and Gallinari, P. (2011). Temporal link predic-tion by integrating content and structure information. In Proceedings ofthe 20th ACM international conference on Information and knowledgemanagement, pages 1169–1174. ACM.
[37] Gargi, U., Lu, W., Mirrokni, V., and Yoon, S. (2011). Large-scalecommunity detection on youtube for topic discovery and exploration.In Fifth International AAAI Conference on Weblogs and Social Media.
[38] Gilbert, A. C. and Levchenko, K. (2004). Compressing networkgraphs. In Proceedings of the LinkKDD workshop at the 10th ACMConference on KDD, volume 124.
[39] Giles, C. L., Bollacker, K. D., and Lawrence, S. (1998). Citeseer: Anautomatic citation indexing system. In ACM DL, pages 89–98.
[40] Girvan, M. and Newman, M. E. (2002). Community structure insocial and biological networks. Proceedings of the national academyof sciences, 99(12):7821–7826.
[41] Goldberger, J., Gordon, S., and Greenspan, H. (2003). An efficientimage similarity measure based on approximations of kl-divergencebetween two gaussian mixtures. In null, page 487. IEEE.
[42] Golub, G. H. and Reinsch, C. (1971). Singular value decomposi-tion and least squares solutions. In Linear Algebra, pages 134–151.Springer.
[43] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley,D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative adver-sarial nets. In Advances in neural information processing systems,pages 2672–2680.
[44] Goyal, P. and Ferrara, E. (2018). Graph embedding techniques,applications, and performance: A survey. Knowledge-Based Systems,151:78–94.
[45] Green, S. B. and Yang, Y. (2009). Reliability of summed item scoresusing structural equation modeling: An alternative to coefficient alpha.Psychometrika, 74(1):155–167.
[46] Grover, A. and Leskovec, J. (2016). node2vec: Scalable featurelearning for networks. In Proceedings of the 22nd ACM SIGKDD inter-national conference on Knowledge discovery and data mining, pages855–864. ACM.
[47] Hamilton, W., Ying, Z., and Leskovec, J. (2017a). Inductive repre-sentation learning on large graphs. In Advances in Neural InformationProcessing Systems, pages 1024–1034.
[48] Hamilton, W. L., Ying, R., and Leskovec, J. (2017b). Represen-tation learning on graphs: Methods and applications. arXiv preprintarXiv:1709.05584.
[49] Harandi, M. T., Sanderson, C., Shirazi, S., and Lovell, B. C. (2011).Graph embedding discriminant analysis on grassmannian manifolds forimproved image set matching. In CVPR 2011, pages 2705–2712. IEEE.
[50] Hu, F., Zhu, Y., Wu, S., Wang, L., and Tan, T. (2019). Semi-supervised node classification via hierarchical graph convolutionalnetworks. arXiv preprint arXiv:1902.06667.
[51] Jolliffe, I. (2011). Principal component analysis. Springer.
[52] Karypis, G. and Kumar, V. (1998). Multilevelk-way partitioningscheme for irregular graphs. Journal of Parallel and Distributed com-puting, 48(1):96–129.
[53] Katz, L. (1953). A new status index derived from sociometricanalysis. Psychometrika, 18(1):39–43.
[54] Kermarrec, A.-M., Leroy, V., and Trédan, G. (2011). Distributedsocial graph embedding. In Proceedings of the 20th ACM interna-tional conference on Information and knowledge management, pages1209–1214. ACM.
[55] Kingma, D. P. and Welling, M. (2013). Auto-encoding variationalbayes. arXiv preprint arXiv:1312.6114.
[56] Kipf, T. N. and Welling, M. (2016). Semi-supervised classificationwith graph convolutional networks. arXiv preprint arXiv:1609.02907.
[57] Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). Imagenetclassification with deep convolutional neural networks. In Advances inneural information processing systems, pages 1097–1105.
[58] Kuo, C.-C. J., Zhang, M., Li, S., Duan, J., and Chen, Y. (2018). Inter-pretable convolutional neural networks via feedforward design. arXivpreprint arXiv:1810.02786.
[59] Le, T. M. and Lauw, H. W. (2014). Probabilistic latent documentnetwork embedding. In 2014 IEEE International Conference on DataMining, pages 270–279. IEEE.
[60] Lee, J. B., Rossi, R. A., Kim, S., Ahmed, N. K., and Koh,E. (2018). Attention models in graphs: A survey. arXiv preprintarXiv:1807.07984.
[61] Leskovec, J. and Faloutsos, C. (2006). Sampling from large graphs.In Proceedings of the 12th ACM SIGKDD international conference onKnowledge discovery and data mining, pages 631–636. ACM.
[62] Lewis, K., Kaufman, J., Gonzalez, M., Wimmer, A., and Christakis,N. (2008). Tastes, ties, and time: A new social network dataset usingfacebook. com. Social networks, 30(4):330–342.
[63] Li, Q., Han, Z., and Wu, X.-M. (2018). Deeper insights into graphconvolutional networks for semi-supervised learning. In Thirty-SecondAAAI Conference on Artificial Intelligence.
[64] Liang, J., Gurukar, S., and Parthasarathy, S. (2018). Mile: Amulti-level framework for scalable graph embedding. arXiv preprintarXiv:1802.09612.
[65] Liben-Nowell, D. and Kleinberg, J. (2007). The link-predictionproblem for social networks. Journal of the American society forinformation science and technology, 58(7):1019–1031.
[66] Mikolov, T., Chen, K., Corrado, G., and Dean, J. (2013a). Effi-cient estimation of word representations in vector space. arXiv preprintarXiv:1301.3781.
[67] Mikolov, T., Karafiát, M., Burget, L., Cernocky, J., and Khudan-pur, S. (2010). Recurrent neural network based language model. InEleventh annual conference of the international speech communicationassociation.
[68] Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., and Dean,J. (2013b). Distributed representations of words and phrases andtheir compositionality. In Advances in neural information processingsystems, pages 3111–3119.
[69] Møller, M. F. (1993). A scaled conjugate gradient algorithm for fastsupervised learning. Neural networks, 6(4):525–533.
[70] Motsinger, A. A., Lee, S. L., Mellick, G., and Ritchie, M. D. (2006).Gpnn: Power studies and applications of a neural network method fordetecting gene-gene interactions in studies of human disease. BMCbioinformatics, 7(1):39.
[71] Newman, M. E. (2004). Detecting community structure in networks.The European Physical Journal B, 38(2):321–330.
16 FENXIAO CHEN, YUNCHENG WANG, BIN WANG AND C.-C. JAY KUO
[72] Ou, M., Cui, P., Pei, J., Zhang, Z., and Zhu, W. (2016). Asymmetrictransitivity preserving graph embedding. In Proceedings of the 22ndACM SIGKDD international conference on Knowledge discovery anddata mining, pages 1105–1114. ACM.
[73] Perozzi, B., Al-Rfou, R., and Skiena, S. (2014). Deepwalk: Onlinelearning of social representations. In Proceedings of the 20th ACMSIGKDD international conference on Knowledge discovery and datamining, pages 701–710. ACM.
[74] Ribeiro, B. and Towsley, D. (2010). Estimating and sampling graphswith multidimensional random walks. In Proceedings of the 10thACM SIGCOMM conference on Internet measurement, pages 390–403.ACM.
[75] Robinson, S. L. and Bennett, R. J. (1995). A typology of deviantworkplace behaviors: A multidimensional scaling study. Academy ofmanagement journal, 38(2):555–572.
[76] Roweis, S. T. and Saul, L. K. (2000). Nonlinear dimensionalityreduction by locally linear embedding. science, 290(5500):2323–2326.
[77] Samko, O., Marshall, A. D., and Rosin, P. L. (2006). Selection of theoptimal parameter value for the isomap algorithm. Pattern RecognitionLetters, 27(9):968–979.
[78] Saul, L. K., Weinberger, K. Q., Ham, J. H., Sha, F., and Lee, D. D.(2006). Spectral methods for dimensionality reduction. Semisupervisedlearning, pages 293–308.
[79] Shaw, B. and Jebara, T. (2009). Structure preserving embedding. InProceedings of the 26th Annual International Conference on MachineLearning, pages 937–944. ACM.
[80] Sigurbjörnsson, B. and Van Zwol, R. (2008). Flickr tag recom-mendation based on collective knowledge. In Proceedings of the 17thinternational conference on World Wide Web, pages 327–336. ACM.
[81] Singh, A. P. and Gordon, G. J. (2008). Relational learning via col-lective matrix factorization. In Proceedings of the 14th ACM SIGKDDinternational conference on Knowledge discovery and data mining,pages 650–658. ACM.
[82] Spitzer, F. (2013). Principles of random walk, volume 34. SpringerScience & Business Media.
[83] Stein, S. K. (1951). Convex maps. Proceedings of the AmericanMathematical Society, 2(3):464–466.
[84] Tang, J. and Liu, H. (2012). Unsupervised feature selection forlinked social media data. In Proceedings of the 18th ACM SIGKDDinternational conference on Knowledge discovery and data mining,pages 904–912. ACM.
[85] Tang, J., Qu, M., Wang, M., Zhang, M., Yan, J., and Mei, Q. (2015).Line: Large-scale information network embedding. In Proceedings ofthe 24th international conference on world wide web, pages 1067–1077. International World Wide Web Conferences Steering Committee.
[86] Taylor, P., Black, A. W., and Caley, R. (2001). Heterogeneous rela-tion graphs as a formalism for representing linguistic information.Speech Communication, 33(1-2):153–174.
[87] Theocharidis, A., Van Dongen, S., Enright, A. J., and Freeman, T. C.(2009). Network visualization and analysis of gene expression datausing biolayout express 3d. Nature protocols, 4(10):1535.
[88] Trochim, W. M. and Donnelly, J. P. (2001). Research methodsknowledge base, volume 2. Atomic Dog Publishing Cincinnati, OH.
[89] Tu, K., Cui, P., Wang, X., Wang, F., and Zhu, W. (2017). Structuraldeep embedding for hyper-networks. CoRR, abs/1711.10146.
[90] Umeyama, S. (1988). An eigendecomposition approach to weightedgraph matching problems. IEEE transactions on pattern analysis andmachine intelligence, 10(5):695–703.
[91] Velickovic, P., Cucurull, G., Casanova, A., Romero, A., Lio, P.,and Bengio, Y. (2017). Graph attention networks. arXiv preprintarXiv:1710.10903.
[92] Wang, D., Cui, P., and Zhu, W. (2016). Structural deep networkembedding. In Proceedings of the 22nd ACM SIGKDD internationalconference on Knowledge discovery and data mining, pages 1225–1234. ACM.
[93] Wang, H., Wang, J., Wang, J., Zhao, M., Zhang, W., Zhang, F.,Xie, X., and Guo, M. (2018). Graphgan: graph representation learningwith generative adversarial nets. In Thirty-Second AAAI Conference onArtificial Intelligence.
[94] Wang, X., Cui, P., Wang, J., Pei, J., Zhu, W., and Yang, S. (2017).Community preserving network embedding. In Thirty-First AAAI Con-ference on Artificial Intelligence.
[95] Wattenhofer, M., Wattenhofer, R., and Zhu, Z. (2012). The youtubesocial network. In Sixth International AAAI Conference on Weblogsand Social Media.
[96] Xu, K., Feng, Y., Huang, S., and Zhao, D. (2015). Semantic relationclassification via convolutional neural networks with simple negativesampling. arXiv preprint arXiv:1506.07650.
[97] Xu, Q., Wang, Q., Xu, C., and Qu, L. (2017a). Attentive graph-based recursive neural network for collective vertex classification. InProceedings of the 2017 ACM on Conference on Information andKnowledge Management, pages 2403–2406. ACM.
[98] Xu, Q., Wang, Q., Xu, C., and Qu, L. (2017b). Collective ver-tex classification using recursive neural network. arXiv preprintarXiv:1701.06751.
[99] Yan, S., Xu, D., Zhang, B., and Zhang, H.-J. (2005). Graph embed-ding: A general framework for dimensionality reduction. In 2005IEEE Computer Society Conference on Computer Vision and PatternRecognition (CVPR’05), volume 2, pages 830–837. IEEE.
[100] Yan, S., Xu, D., Zhang, B., Zhang, H.-J., Yang, Q., and Lin, S.(2007). Graph embedding and extensions: A general framework fordimensionality reduction. IEEE Transactions on Pattern Analysis &Machine Intelligence, (1):40–51.
[101] Yang, C., Liu, Z., Zhao, D., Sun, M., and Chang, E. (2015a).Network representation learning with rich text information. In Twenty-Fourth International Joint Conference on Artificial Intelligence.
[102] Yang, T., Jin, R., Chi, Y., and Zhu, S. (2009). Combining linkand content for community detection: a discriminative approach. InProceedings of the 15th ACM SIGKDD international conference onKnowledge discovery and data mining, pages 927–936. ACM.
[103] Yang, Z., Cohen, W. W., and Salakhutdinov, R. (2016). Revisit-ing semi-supervised learning with graph embeddings. arXiv preprintarXiv:1603.08861.
[104] Yang, Z., Tang, J., and Cohen, W. (2015b). Multi-modal bayesianembeddings for learning social knowledge graphs. arXiv preprintarXiv:1508.00715.
[105] Ye, J., Janardan, R., and Li, Q. (2005). Two-dimensional lineardiscriminant analysis. In Advances in neural information processingsystems, pages 1569–1576.
[106] Ying, R., He, R., Chen, K., Eksombatchai, P., Hamilton, W. L., andLeskovec, J. (2018a). Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDDInternational Conference on Knowledge Discovery & Data Mining,pages 974–983. ACM.
[107] Ying, Z., You, J., Morris, C., Ren, X., Hamilton, W., and Leskovec,J. (2018b). Hierarchical graph representation learning with differen-tiable pooling. In Advances in Neural Information Processing Systems,pages 4800–4810.
GRAPH REPRESENTATION LEARNING: A SURVEY 17
[108] Yuruk, N., Mete, M., Xu, X., and Schweiger, T. A. (2009).Ahscan: Agglomerative hierarchical structural clustering algorithm fornetworks. In 2009 International Conference on Advances in SocialNetwork Analysis and Mining, pages 72–77. IEEE.
[109] Zachary, W. W. (1977). An information flow model for conflictand fission in small groups. Journal of anthropological research,33(4):452–473.
[110] Zhang, C., Zhang, K., Yuan, Q., Peng, H., Zheng, Y., Hanratty, T.,Wang, S., and Han, J. (2017). Regions, periods, activities: Uncover-ing urban dynamics via cross-modal representation learning. In Pro-ceedings of the 26th International Conference on World Wide Web,pages 361–370. International World Wide Web Conferences SteeringCommittee.
[111] Zhang, D., Yin, J., Zhu, X., and Zhang, C. (2016a). Homophily,structure, and content augmented network representation learning. InData Mining (ICDM), 2016 IEEE 16th International Conference on,pages 609–618. IEEE.
[112] Zhang, F., Yuan, N. J., Lian, D., Xie, X., and Ma, W.-Y. (2016b).Collaborative knowledge base embedding for recommender systems.In Proceedings of the 22nd ACM SIGKDD international conference onknowledge discovery and data mining, pages 353–362. ACM.
[113] Zhang, Y. and Wu, B. (2015). Finding community structure viarough k-means in social network. In 2015 IEEE International Con-ference on Big Data (Big Data), pages 2356–2361. IEEE.
[114] Zheleva, E., Getoor, L., Golbeck, J., and Kuter, U. (2008). Usingfriendship ties and family circles for link prediction. In InternationalWorkshop on Social Network Mining and Analysis, pages 97–113.Springer.
[115] Zhou, C., Liu, Y., Liu, X., Liu, Z., and Gao, J. (2017). Scal-able graph embedding for asymmetric proximity. In Thirty-First AAAIConference on Artificial Intelligence.
[116] Zhou, D., Bousquet, O., Lal, T. N., Weston, J., and Schölkopf, B.(2004). Learning with local and global consistency. In Advances inneural information processing systems, pages 321–328.
[117] Zhou, D., Huang, J., and Schölkopf, B. (2007). Learning withhypergraphs: Clustering, classification, and embedding. In Advancesin neural information processing systems, pages 1601–1608.
