Lovász Convolutional Networks

Prateek YadavIISc Bangalore

Madhav NimishakaviIISc Bangalore

Naganand YadatiIISc Bangalore

Shikhar VashishthIISc Bangalore

Arun RajkumarIIT Madras

Partha TalukdarIISc Bangalore

AbstractSemi-supervised learning on graph structureddata has received significant attention withthe recent introduction of Graph Convolu-tion Networks (GCN). While traditional meth-ods have focused on optimizing a loss aug-mented with Laplacian regularization frame-work, GCNs perform an implicit Laplaciantype regularization to capture local graphstructure. In this work, we propose LovászConvolutional Network (LCNs) which are ca-pable of incorporating global graph proper-ties. LCNs achieve this by utilizing Lovász’sorthonormal embeddings of the nodes. Weanalyse local and global properties of graphsand demonstrate settings where LCNs tendto work better than GCNs. We validatethe proposed method on standard randomgraph models such as stochastic block mod-els (SBM) and certain community structurebased graphs where LCNs outperform GCNsand learn more intuitive embeddings. We alsoperform extensive binary and multi-class clas-sification experiments on real world datasetsto demonstrate LCN’s effectiveness. In addi-tion to simple graphs, we also demonstratethe use of LCNs on hyper-graphs by identify-ing settings where they are expected to workbetter than GCNs.

1 IntroductionLearning on structured data has received significantinterest in recent years (Getoor and Taskar, 2007; Sub-ramanya and Talukdar, 2014). Graphs are ubiquitous,several real world data-sets can be naturally repre-sented as graphs; knowledge graphs (Suchanek et al.,

Proceedings of the 22nd International Conference on Ar-tificial Intelligence and Statistics (AISTATS) 2019, Naha,Okinawa, Japan. PMLR: Volume 89. Copyright 2019 bythe author(s).

2007; Auer et al., 2007; Bollacker et al., 2008), proteininteraction graphs (Zitnik and Leskovec, 2017), socialnetwork graphs (Leskovec et al., 2010b,a; McAuley andLeskovec, 2012), citation networks (Giles et al., 1998;Lu and Getoor, 2003; Sen et al., 2008) to name a few.These graphs typically have a large number of nodesand manually labeling them as belonging to a certainclass is often prohibitive in terms of resources needed.A common approach is to pose the classification prob-lem as a semi-supervised graph transduction problemwhere one wishes to label all the nodes of a graph usingthe labels of a small subset of nodes.

Recent approaches to the graph transduction problemrely on the assumption that the labels of nodes arerelated to the structure of the graph. A common ap-proach is to use the Laplacian matrix associated with agraph as form of a structural regularizer for the learningproblem. While the Laplacian regularization is doneexplicitly in (Agarwal, 2006; Zhu et al., 2003; Zhouet al., 2004; Belkin et al., 2006; Yang et al., 2016),more recent deep learning based Graph ConvolutionNetwork (GCN) approaches do an implicit Laplaciantype regularization (Atwood and Towsley, 2016; Kipfand Welling, 2017; Li et al., 2018; Zhuang and Ma,2018). While these traditional methods work reason-ably well for several real world problems, our extensiveexperiments show that they may not be the best meth-ods for tasks involving communities and there is a scopefor significant improvement in such cases.

In this work, we propose a graph convolutional networkbased approach to solve the semi-supervised learningproblem on graphs that typically have a communitystructure. An extensively studied model for commu-nities is the Stochastic block model (SBM) which isa random graph model where the nodes of a graphexhibit community structure i.e., the nodes belong-ing to same community have a larger probability ofhaving an edge between them than those in differentcommunities. In this work, we propose the LovászConvolutional Network (LCN) which, instead of the

Lovász Convolutional Networks

Kipf-GCN Embeddings LCN Embeddings

(a) Binary classification

Kipf-GCN Embeddings LCN Embeddings

(b) 3-class classification

Figure 1: Node embeddings for SBM Experiments. Note that these figures are obtained by projecting higher-dimensionalembeddings to lower-dimensional space using t-sne (van der Maaten and Hinton, 2008).

traditional Laplacian, uses the embeddings of nodesthat arise from Lovász’s orthogonal representations asan implicit regularizer. The Lovász regularization, aswe will see, is tightly coupled to the coloring of thecomplement graph of a given graph and hence oftenproduce remarkably superior embeddings than thoseobtained using the Laplacian regularization for graphswhich have a community structure. Intuitively, theoptimal coloring of the complement of a graph can beviewed as a way to associate same color to nodes belong-ing to a same community. As Lovász embeddings alsotend to embed nodes with same colors to similar pointsin Euclidean space, the proposed model performs muchwell in practice. Figure 1a and Figure 1b illustrate thisphenomenon using examples for a binary and a threeclass classification problem where the graph is gener-ated using a stochastic block model. As can be seen,the average distance between embeddings learnt usingthe LCN is much better than using traditional graphbased convolution networks. We make the followingcontributions in this work:• We propose the Lovász Convolutional Network(LCN) for the problem of semi-supervised learningon graphs. LCN combines the power of using theLovász embeddings with GCNs.

• We analyze various types of graphs and identifythe classes of graphs where LCN performs muchbetter than existing methods. In particular, wedemonstrate that by keeping the optimal coloring,a global property of the graph, fixed and increasingthe number edges to the graph, LCNs outperformstraditional GCNs.

• We carry out extensive experiments on both syn-thetic and real world datasets and show significantimprovement using LCNs than state of the art al-gorithms for semi-supervised graph transduction.

The Source code for our model can be found athttps://github.com/malllabiisc/lcn.

2 Related WorkThe work that is most closely related to ours is (Shiv-anna et al., 2015) which proposes a spectral regularizedorthogonal embedding method for graph transduction(SPORE). While they use a Lovász embedding basedkernel for explicit regularization, the focus is on comput-ing the embedding efficiently using a special purposeoptimization routine. Our work on the other handproposes a deep learning based Lovász convolutionalnetwork which differs from the traditional loss plusexplicit regularizer approach of (Shivanna et al., 2015)and our experimental results confirm that the pro-posed LCN approach performs significantly better thanSPORE. The use of explicit Laplacian regularizer forsemi-supervised learning problems on graphs has beenexplored in (Ando and Zhang, 2007; Agarwal, 2006),where the focus is to derive generalization bounds forlearning on graphs. However, as we will discuss in thesequel, there are settings where Lovász embeddings aremore natural in capturing the global property of graphsthan the Laplacian embeddings and this reflects in ourexperimental results as well. More recently (Zhuangand Ma, 2018) propose a dual convolution approachto capture global graph property using positive point-wise mutual information (PPMI). We differ from thisapproach in defining global property in terms of color-ing of the complement graph as opposed to computingsemantic similarity using random walks on the graphas done in (Zhuang and Ma, 2018). Lovász based ker-nels for graphs have been explored in the context ofother machine learning problems such as clustering in(Johansson et al., 2014). Jethava et al. (2013) show aninteresting connection between Lovász ϑ function andone class SVMs.There has been considerable amount of work in ex-tending well established deep learning architecturesfor graphs. Bruna et al. (2014); Henaff et al. (2015);Duvenaud et al. (2015); Defferrard et al. (2016) ex-tend Convolutional Neural Networks (CNN) for graphs,while Jain et al. (2016) propose Recurrent Neural Net-

Yadav, Nimishakavi, Yadati, Vashihth, Rajkumar, Talukdar

works (RNN) for graphs. Kipf and Welling (2017)propose Graph Convolutional Networks which achievepromising results for the problem of semi-supervisedclassification on graphs. Most recently, a faster versionof GCN, for inductive learning on graphs, has beenproposed by (Chen et al., 2018). An extension to GCNsbased on graph partition is proposed recently by (Liaoet al., 2018). Recently, GCNs with confidence scoresfor embeddings has been proposed by (Vashishth et al.,2019). GCNs have been shown to be effective for sev-eral tasks Marcheggiani and Titov (2017); Vashishthet al. (2018a,b,c); Ray et al. (2018). However, as weshow in our experiments, there are several natural set-tings where the proposed LCN performs much betterthan the state of the art GCNs in various problems ofinterest.

3 Problem Setting and PreliminariesWe work in the semi-supervised graph transductionsetting where we are given a graph G(V,E), where Vdenotes the set of vertices with cardinality n and E isthe edge set. We are given the labels ({0, 1} in the caseof binary classification) of a subset of nodes (m < n) ofV and the goal is to predict the labels of the remainingnodes as accurately as possible. Given a graph G(V,E),α(G) denotes the maximum independence number ofthe graph i.e., the size of the set containing the maxi-mum number of non-adjacent nodes in G. A coloringof G corresponds to an assignment of colors to nodesof the graph such that no two nodes with the samecolor have an edge between them. χ(G) denotes thechromatic number of G which is the minimum numberof colors needed to color G. We denote the complementof a graph by G(V, E). An edge (u, v) is present in Gif and only if it is not present in G. It is easily seenthat for any graph G, α(G) ≤ χ(G). A clique is a fullyconnected graph which has edges between all pairs ofnodes. We assume that there is a natural underlyingmanner in which the graph structure is related to theclass labels. In what follows, we recall certain classesof graphs and a certain type of graph embedding whichwill be of interest in the rest of the paper.SBM Graphs: The Stochastic Block Model (SBM)(Holland et al., 1983; Condon and Karp, 1999) is agenerative model for random graphs. They are a gen-eralization of the Erdos-Renyi graphs where the edgesbetween nodes of the same community are chosen witha certain probability (p) while the edges across com-munities are chosen with a certain other probability (qwhere q < p). SBMs tend to have community struc-ture and hence are used to model several applicationsincluding protein interactions, social network analysisand have been extensively studied in machine learning,statistics, theoretical computer science and networkscience literature.

0

11

1

Figure 2: Lovász embeddings for a graph consisting of setcliques are mapped orthogonal dimensions. Refer Section 3for more details.

Perfect Graphs: Perfect graphs are a class of graphswhose chromatic number χ(G) equals the size of thelargest clique for every induced subgraph. Severalimportant class of graphs including bipartite graphs,interval graphs, chordal graphs, caveman graphs etc.are all perfect graphs. We refer to (Ramírez-Alfonsínand Reed, 2001) for graph theoretical analysis of per-fect graphs. Our interest in these graphs is due to thefact that the the chromatic number of these graphs canbe computed in polynomial time (Lovász, 2009) andthey coincide with the Lovász ϑ number of the graphwhich we discuss next.Lovász Embeddings: Lovász (Lovász, 1979) intro-duced the concept of orthogonal embedding in the con-text of the problem of embedding a graph G = (V,E)on a unit sphere Sd−1.Definition 3.1 (Orthogonal embedding (Lovász, 1979;Lovász and Vesztergombi, 1999)). An orthogonal em-bedding of a graph G(V,E) with |V | = n, is a matrixU = [u1, . . . ,un] ∈ Rd×n such that u>i uj = 0 when-ever (i, j) /∈ E and ui ∈ Sd−1 ∀i ∈ [n].

Let Lab(G) denote the set of all possible orthogo-nal embeddings of the graph G, given by Lab(G) ={U|U is an orthogonal embedding}. The Lovász thetafunction is defined as:

ϑ(G) = minU∈Lab(G)

minc∈Sd−1

maxi

(c>ui)−2.

The famous sandwich theorem of Lovász (Lovász, 1979)states that α(G) ≤ ϑ(G) ≤ χ(G), where α(G) is theindependence number of the graph and χ(G) is thechromatic number of the complement of G. Perfectgraphs are of interest to us as both the above inequali-ties are equalities for them (Lovász, 2009).A few examples are helpful to gain intuition about therelation of Lovász embeddings to community structures.For a complete graph, the complement can be coloredusing just one color, the Lovász embeddings of all thenodes are trivial and in 1-dimension. These embed-dings are exactly the same as there are no orthogonal

Lovász Convolutional Networks

Color Fraction

Test

Acc

ura

cy

Figure 3: Variation of test accuracy (higher is better)for GCN and LCN- with variation in the graph structure.GCN fails to perform as the number color fraction increases.Refer Section 4 for more details.

constraints imposed by the edges. As a generalizationof this example (Figure 2), for a graph that is a dis-joint union of k cliques of possible variable number ofnodes in each clique, the complement is a complete kpartite graph and hence can be colored using k colorswhere each partition corresponds to a single color. Itturns out the Lovász embeddings for this graph area set of orthonormal vectors in Rk. In practice, thecommunities that occur are not exactly cliques i.e., notall edges in a community are connected to each other.However, the Lovász embeddings still capture the nec-essary structure as we will see in our experiments.Graph convolutional networks (GCN): GCNs(Kipf and Welling, 2017) extend the idea of Convo-lutional Neural Networks (CNNs) for graphs. LetG(V,E) be an undirected graph with adjacency ma-trix A and let A = A + I be the adjacency withadded self-connections and Dii =

∑j Aij . Let

X ∈ Rn×d represent the input feature matrix ofthe nodes. A simple two-layer GCN for the prob-lem of semi-supervised node classification assumes theform : f(X,A) = softmax(A ReLU(AXW(0))W(1)).Where, A = D

− 12 AD

− 12 , W(0) ∈ Rd×h is an input-

to-hidden weight matrix for a hidden layer with hunits and W(1) ∈ Rh×F is hidden-to-output weightmatrix. The softmax activation function, defined assoftmax(xi) = 1

Z exp(xi) with Z =∑i exp(xi) is ap-

plied row-wise.

For semi-supervised multi-class classification, cross-entropy loss over the labeled examples is given by

L =∑l∈YL

F∑f=1

Ylf ln Zlf , (1)

where, YL is the set of labeled nodes. The weightsW(0) and W(1) are learnt using gradient descent.

4 Motivating ExampleIn this section we present a motivating example todemonstrate the use of the Lovász orthogonal embed-dings in the semi-supervised graph transduction task.In particular, we want to show how the embeddingslearnt using the Lovász kernel results in improved ac-curacy as a parameter called coloring fraction, whichwe define below, varies. To illustrate our hypothesis,we consider a bipartite graph as input to the problem.The reason for this choice is that bipartite graphs areperfect and hence optimal coloring of both the graphG and its complement G (which is also perfect by theperfect graph theorem (Chudnovsky et al., 2006)) areeasy to compute in polynomial time. Before explainingthe experiment, we start with the following definition.Coloring Fraction: Given a graph G = (V,E), con-sider the optimal coloring of the complement graph G.According to this coloring scheme of the nodes, let ndrepresent the number of edges in G such that the pairof nodes each edge connects have different colors. Andlet nt represent the total number of pairs of nodes in Gsuch that the nodes in each pair have different colors.Then the coloring fraction is defined as nd/nt.As an example, for a complete bipartite graph G =K(n, n) on 2n nodes, the complement graph is theunion of 2 disjoint cliques of n nodes each and hence thegraph can be colored using n colors. The coloring frac-tion is then n(n−1)

2(n−1)(n) = 0.5. The following propositionestablishes how coloring fraction varies with removalof edges from a graph.Proposition 1. Let G(V,E) be a graph where χ(G)is the chromatic number of the complement of G. Letβ(G) be the coloring fraction of G. Let G′ be the graphobtained from G by removing a set of edges whose nodeshave different colors with respect to the optimal coloringof G. Then χ(G′) = χ(G) whereas β(G′) < β(G).

Proof. It should be observed that the optimal coloringfor G is also a valid coloring of G′ as the edges removedfrom G are only from nodes with different colors withrespect to coloring of G. To see why it is also anoptimal coloring, we use contradiction. If there existsa coloring of G′ with strictly smaller number of colorsthan χ(G), then we can remove edges to form G′ toobtain G such that it is also a valid coloring of G asremoving edges does not affect the validity of a coloring.However, this contradicts the optimality of the originalcoloring for G. Thus χ(G) = χ(G′). Moreover, sincewe are removing edges from G, the coloring fractionincreases by definition and hence β(G′) < β(G).The above proposition says that by removing edgescarefully, a local property of the graph (coloring frac-tion) changes whereas a global property (chromaticnumber of complement graph) does not change. If thelabels of nodes depends on the global property of the

Yadav, Nimishakavi, Yadati, Vashihth, Rajkumar, Talukdar

Algorithm 1: Lovasz Kernel Matrix ComputationInput: A, Adjacency matrix of Graph GOutput: K: Lovasz Kernel[SDP ]Y←minimize t, subject to:

Y � 0, Yij = −1, ∀(i, j) /∈ E, Yii = t− 1P ∈ Rn×n ← Cholesky(Y);if rank(P) < n then

c← random basis element from Null(P);ui = c+pi√

twhere pi is ith column of P;

endif rank(P) = n then

pi = [pi 0] ∈ Rn+1 ∀ i ∈ {1, 2, . . . , n};ui = en+1+pi√

twhere en+1 ∈ Rn+1 is the

standard basis element;endU = [u1u2 . . . un];K = U>U;

graph, then a natural question of interest is to studythe sensitivity of algorithms to change in the local prop-erty while keeping the global property fixed. This isprecisely what we do as we explain below.We begin with a complete bipartite graph K(n, n)whose coloring fraction as computed above is 0.5. We re-move m edges in each step where the nodes of removededges have different colors (w.r.t optimal coloring of G).In each case, the labels are assigned such that nodeswith half the colors are assigned to class 0 and remain-ing to class 1. We compute the accuracy of a Laplacianbased GCN model vs the proposed LCN model. In ourexperiment we set n = 50 and m = 250. The resultsaveraged over 10 random splits of 20% − 20% − 60%train-validation-test are presented in Figure 3. It isclear that as the color fraction increases, the accuracyof the standard GCN drops while that of Lovász doesnot. This is because the standard GCN depends onlocal connectivity property of the graph whereas theorthogonal labeling is done in accordance to the globalcoloring of the complement graph and is better cap-tured by the proposed LCN.The above example motivates our study of Lovász basedembeddings in cases where the global structure of thegraph is related to the class labels. With this motiva-tion, we propose the Lovász convolution network in thefollowing section.

5 LCN: Proposed ModelIn this section, we present our proposed method,the Lovász Convolution Network (LCN), for semi-supervised graph transduction. As motivated in theprevious section, when the class labels depend on thecoloring (a global property) of the given graph, it isnatural to start training a graph based convolutionnetwork which incorporates this property into learning.Let Lab(G), as defined in Section 3, represent the set of

all possible orthonormal embeddings for a given graphG. The set of graph kernel matrices is defined as

K(G) := {K ∈ S+n |Kii = 1,∀i ∈ [n];Kij = 0.∀(i, j) /∈ E},

where S+n is the set of all positive semidefinite matrices.

Jethava et al. (2013) showed the equivalence betweenLab(G) and K(G). Since K ∈ K(G) is positive semidef-inite, there exists a U ∈ Rd×n such that K = U>U.It should be noted that Kij = u>i uj , where ui isthe i-th column of U, which implies U ∈ Lab(G).Similarly, it can be shown that for any U ∈ Lab(G),K = U>U ∈ K(G). Given a graph G, we follow theprocedure described in (Lovász and Vesztergombi, 1999,Proposition 9.2.9) for computing the Lovász orthonor-mal embedding U and the associated kernel matrix Koptimally. The procedure is summarized in Algorithm1. Similar to the normalized Laplacian of a graph, thekernel matrix is also positive semidefinite.

The kernel computation explained in Algorithm 1 re-quires solving a Semi Definite Program (SDP), the com-putational complexity of which is O(n6). This becomesa huge bottle-neck for large scale datasets. Therefore,for large scale datasets, we exploit the following char-acterization of ϑ(G) given by Luz and Schrijver (2005):Theorem 5.1 (Luz and Schrijver (2005)). For a graphG = (V,E) with |V | = n, and let C ∈ Rn×n be anynon-null symmetric matrix with Cij = 0 whenever(i, j) /∈ E. Then,

ϑ(G) = minC

ν(G,C), where

ν(G,C) = maxx≥02x>e− x>( C

−λmin(C) + I)x,

where e = [1, 1, . . . , 1]> and λmin(C) is the minimumeigen value of C.

Note that the matrix KLS = A−λmin(A)

+ I obtainedby fixing C = A in Theorem 5.1 is positive semidefi-nite. Therefore, there exists a labeling U ∈ Rd×n suchthat U>U = KLS , which is referred to as LS labeling(Jethava et al., 2013). From Theorem 5.1, for any graphG, we have

ϑ(G) ≤ ν(G,A),

an upper bound on ϑ(G), and the equality holds for aclass of graphs called Q graphs (Luz, 1995). Computa-tion of KLS has a complexity of only O(n3), hence forlarge scale datasets we approximate the Lovász kernelby K = KLS .

We propose to use the following two layered architecturefor the problem of semi-supervised classification,

f(X,K) = softmax(K ReLU(KXW(0))W(1)). (2)

Lovász Convolutional Networks

0.5 0.50.5 0.5

0.55 0.450.45 0.50

0.6 0.50.5 0.6

0.7 0.30.3 0.7

0.6 0.4 0.40.4 0.6 0.40.4 0.4 0.6

0.70 0.4 0.450.40 0.6 0.300.45 0.3 0.50

0.7 0.6 0.50.6 0.7 0.40.5 0.4 0.6

0.7 0.6 0.60.6 0.7 0.60.6 0.6 0.6

Number of nodesNumber of nodes Number of nodes Number of nodes

Number of nodesNumber of nodesNumber of nodesNumber of nodes

Acc

urac

yA

ccur

acy

Acc

urac

yA

ccur

acy

Acc

urac

y

Acc

urac

y

Acc

urac

y

Acc

urac

y

(a) (b) (c) (d)

(h)(g)(f)(e)

Figure 4: Test accuracy plots for various synthetically generated graphs from stochastic block model. The matrix in plotdenotes the connection probabilities between classes.

Similar to GCN, we minimize the cross-entropy lossgiven in Equation (1) for semi-supervised multi-classclassification. We use batch gradient descent for learn-ing the weights W(0) and W(1).We note that when the class labels are a non-linearmapping of the optimal coloring of G, LCN with Lovászkernel K(G) tunes the weights of the network to learnthe mapping.

6 Experimental ResultsIn this section, we report the results of our experimentson several synthetic and real world datasets. We demon-strate the usefulness of the embeddings learnt using theLovász convolution networks over several state of theart methods including GCNs, SPORE, normalized andunnormalized laplacian based regularization along withother embeddings such as KS labelings that are de-scribed in (Shivanna et al., 2015). We demonstrate ourresults on Stochastic block models, real world MNISTdatasets (binary and multiclass) and several real worldUCI datasets. We also run experiments on large scalereal world datasets Citeseer, Cora and Pubmed whichare standard in GCN literature. In addition to this,we test the goodness of Lovász based embeddings incertain perfect graphs called caveman graphs whichhave been used to model simple social network commu-nities. In addition to simple graphs, we also experimentwith hypergraphs with clique expansion to see how theproposed method performs.

Stochastic Block Model: Synthetic Data Exper-iments: We start by describing our experiments onsynthetically generated stochastic block model graphs.We perform the experiment on binary as well as threeclass classification problem. We report several settingsof inter cluster and intra cluster probabilities in Fig-ure 4 and corresponding embeddings in Figure 5. Ineach of the experiments, the input features are fixed to

identity. We varied the number of nodes from 100 to1000 and used a 20%− 10%− 70% train-validation-testsplit where we use early stopping during training (Kipfand Welling, 2017). The test accuracy is comparedagainst the standard GCNs (denoted by Kipf-GCN).We make several observations from the results in Figure4. Firstly, as the inter and intra cluster probabilitiesget closer, it becomes much harder for GCN to classifywell whereas LCN outperforms GCN by a significantmargin. Secondly, as the size of the graph increases,the differences in connections become more critical andthis is reflected in the increased accuracy with increasein nodes for LCN, whereas accuracy of GCN is almostagnostic to the number of nodes. Finally, as the graphbecomes denser i.e., as connection probabilities tendtowards 1, LCN performs much better than GCN in thethree class setting. These results demonstrate the ad-vantage of using LCNs over GCNs for semi-supervisedclassification tasks for SBM models.

Real World Data Experiments: We run severalexperiments on real world datasets including MNISTand UCI datasets. To make a fair comparison withstate of the art, we first run the same set of binary clas-sification experiments as in (Shivanna et al., 2015) andcompare it with GCNs (Kipf and Welling, 2017), GraphPartition Neural Networks (GPNN) (Liao et al., 2018)and our proposed LCNs. These include experimentson 6 UCI datasets (breast-cancer n = 683, diabetesn = 768, fourclass n = 862, heart n = 270, ionospheren = 351 and sonar n = 208) and experiments on certainpair of classes from a subsampled set of images fromthe MNIST datasets. For both the UCI and MNISTdatasets features and labels are available, we used anRBF kernel on features to construct the graph. Table1 reports the results for these experiments with variousinput embeddings including Laplacian (normalized, un-normalized), KS embedding and others as considered

Yadav, Nimishakavi, Yadati, Vashihth, Rajkumar, Talukdar

Kipf-GCN Embeddings LCN Embeddings

(b)

Kipf-GCN Embeddings LCN Embeddings

(c)Kipf-GCN Embeddings LCN Embeddings

(e)

Kipf-GCN Embeddings LCN Embeddings

(f)

Figure 5: Embeddings learnt for settings corresponding to Figure 4 (b), (c), (e), (f) for n=1000

in (Shivanna et al., 2015). For MNIST datasets, theresults are averaged over five randomly sampled graphsand for UCI datasets, the results are averaged over fiverandom splits. As can be seen, LCN performs signifi-cantly better than SPORE and performs much betterthan GCNs in all datasets except two. In addition tothe binary classification experiment, we also conductedthree class classification on 500 and 2000 images fromMNIST (randomly subsampled from classes 1, 2 and7) and also 10 class classification where we randomlysubsample 2000 images from all classes. The resultsare reported in Table 2 As the classes increase, LCNsignificantly outperforms other GCN baselines.

To be consistent with the GCN literature, we alsorun experiments on large scale datasets Citeseer, Coraand Pubmed. All these datasets are citation networks,where each document is represented as a node in thegraph with an edge between nodes indicating the cita-tion relation. The aim is to classify the documents intoone of the predefined classes. We use the same splitsas in (Yang et al., 2016). Table 3 shows the resultson these large scale datasets, as explained in Section 5we use the approximate KLS kernel for these datasetsand LCN (LS) refers to this setting. LCN outperformsother state-of-the-art baselines on all three datasets,Citeseer, Cora and Pubmed. Node2vec (Grover andLeskovec, 2016) is an unsupervised method for learningnode representations for a given graph using just thestructure of the graph. In table 3, Node2vec refers tothe model when the kernel is obtained from normalizedNode2vec embeddings, which achieves a significantly

Heterogeneous Edges

Test

Acc

ura

cy

Figure 6: (c) Behavior of test accuracy with increase inthe heterogeneous edges in the hypergraph.

poor performance.

Caveman graph: Goodness of Embeddings Ex-periment: A connected caveman graph of size (n, k)is formed by modifying a set of isolated k-cliques (orcaves) by removing one edge from each clique and usingit to connect to a neighboring clique along a centralcycle such that all n cliques form a single unbrokenloop (Watts, 1999). Caveman graphs are perfect graphsand are used for modeling simple communities in so-cial networks (Kang and Faloutsos, 2011). We run ourexperiments on various synthetic caveman graphs. Forevery caveman graph, we compute the optimal color-ing of the complement graph. We consider a binaryclassification setting and randomly assign nodes corre-sponding to half of the colors to class 0 and the otherhalf to the class 1. We set initial features to be identityand we work with a 20%− 20%− 60% train-validation-test splits, for every graph we run the experiments on

Lovász Convolutional Networks

Dataset Un-Lap N-Lap KS SPORE Kipf-GCN GPNN LCNbreast-cancer 88.2 93.3 92.8 96.7 97.6 95.5 97.2diabetes 68.9 69.3 69.4 73.3 71.4 68.0 76.3fourclass 70.0 70.0 70.4 78.0 80.5 73.9 81.7heart 72.0 75.6 76.4 82.0 85.1 81.1 82.5ionosphere 67.8 68.0 68.1 76.1 76.1 70.0 87.9sonar 58.8 59.0 59.3 63.9 71.4 64.8 73.2mnist-500 1 vs 2 75.6 80.6 79.7 85.8 98.0 96.2 99.0mnist-500 3 vs 8 76.9 81.9 83.3 86.1 92.3 83.1 93.7mnist-500 4 vs 9 68.4 72.0 72.2 74.9 89.4 88.5 83.3mnist-2000 1 vs 2 83.8 96.2 95.0 96.7 99.0 97.5 99.2mnist-2000 3 vs 8 55.2 87.4 87.4 91.4 94.7 89.6 95.7mnist-2000 1 vs 7 90.7 96.8 96.6 97.3 98.8 96.4 98.7

Table 1: Binary Classification with Random label-to-color assignment in UCI and MNIST datasets.

Dataset Kipf-GCN GPNN LCNmnist 500 127 96.1 93.8 97.5mnist 2000 127 97.2 94.4 97.4mnist all 2000 84.4 56.9 85.1

Table 2: Multi class Classification in MNIST dataset

Dataset Node2vec Kipf-GCN GPNN LCNCiteseer 23.1 70.3 69.7 73.5Cora 31.9 81.5 81.8 82.6Pubmed 42.3 79 79.3 79.7

Table 3: Performance for semi-supervised on Citeseer,Cora, Pubmed datasets

10 random splits. Table 4 shows the test accuracy ofKipf-GCN and LCN. In Table 4, Avg_same standsfor average inner product of the representations of thenodes with same color and Avg_diff stands for thatof the nodes with different colors. As we see, LCNperforms better on all cases considered. Also, the aver-age dot products of nodes with same color is high andthose with different colors is close to zero showing thatthe representations are as well separated as possiblefor nodes with different colors.

Hypergraphs: Homogeneous vs HeterogeneousEdges Experiment: Though our main focus is onsimple graphs, we also experiment with synthetic hy-pergraphs. A hypergraph is a generalized version of

(n, k) Kipf-GCN LCN Avg_same Avg_diff(50, 10) 0.92 0.93 0.83 -0.008(75, 6) 0.77 0.80 0.80 -0.005(100, 5) 0.71 0.73 0.79 -0.003(100, 7) 0.81 0.81 0.80 -0.003

Table 4: Caveman graph experiment: Average testaccuracy of Kipf-GCN and LCN on caveman graphs.

a graph where a hyper edge consists of a set of nodes.However, for hypergraphs, to the best of our knowledge,orthogonal embeddings and Lovász theta function arenot defined. Therefore, we consider the clique expan-sion of the hypergraphs (Zhou et al., 2006). Cliqueexpansion creates a simple graph from a hypergraphby replacing every hyperedge with a clique. In ourexperiments, we generated a hypergraph of 100 nodeswith every hyperedge containing 35 nodes. We considera binary classification setting and assign randomly 50nodes to one class and the other 50 to a different class.We randomly create 20 hyperedges such that all thenodes in the hyperedge belong to same class, we callthese edges homogeneous edges. We also create mrandom hyperedges such that the label distributionof the nodes in the hyperedge is 2:3, we call theseedges heterogeneous. We vary m between 10 and 30and create multiple hypergraphs. We set the initialfeatures to identity and work with a 20%-20%-60%train-validation-test split and average across ten runsper each hypergraph. Figure 6 shows the behavior oftest accuracy with increase in the number of heteroge-neous edges. As one can see LCN performs much betterthan GCN when the number of heterogeneous edges aresmall (and hence the clique expansion has a community-like structure) whereas GCNs tend to perform betterwith increase in the number of heterogeneous edges.

7 ConclusionWe propose Lovász Convolution Networks for the prob-lem of semi supervised learning on graphs. Our analysisshows settings where LCNs perform much better thanGCNs. Our results on real world and synthetic datasetsdemonstrate the superior embeddings learnt by LCNsand show that they significantly outperform GCNs.Future work includes detailed analysis of Lovász em-beddings for hypergraphs, use of LCNs for communitydetection and clustering.

Yadav, Nimishakavi, Yadati, Vashihth, Rajkumar, Talukdar

References

Agarwal, S. (2006). Ranking on graph data. In Inter-national Conference on Machine Learning.

Ando, R. K. and Zhang, T. (2007). Learning on graphwith laplacian regularization. In Advances in neuralinformation processing systems.

Atwood, J. and Towsley, D. (2016). Diffusion-convolutional neural networks. In Advances in neuralinformation processing systems.

Auer, S., Bizer, C., Kobilarov, G., Lehmann, J., Cyga-niak, R., and Ives, Z. (2007). Dbpedia: A nucleus fora web of open data. In Proceedings of the 6th Interna-tional The Semantic Web and 2Nd Asian Conferenceon Asian Semantic Web Conference.

Belkin, M., Niyogi, P., and Sindhwani, V. (2006). Mani-fold regularization: A geometric framework for learn-ing from labeled and unlabeled examples. JMLR.

Bollacker, K., Evans, C., Paritosh, P., Sturge, T., andTaylor, J. (2008). Freebase: A collaboratively createdgraph database for structuring human knowledge. InProceedings of the 2008 ACM SIGMOD InternationalConference on Management of Data.

Bruna, J., Zaremba, W., Szlam, A., and LeCun, Y.(2014). Spectral networks and locally connectednetworks on graphs. In ICLR.

Chen, J., Ma, T., and Xiao, C. (2018). Fastgcn: Fastlearning with graph convolutional networks via im-portance sampling. arXiv preprint arXiv:1801.10247.

Chudnovsky, M., Robertson, N., Seymour, P., andThomas, R. (2006). The strong perfect graph theo-rem. Annals of mathematics.

Condon, A. and Karp, R. M. (1999). Algorithms forgraph partitioning on the planted partition model. InRandomization, Approximation, and CombinatorialOptimization. Algorithms and Techniques.

Defferrard, M., Bresson, X., and Vandergheynst, P.(2016). Convolutional neural networks on graphswith fast localized spectral filtering. In Advances inneural information processing systems.

Duvenaud, D. K., Maclaurin, D., Iparraguirre, J., Bom-barell, R., Hirzel, T., Aspuru-Guzik, A., and Adams,R. P. (2015). Convolutional networks on graphsfor learning molecular fingerprints. In Advances inneural information processing systems.

Getoor, L. and Taskar, B. (2007). Introduction tostatistical relational learning.

Giles, C. L., Bollacker, K. D., and Lawrence, S. (1998).Citeseer: An automatic citation indexing system. InProceedings of the Third ACM Conference on DigitalLibraries.

Grover, A. and Leskovec, J. (2016). node2vec: Scalablefeature learning for networks. In Proceedings ofthe 22nd ACM SIGKDD international conference onKnowledge discovery and data mining, pages 855–864.ACM.

Henaff, M., Bruna, J., and LeCun, Y. (2015). Deepconvolutional networks on graph-structured data.CoRR.

Holland, P. W., Laskey, K. B., and Leinhardt, S. (1983).Stochastic blockmodels: First steps. Social networks.

Jain, A., Zamir, A. R., Savarese, S., and Saxena, A.(2016). Structural-rnn: Deep learning on spatio-temporal graphs. In Proceedings of the IEEE Confer-ence on Computer Vision and Pattern Recognition.

Jethava, V., Martinsson, A., Bhattacharyya, C., andDubhashi, D. (2013). Lovasz theta function, svmsand finding dense subgraphs. Journal of MachineLearning Research.

Johansson, F., Jethava, V., Dubhashi, D., and Bhat-tacharyya, C. (2014). Global graph kernels usinggeometric embeddings. In Proceedings of the 31st In-ternational Conference on Machine Learning, ICML2014, Beijing, China, 21-26 June 2014.

Kang, U. and Faloutsos, C. (2011). Beyond’cavemancommunities’: Hubs and spokes for graph compres-sion and mining. In Data Mining (ICDM), 2011IEEE 11th International Conference on.

Kipf, T. N. and Welling, M. (2017). Semi-supervisedclassification with graph convolutional networks. InICLR.

Leskovec, J., Huttenlocher, D., and Kleinberg, J.(2010a). Predicting positive and negative links inonline social networks. In Proceedings of the 19thInternational Conference on World Wide Web.

Leskovec, J., Huttenlocher, D., and Kleinberg, J.(2010b). Signed networks in social media. In Proceed-ings of the SIGCHI Conference on Human Factorsin Computing Systems.

Li, Q., Han, Z., and Wu, X.-M. (2018). Deeper in-sights into graph convolutional networks for semi-supervised learning. In AAAI.

Liao, R., Brockschmidt, M., Tarlow, D., Gaunt, A. L.,Urtasun, R., and Zemel, R. (2018). Graph partitionneural networks for semi-supervised classification.arXiv preprint arXiv:1803.06272.

Lovász, L. (1979). On the shannon capacity of a graph.IEEE Transactions on Information theory.

Lovász, L. (2009). A characterization of perfect graphs.In Classic Papers in Combinatorics, pages 447–450.Springer.

Lovász Convolutional Networks

Lovász, L. and Vesztergombi, K. (1999). Geometricrepresentations of graphs. In IN PAUL ERDÖS,PROC. CONF.

Lu, Q. and Getoor, L. (2003). Link-based classifica-tion. In Proceedings of the Twentieth InternationalConference on International Conference on MachineLearning.

Luz, C. J. (1995). An upper bound on the independencenumber of a graph computable in polynomial-time.Operations Research Letters, 18(3):139–145.

Luz, C. J. and Schrijver, A. (2005). A convex quadraticcharacterization of the lovász theta number. SIAMJournal on Discrete Mathematics, 19(2):382–387.

Marcheggiani, D. and Titov, I. (2017). Encoding sen-tences with graph convolutional networks for seman-tic role labeling. In EMNLP.

McAuley, J. and Leskovec, J. (2012). Learning to dis-cover social circles in ego networks. In Proceedingsof the 25th International Conference on Neural In-formation Processing Systems - Volume 1.

Ramírez-Alfonsín, J. L. and Reed, B. A. (2001). Perfectgraphs, volume 44. Wiley.

Ray, S. N., Dasgupta, S. S., and Talukdar, P. (2018).Ad3: Attentive deep document dater. In Proceed-ings of the 2018 Conference on Empirical Methodsin Natural Language Processing, pages 1871–1880.Association for Computational Linguistics.

Sen, P., Namata, G. M., Bilgic, M., Getoor, L., Gal-lagher, B., and Eliassi-Rad, T. (2008). Collectiveclassification in network data. AI Magazine.

Shivanna, R., Chatterjee, B. K., Sankaran, R., Bhat-tacharyya, C., and Bach, F. (2015). Spectral normregularization of orthonormal representations forgraph transduction. In Advances in Neural Infor-mation Processing Systems 28. Curran Associates,Inc.

Subramanya, A. and Talukdar, P. P. (2014). Graph-based semi-supervised learning. Synthesis Lectureson Artificial Intelligence and Machine Learning,8(4):1–125.

Suchanek, F. M., Kasneci, G., and Weikum, G. (2007).Yago: A core of semantic knowledge. In Proceedingsof the 16th International Conference on World WideWeb.

van der Maaten, L. and Hinton, G. (2008). VisualizingData using t-SNE. Journal of Machine LearningResearch, 9:2579–2605.

Vashishth, S., Dasgupta, S. S., Ray, S. N., and Taluk-dar, P. (2018a). Dating documents using graph con-volution networks. In Proceedings of the 56th Annual

Meeting of the Association for Computational Lin-guistics (Volume 1: Long Papers), pages 1605–1615.Association for Computational Linguistics.

Vashishth, S., Joshi, R., Prayaga, S. S., Bhattacharyya,C., and Talukdar, P. (2018b). RESIDE: Improv-ing distantly-supervised neural relation extractionusing side information. In Proceedings of the 2018Conference on Empirical Methods in Natural Lan-guage Processing, pages 1257–1266. Association forComputational Linguistics.

Vashishth, S., Yadav, P., Bhandari, M., Rai, P., Bhat-tacharyya, C., and Talukdar, P. (2018c). Graph con-volutional networks based word embeddings. CoRR,abs/1809.04283.

Vashishth, S., Yadav, P., Bhandari, M., and Talukdar,P. (2019). Confidence-based graph convolutional net-works for semi-supervised learning. In InternationalConference on Artificial Intelligence and Statistics(AISTATS).

Watts, D. J. (1999). Networks, dynamics, and the small-world phenomenon. American Journal of sociology.

Yang, Z., Cohen, W. W., and Salakhutdinov, R. (2016).Revisiting semi-supervised learning with graph em-beddings. In International Conference on MachineLearning.

Zhou, D., Bousquet, O., Lal, T. N., Weston, J., andSchölkopf, B. (2004). Learning with local and globalconsistency. In Advances in neural information pro-cessing systems.

Zhou, D., Huang, J., and Schölkopf, B. (2006). Learn-ing with hypergraphs: Clustering, classification, andembedding. In Advances in neural information pro-cessing systems.

Zhu, X., Ghahramani, Z., and Lafferty, J. (2003). Semi-supervised learning using gaussian fields and har-monic functions. In International Conference onMachine Learning.

Zhuang, C. and Ma, Q. (2018). Dual graph convo-lutional networks for graph-based semi-supervisedclassification. In TheWebConf.

Zitnik, M. and Leskovec, J. (2017). Predicting multi-cellular function through multi-layer tissue networks.Bioinformatics.