Content-Based Citation Recommendation Chandra Bhagavatula Allen Institute for AI [email protected]Sergey Feldman Data Cowboys * [email protected]Russell Power Independent Researcher † [email protected]Waleed Ammar Allen Institute for AI [email protected]Abstract We present a content-based method for rec- ommending citations in an academic paper draft. We embed a given query document into a vector space, then use its nearest neighbors as candidates, and rerank the candidates us- ing a discriminative model trained to distin- guish between observed and unobserved cita- tions. Unlike previous work, our method does not require metadata such as author names which can be missing, e.g., during the peer review process. Without using metadata, our method outperforms the best reported results on PubMed and DBLP datasets with relative improvements of over 18% in F1@20 and over 22% in MRR. We show empirically that, although adding metadata improves the per- formance on standard metrics, it favors self- citations which are less useful in a citation rec- ommendation setup. We release an online por- tal for citation recommendation based on our method, 1 and a new dataset OpenCorpus of 7 million research articles to facilitate future research on this task. 1 Introduction Due to the rapid growth of the scientific litera- ture, conducting a comprehensive literature review has become challenging, despite major advances in digital libraries and information retrieval sys- tems. Citation recommendation can help improve the quality and efficiency of this process by sug- gesting published scientific documents as likely citations for a query document, e.g., a paper draft to be submitted for ACL 2018. Existing citation recommendation systems rely on various informa- tion of the query documents such as author names and publication venue (Ren et al., 2014; Yu et al., * Work done while on contract with AI2 † Work done while at AI2 1 http://labs.semanticscholar.org/ citeomatic/ 2012), or a partial list of citations provided by the author (McNee et al., 2002; Liu et al., 2015; Jia and Saule, 2017) which may not be available, e.g., during the peer review process or in the early stage of a research project. Our method uses a neural model to embed all available documents into a vector space by encod- ing the textual content of each document. We then select the nearest neighbors of a query document as candidates and rerank the candidates using a second model trained to discriminate between ob- served and unobserved citations. Unlike previous work, we can embed new documents in the same vector space used to identify candidate citations based on their text content, obviating the need to re-train the models to include new published pa- pers. Further, unlike prior work (Yang et al., 2015; Ren et al., 2014), our model is computationally ef- ficient and scalable during both training and test time. We assess the feasibility of recommending ci- tations when some metadata for the query docu- ment is missing, and find that we are able to out- perform the best reported results on two datasets while only using papers’ textual content (i.e., title and abstract). While adding metadata helps further improve the performance of our method on stan- dard metrics, we found that it introduces a bias for self-citation which might not be desirable in a ci- tation recommendation system. See §5 for details of our experimental results. Our main contributions are: • a content-based method for citation recom- mendation which remains robust when meta- data are missing for query documents, • large improvements over state of the art results on two citation recommendation datasets despite omitting the metadata, • a new dataset of seven million research pa- pers, addressing some of the limitations in arXiv:1802.08301v1 [cs.CL] 22 Feb 2018
We present a content-based method for rec-ommending citations in an academic paperdraft. We embed a given query document intoa vector space, then use its nearest neighborsas candidates, and rerank the candidates us-ing a discriminative model trained to distin-guish between observed and unobserved cita-tions. Unlike previous work, our method doesnot require metadata such as author nameswhich can be missing, e.g., during the peerreview process. Without using metadata, ourmethod outperforms the best reported resultson PubMed and DBLP datasets with relativeimprovements of over 18% in F1@20 andover 22% in MRR. We show empirically that,although adding metadata improves the per-formance on standard metrics, it favors self-citations which are less useful in a citation rec-ommendation setup. We release an online por-tal for citation recommendation based on ourmethod,1 and a new dataset OpenCorpus of7 million research articles to facilitate futureresearch on this task.
1 Introduction
Due to the rapid growth of the scientific litera-ture, conducting a comprehensive literature reviewhas become challenging, despite major advancesin digital libraries and information retrieval sys-tems. Citation recommendation can help improvethe quality and efficiency of this process by sug-gesting published scientific documents as likelycitations for a query document, e.g., a paper draftto be submitted for ACL 2018. Existing citationrecommendation systems rely on various informa-tion of the query documents such as author namesand publication venue (Ren et al., 2014; Yu et al.,
∗Work done while on contract with AI2†Work done while at AI2
1http://labs.semanticscholar.org/citeomatic/
2012), or a partial list of citations provided by theauthor (McNee et al., 2002; Liu et al., 2015; Jiaand Saule, 2017) which may not be available, e.g.,during the peer review process or in the early stageof a research project.
Our method uses a neural model to embed allavailable documents into a vector space by encod-ing the textual content of each document. We thenselect the nearest neighbors of a query documentas candidates and rerank the candidates using asecond model trained to discriminate between ob-served and unobserved citations. Unlike previouswork, we can embed new documents in the samevector space used to identify candidate citationsbased on their text content, obviating the need tore-train the models to include new published pa-pers. Further, unlike prior work (Yang et al., 2015;Ren et al., 2014), our model is computationally ef-ficient and scalable during both training and testtime.
We assess the feasibility of recommending ci-tations when some metadata for the query docu-ment is missing, and find that we are able to out-perform the best reported results on two datasetswhile only using papers’ textual content (i.e., titleand abstract). While adding metadata helps furtherimprove the performance of our method on stan-dard metrics, we found that it introduces a bias forself-citation which might not be desirable in a ci-tation recommendation system. See §5 for detailsof our experimental results.
Our main contributions are:• a content-based method for citation recom-
mendation which remains robust when meta-data are missing for query documents,• large improvements over state of the art
results on two citation recommendationdatasets despite omitting the metadata,• a new dataset of seven million research pa-
pers, addressing some of the limitations in
arX
iv:1
802.
0830
1v1
[cs
.CL
] 2
2 Fe
b 20
18
dq
d1 d2 d3
d4 d5 d6 d7
dq
d1
d2 d3d6 d7
d4d5 dq
d1 citesd4d1 citesd5d3 citesd7…
K=4
Phase1:candidateselection
Phase2:reranking
NNRankd2
0.7reranked
list
citedinnearestneighbors:
dqNNRank
d60.8
dqNNRank
d30.5
dqNNRank
d40.3
dqNNRank
d70.9
d7
d6
d2
d3
d4
topN=3
recommen
datio
ns
querydocument
documentembeddings
nearestneighborsofdq:
Figure 1: An overview of our Citation Recommendation system. In Phase 1 (NNSelect), we project all docu-ments in the corpus (7 in this toy example) in addition to the query document dq into a vector space, and use its(K=4) nearest neighbors: d2, d6, d3, and d4 as candidates. We also add d7 as a candidate because it was cited ind3. In Phase 2 (NNRank), we score each pair (dq, d2), (dq, d6), (dq, d3), (dq, d4), and (dq, d7) separately to rerankthe candidates and return the top 3 candidates: d7, d6 and d2.
previous datasets used for citation recom-mendation, and• a scalable web-based literature review tool
based on this work.2
2 Overview
We formulate citation recommendation as a rank-ing problem. Given a query document dq and alarge corpus of published documents, the task isto rank documents which should be referenced indq higher than other documents. Following pre-vious work on citation recommendation, we usestandard metrics (precision, recall, F-measure andmean reciprocal rank) to evaluate our predictionsagainst gold references provided by the authors ofquery documents.
Since the number of published documents in thecorpus can be large, it is computationally expen-sive to score each document as a candidate refer-ence with respect to dq. Instead, we recommendcitations in two phases: (i) a fast, recall-orientedcandidate selection phase, and (ii) a feature rich,
2https://github.com/allenai/citeomatic
precision-oriented reranking phase. Figure 1 pro-vides an overview of the two phases using a toyexample.
Phase 1 - Candidate Selection: In this phase,our goal is to identify a set of candidate referencesfor dq for further analysis without explicitly iterat-ing over all documents in the corpus.3 Using atrained neural network, we first project all pub-lished documents into a vector space such that adocument tends to be close to its references. Sincethe projection of a document is independent of thequery document, the entire corpus needs to be em-bedded only once and can be reused for subse-quent queries. Then, we project each query doc-ument dq to the same vector space and identify itsnearest neighbors as candidate references. See §3for more details about candidate selection.
3In order to increase the chances that all references arepresent in the list of candidates, the number of candidatesmust be significantly larger than the total number of citationsof a document, but also significantly smaller than the numberof documents in the corpus.
Phase 2 - Reranking: Phase 1 yields a manage-able number of candidates making it feasible toscore each candidate di by feeding the pair (dq, di)into another neural network trained to discriminatebetween observed and unobserved citation pairs.The candidate documents are sorted by their es-timated probability of being cited in dq, and topcandidates are returned as recommended citations.See §4 for more details about the reranking modeland inference in the candidate selection phase.
3 Phase 1: Candidate Selection(NNSelect)
In this phase, we select a pool of candidate cita-tions for a given query document to be reranked inthe next phase. First, we compute a dense embed-ding of the query document dq using the documentembedding model (described next), and select Knearest neighbor documents in the vector space ascandidates.4 Following Strohman et al. (2007), wealso include the outgoing citations of the K near-est neighbors as candidates.
The output of this phase is a list of candi-date documents di and their corresponding scoresNNSelect(dq, di), defined as the cosine similar-ity between dq and di in the document embeddingspace.
Document embedding model. We use asupervised neural model to project any doc-ument d to a dense embedding based on itstextual content. We use a bag-of-word repre-sentation of each textual field, e.g., d[title] ={‘content-based’, ‘citation’, ‘recommendation’},and compute the feature vector:
fd[title] =∑
t∈d[title]
wmagt
wdirt
‖wdirt ‖2
, (1)
where wdirt is a dense direction embedding and
wmagt is a scalar magnitude for word type t.5 We
then normalize the representation of each field andcompute a weighted average of fields to get thedocument embedding, ed. In our experiments, weuse the title and abstract fields of a document d:
ed = λtitle fd[title]
‖fd[title]‖2+ λabstract fd[abstract]
‖fd[abstract]‖2,
4We tune K as a hyperparameter of our method.5The magnitude-direction representation is based on Sal-
imans and Kingma (2016) and was found to improve re-sults in preliminary experiments, compared to the standard“direction-only” word representation.
where λtitle and λabstract are scalar model parame-ters.
Training. We learn the parameters of the doc-ument embedding model (i.e., λ∗, wmag
∗ ,wdir∗ ) us-
ing a training set T of triplets 〈dq, d+, d−〉 wheredq is a query document, d+ is a document citedin dq, and d− is a document not cited in dq. Themodel is trained to predict a high cosine similarityfor the pair (dq, d
+) and a low cosine similarity forthe pair (dq, d
−) using the per-instance triplet loss(Wang et al., 2014):
loss = max(α+ s(dq, d
−)− s(dq, d+), 0), (2)
where s(di, dj) is defined as the cosine similaritybetween document embeddings cos-sim(edi , edj ).We tune the margin α as a hyperparameter of themodel. Next, we describe how negative examplesare selected.
Selecting negative examples. Defining positiveexamples is straight-forward; we use any (dq, d
+)pair where a document dq in the training set citesd+. However, a careful choice of negative trainingexamples is critical for model performance. Weuse three types of negative examples:
1. Random: any document not cited by dq.2. Negative nearest neighbors: documents
that are close to dq in the embedding space,but are not cited in it.6
3. Citation-of-citation: documents referencedin positive citations of dq, but are not citeddirectly in dq.
In Appendix §A, we describe the number of nega-tive examples of each type used for training. Next,we describe how to rerank the candidate docu-ments.
4 Phase 2: Reranking Candidates(NNRank)
In this phase, we train another model which takesas input a pair of documents (dq, di) and estimatesthe probability that di should be cited in dq.
Input features. A key point of this work is toassess the feasibility of recommending citationswithout using metadata, but we describe all fea-tures here for completeness and defer this discus-sion to §5. For each document, we compute dense
6Since the set of approximate neighbors depend on modelparameters, we recompute a map from each query documentto its K nearest neighbors before each epoch while trainingthe document embedding model.
elu
sigmoid
elu
fd[keyphrases]
query fd[keyphrases]
candidate
fd[abstract] fd[authors] fd[venue]
concat
abstract intersection
di[in-citations]
NNSelect(dq,di)
output is a vector
output is a scalar
output type
fd[title]
fd[title]
cos-sim
title intersection
fd[abstract] fd[authors] fd[venue]
cos-simcos-simcos-sim cos-sim
Figure 2: NNRank architecture. For each of the textual and categorical fields, we compute the cosine similaritybetween the embedding for dq and the corresponding embedding for di. Then, we concatenate the cosine similarityscores, the numeric features and the summed weights of the intersection words, followed by two dense layers withELU non-linearities The output layer is a dense layer with sigmoid non-linearity, which estimates the probabilitythat dq cites di.
feature vectors fd[field] as defined in Eq. 1 for thefollowing fields: title, abstract, authors, venue andkeyphrases (if available). For the title and abstract,we identify the subset of word types which appearin both documents (intersection), and compute thesum of their scalar weights as an additional fea-ture, e.g.,
∑t∈∩title w
∩t . We also use log number
of times the candidate document di has been citedin the corpus, i.e., log(di[in-citations]). Finally, weuse the cosine similarity between dq and di in theembedding space, i.e., cos-sim(edq , edi).
Model architecture. We illustrate theNNRank model architecture in Figure 2.The output layer is defined as:
s(di, dj) = FeedForward(h), (3)
h =[gtitle;gabstract;gauthors;gvenue;
gkeyphrases; cos-sim(edq , edi);∑t∈∩title
w∩t ;∑
t∈∩abstractw∩t ;
di[in-citations]],
gfield = cos-sim(fdq [field], fdi[field]),
where ‘FeedForward’ is a three layer feed-forwardneural network with two exponential linear unitlayers (Clevert et al., 2015) and one sigmoid layer.
Training. The parameters of the NNRankmodelare w
mag∗ ,wdir
∗ , w∩∗ and parameters of the three
dense layers in ‘FeedForward’. We reuse the
triplet loss in Eq. 2 to learn these parameters, butredefine the similarity function s(di, dj) as the sig-moid output described in Eq. 3.
At test time, we use this model to recommendcandidates di with the highest s(dq, di) scores.
5 Experiments
In this section, we describe experimental results ofour citation recommendation method and compareit to previous work.
Datasets. We use the DBLP and PubMeddatasets (Ren et al., 2014) to compare with previ-ous work on citation recommendation. The DBLPdataset contains over 50K scientific articles in thecomputer science domain, with an average of 5citations per article. The PubMed dataset con-tains over 45K scientific articles in the medicaldomains, with an average of 17 citations per ar-ticle. In both datasets, a document is accompa-nied by a title, an abstract, a venue, authors, cita-tions and keyphrases. We replicate the experimen-tal setup of Ren et al. (2014) by excluding paperswith fewer than 10 citations and using the standardtrain, dev and test splits.
We also introduce OpenCorpus,7 a newdataset of 7 million scientific articles primar-ily drawn from the computer science and neuro-science domain. Due to licensing constraints, doc-
7http://labs.semanticscholar.org/corpus/
uments in the corpus do not include the full textof the scientific articles, but include the title, ab-stract, year, author, venue, keyphrases and citationinformation. The mutually exclusive training, de-velopment, and test splits were selected such thatno document in the development or test set has apublication year less than that of any document inthe training set. Papers with zero citations wereremoved from the development and test sets. Wedescribe the key characteristics of OpenCorpusin Table 1.
Statistic Value# of documents in corpus 6.9 million# of unique authors 8.3 million# of unique keyphrases 823,677# of unique venues 23,672avg. # of incoming citations 7.4 (± 38.1)avg. # of outgoing citations 8.4 (± 14.4)size of training set [years 1991 to 2014] 5.5 millionsize of dev set [years 2014 to 2015] 689,000size of test set [years 2015 to 2016] 20,000
Table 1: Characteristics of the OpenCorpus.
Baselines. We compare our method to two base-line methods for recommending citations: Clus-Cite and BM25. ClusCite (Ren et al., 2014) clus-ters nodes in a heterogeneous graph of terms, au-thors and venues in order to find related documentswhich should be cited. We use the ClusCite resultsas reported in Ren et al. (2014), which compared itto several other citation recommendation methodsand found that it obtains state of the art results onthe PubMed and DBLP datasets. The BM25 re-sults are based on our implementation of the pop-ular ranking function Okapi BM25 used in manyinformation retrieval systems. See Appendix §Dfor details of our BM25 implementation.
Evaluation. We use Mean Reciprocal Rank(MRR) and F1@20 to report the main results inthis section. In Appendix §F, we also report ad-ditional metrics (e.g., precision and recall at 20)which have been used in previous work. We com-pute F1@20 as the harmonic mean of the corpus-level precision and recall at 20 (P@20 and R@20).Following (Ren et al., 2014), precision and recallat 20 are first computed for each query documentthen averaged over query documents in the test setto compute the corpus-level P@20 and R@20.
Configurations. To find candidates inNNSelect, we use the approximate nearest
neighbor search algorithm Annoy8, which buildsa binary-tree structure that enables searching fornearest neighbors in O(log n) time. To build thistree, points in a high-dimensional space are splitby choosing random hyperplanes. We use 100trees in our approximate nearest neighbors index,and retrieve documents using the cosine distancemetric.
We use the hyperopt library9 to optimize var-ious hyperparameters of our method such as sizeof hidden layers, regularization strength and learn-ing rate. To ensure reproducibility, we provide adetailed description of the parameters used in bothNNSelect and NNRank models, our hyperpa-rameter optimization method and parameter val-ues chosen in Appendix §A.
Main results. Table 2 reports the F1@20 andMRR results for the two baselines and three vari-ants of our method. Since the OpenCorpusdataset is much bigger, we were not able to trainthe ClusCite baseline for it. Totti et al. (2016) havealso found it difficult to scale up ClusCite to largerdatasets. Where available, we report the mean ±standard deviation based on five trials.
The first variant, labeled “NNSelect,” onlyuses the candidate selection part of our method(i.e., phase 1) to rank candidates by their cosinesimilarity to the query document in the embeddingspace as illustrated in Fig. 1. Although the docu-ment embedding space was designed to efficientlyselect candidates for further processing in phase 2,recommending citations directly based on the co-sine distance in this space outperforms both base-lines.
The second variant, labeled “NNSelect +NNRank,” uses the discriminative model (i.e.,phase 2) to rerank candidates selected byNNSelect, without encoding metadata (venues,authors, keyphrases). Both the first and secondvariants show that improved modeling of papertext can significantly outperform previous meth-ods for citation recommendation, without usingmetadata.
The third variant, labeled “NNSelect +NNRank + metadata,” further encodes the meta-data features in the reranking model, and givesthe best overall results. On both the DBLP andPubMed datasets, we obtain relative improve-ments over 20% (for F1@20) and 25% (for MRR)
Table 2: F1@20 and MRR results for two baselines and three variants of our method. BM25 results are basedon our implementation of this baseline, while ClusCite results are based on the results reported in Ren et al.(2014). “NNSelect” ranks candidates using cosine similarity between the query and candidate documents inthe embedding space (phase 1). “NNSelect + NNRank” uses the discriminative reranking model to rerankcandidates (phase 2), without encoding any of the metadata features. “+ metadata” encodes the metadata features(i.e., keyphrases, venues and authors), achieving the best results on all datasets. Mean and standard deviations arereported based on five trials.
compared to the best reported results of ClusCite.In the rest of this section, we describe controlled
experiments aimed at analyzing different aspectsof our proposed method.
Choice of negative samples. As discussed in §3,we use different types of negative samples to trainour models. We experimented with using only asubset of the types, while controlling for the totalnumber of negative samples used, and found thatusing negative nearest neighbors while training themodels is particularly important for the method towork. As illustrated in Table 3, on the PubMeddataset, adding negative nearest neighbors whiletraining the models improves the F1@20 scorefrom 0.306 to 0.329, and improves the MRR scorefrom 0.705 to 0.771. Intuitively, using nearestneighbor negative examples focuses training onthe harder cases on which the model is more likelyto make mistakes.
F1@20 ∆ MRR ∆
Full model 0.329 0.771without intersection 0.296 0.033 0.653 0.118without -ve NNs 0.306 0.016 0.705 0.066without numerical 0.314 0.008 0.735 0.036
Table 3: Comparison of PubMed results of the fullmodel with model without (i) intersection features, (ii)negative nearest neighbors in training samples, and (iii)numerical features.
Valuable features. We experimented with dif-ferent subsets of the optional features used inNNRank in order to evaluate the contributionof various features. We found intersection fea-tures, NNSelect scores, and the number of
incoming citations to be the most valuable fea-ture. As illustrated in Table 3, the intersectionfeatures improves the F1@20 score from 0.296 to0.329, and the MRR score from 0.653 to 0.771,on the PubMed dataset. The numerical features(NNSelect score and incoming citations) im-prove the F1@20 score from 0.314 to 0.329, andimproves the MRR score from 0.735 to 0.771.This shows that, in some applications, feeding en-gineered features to neural networks can be an ef-fective strategy to improve their performance.
Encoding textual features. We also experi-mented with using recurrent and convolutionalneural network to encode the textual fields ofquery and candidate documents, instead of us-ing a weighted sum as described in Eq. 1. Wefound that recurrent and convolutional encodersare much slower, and did not observe a significantimprovement in the overall performance as mea-sured by the F1@20 and MRR metrics. This resultis consistent with previous studies on other tasks,e.g., Iyyer et al. (2015).
Number of nearest neighbors. As discussedin §3, the candidate selection step is crucial forthe scalability of our method because it reducesthe number of computationally expensive pair-wise comparisons with the query document at run-time. We did a controlled experiment on theOpenCorpus dataset (largest among the threedatasets) to measure the effect of using differentnumbers of nearest neighbors, and found that bothP@20 and R@20 metrics are maximized whenNNSelect fetches five nearest neighbors usingthe approximate nearest neighbors index (and theirout-going citations), as illustrated in Table 4.
Table 4: OpenCorpus results for NNSelect stepwith varying number of nearest neighbors on 1,000 val-idation documents.
Self-citation bias. We hypothesized that amodel trained with the metadata (e.g., authors)could be biased towards self-citations and otherwell-cited authors. To verify this hypothesis, wecompared two NNRank models – one with meta-data, and one without. We measured the mean andmax rank of predictions that had at least one authorin common with the query document. This ex-periment was performed with the OpenCorpusdataset.
A lower mean rank for NNRank + Metadata in-dicates that the model trained with metadata tendsto favor documents authored by one of the querydocument’s authors. We verified the prevalence ofthis bias by varying the number of predictions foreach model from 1 to 100. Figure 3 shows thatthe mean and max rank of the model trained withmetadata is always lower than those for the modelthat does not use metadata.
Figure 3: Mean and Max Rank of predictions withvarying number of candidates.
6 Related Work
Citation recommendation systems can be dividedinto two categories – local and global. A localcitation recommendation system takes a few sen-tences (and an optional placeholder for the can-didate citation) as input and recommends cita-tions based on the local context of the input sen-tences (Huang et al., 2015; He et al., 2010; Tangand Zhang, 2009; Huang et al., 2012; He et al.,2011). A global citation recommendation systemtakes the entire scholarly article as input and rec-ommends citations for the paper (McNee et al.,2002; Strohman et al., 2007; Nallapati et al., 2008;Kataria et al., 2010; Ren et al., 2014). We ad-dress the global citation recommendation problemin this paper.
A key difference of our proposed method com-pared to previous work is that our method iscontent-based and works well even in the absenceof metadata (e.g. authors, venues, key phrases,seed list of citations). Many citation recommenda-tion systems crucially rely on a query document’smetadata. For example, the collaborative filteringbased algorithms of McNee et al. (2002); Jia andSaule (2017); Liu et al. (2015) require seed cita-tions for a query document. (Ren et al., 2014; Yuet al., 2012) require authors, venues and key termsof the query documents to infer interest groups andto extract features based on paths in a heteroge-neous graph. In contrast, our model performs wellsolely based on the textual content of the querydocument.
Some previous work (e.g. (Ren et al., 2014; Yuet al., 2012)) have addressed the citation recom-mendation problem using graph-based methods.But, training graph-based citation recommenda-tion models has been found to be expensive. Forexample, the training complexity of the ClusCitealgorithm (Ren et al., 2014) is cubic in the numberof edges in the graph of authors, venues and terms.This can be prohibitively expensive for datasetsas large as OpenCorpus. On the other handour model is a neural network trained via batchedstochastic gradient descent that scales very well tolarge datasets (Bottou, 2010).
Another crucial difference between our ap-proach and some prior work in citation predic-tion is that we build up a document representa-tion using its constituent words only. Prior algo-rithms (Huang et al., 2015, 2012; Nallapati et al.,2008; Tanner and Charniak, 2015) learn an explicit
representation for each training document sepa-rately that isn’t a deterministic function of the doc-ument’s words. This makes the model effectivelytransductive since a never-before-seen documentdoes not have a ready-made representation. Simi-larly, Huang et al. (2012)’s method needs a candi-date document to have at least one in-coming cita-tion to be eligible for citation – this disadvantagesnewly published documents. Liu et al. (2015)form document representations using citation re-lations, which are not available for unfinished ornew documents. In contrast, our method does notneed to be re-trained as the corpus of potential can-didates grows. As long as the new documents arein the same domain as that of the model’s trainingdocuments, they can simply be added to the cor-pus and are immediately available as candidatesfor future queries.
While the citation recommendation task has at-tracted a lot of research interest, a recent surveypaper (Beel et al., 2016) has found three main con-cerns with existing work: (i) limitations in evalu-ation due to strongly pruned datasets, (ii) lack ofdetails for re-implementation, and (iii) variationsin performance across datasets. For example, theaverage number of citations per document in theDBLP dataset is 5, but Ren et al. (2014) filteredout documents with fewer than 10 citations fromthe test set. This drastically reduced the size of thetest set. We address these concerns by releasinga new large scale dataset for future citation rec-ommendation systems. In our experiments on theOpenCorpus dataset, we only prune documentswith zero outgoing citations. We provide extensivedetails of our system (see Appendix §A) to facil-itate reproducibility and release our code10. Wealso show in experiments that our method consis-tently outperforms previous systems on multipledatasets.
Finally, recent work has combined graph noderepresentations and text-based document repre-sentations using CCA (Gupta and Varma, 2017).This sort of approach can enhance our text-baseddocument representations if a technique to creategraph node representations at test-time is avail-able.
7 Conclusion
In this paper, we present a content-based cita-tion recommendation method which remains ro-
10https://github.com/allenai/citeomatic
bust when metadata is missing for query docu-ments, enabling researchers to do an effective liter-ature search early in their research cycle or duringthe peer review process, among other scenarios.We show that our method obtains state of the artresults on two citation recommendation datasets,even without the use of metadata available to thebaseline method. We make our system publiclyaccessible online. We also introduce a new datasetof seven million scientific articles to facilitate fu-ture research on this problem.
Acknowledgements
We would like to thank Oren Etzioni, Luke Zettle-moyer, Doug Downey and Iz Beltagy for partic-ipating in discussions and for providing helpfulcomments on the paper draft; Hsu Han and restof the Semantic Scholar team at AI2 for creatingthe OpenCorpus dataset. We also thank Xiang Renfor providing the data used in their experiments onthe DBLP and Pubmed datasets. Finally, we thankthe anonymous reviewers for insightful commentson the draft.
ReferencesJoeran Beel, Bela Gipp, Stefan Langer, and Corinna
Breitinger. 2016. Research-paper recommender sys-tems: a literature survey. International Journal onDigital Libraries 17(4):305–338.
Leon Bottou. 2010. Large-scale machine learningwith stochastic gradient descent. In Proceedings ofCOMPSTAT’2010, Springer, pages 177–186.
Jane Bromley, James W Bentz, Leon Bottou, Is-abelle Guyon, Yann LeCun, Cliff Moore, EduardSackinger, and Roopak Shah. 1993. Signature veri-fication using a siamese time delay neural network.International Journal of Pattern Recognition andArtificial Intelligence 7(04):669–688.
Cornelia Caragea, Florin Adrian Bulgarov, AndreeaGodea, and Sujatha Das Gollapalli. 2014a. Citation-enhanced keyphrase extraction from research pa-pers: A supervised approach. In Proceedings ofthe 2014 Conference on Empirical Methods in Natu-ral Language Processing (EMNLP). Association forComputational Linguistics, pages 1435–1446.
Cornelia Caragea, Florin Adrian Bulgarov, AndreeaGodea, and Sujatha Das Gollapalli. 2014b. Citation-enhanced keyphrase extraction from research pa-pers: A supervised approach. In EMNLP. vol-ume 14, pages 1435–1446.
Djork-Arne Clevert, Thomas Unterthiner, and SeppHochreiter. 2015. Fast and accurate deep network
learning by exponential linear units (elus). arXivpreprint arXiv:1511.07289 .
Shashank Gupta and Vasudeva Varma. 2017. Scientificarticle recommendation by using distributed repre-sentations of text and graph. In WWW.
Qi He, Daniel Kifer, Jian Pei, Prasenjit Mitra, andC. Lee Giles. 2011. Citation recommendation with-out author supervision. In WSDM.
Qi He, Jian Pei, Daniel Kifer, Prasenjit Mitra, and LeeGiles. 2010. Context-aware citation recommenda-tion. In Proceedings of the 19th international con-ference on World wide web. ACM, pages 421–430.
Wenyi Huang, Saurabh Kataria, Cornelia Caragea,Prasenjit Mitra, C Lee Giles, and Lior Rokach. 2012.Recommending citations: translating papers intoreferences. In Proceedings of the 21st ACM inter-national conference on Information and knowledgemanagement. ACM, pages 1910–1914.
Wenyi Huang, Zhaohui Wu, Liang Chen, Prasenjit Mi-tra, and C Lee Giles. 2015. A neural probabilisticmodel for context based citation recommendation.In AAAI. pages 2404–2410.
Mohit Iyyer, Varun Manjunatha, Jordan L. Boyd-Graber, and Hal Daume. 2015. Deep unorderedcomposition rivals syntactic methods for text clas-sification. In ACL.
Haofeng Jia and Erik Saule. 2017. An analysis of cita-tion recommender systems: Beyond the obvious. InASONAM.
Saurabh Kataria, Prasenjit Mitra, and Sumit Bhatia.2010. Utilizing context in generative bayesian mod-els for linked corpus. In AAAI.
Patrice Lopez and Laurent Romary. 2010. Humb: Au-tomatic key term extraction from scientific articlesin grobid. In Proceedings of the 5th internationalworkshop on semantic evaluation. Association forComputational Linguistics, pages 248–251.
Christopher D. Manning, Mihai Surdeanu, John Bauer,Jenny Rose Finkel, Steven Bethard, and David Mc-Closky. 2014. The stanford corenlp natural languageprocessing toolkit. In ACL.
Sean M McNee, Istvan Albert, Dan Cosley, Pra-teep Gopalkrishnan, Shyong K Lam, Al MamunurRashid, Joseph A Konstan, and John Riedl. 2002.On the recommending of citations for research pa-pers. In Proceedings of the 2002 ACM conferenceon Computer supported cooperative work. ACM,pages 116–125.
Ramesh Nallapati, Amr Ahmed, Eric P. Xing, andWilliam W. Cohen. 2008. Joint latent topic modelsfor text and citations. In KDD.
Xiang Ren, Jialu Liu, Xiao Yu, Urvashi Khandelwal,Quanquan Gu, Lidan Wang, and Jiawei Han. 2014.Cluscite: Effective citation recommendation by in-formation network-based clustering. In Proceedingsof the 20th ACM SIGKDD international conferenceon Knowledge discovery and data mining. ACM,pages 821–830.
Tim Salimans and Diederik P. Kingma. 2016. Weightnormalization: A simple reparameterization to ac-celerate training of deep neural networks. CoRRabs/1602.07868. http://arxiv.org/abs/1602.07868.
Trevor Strohman, W. Bruce Croft, and David D.Jensen. 2007. Recommending citations for aca-demic papers. In SIGIR.
Jie Tang and Jing Zhang. 2009. A discriminative ap-proach to topic-based citation recommendation. InPAKDD.
Chris Tanner and Eugene Charniak. 2015. A hybridgenerative/discriminative approach to citation pre-diction. In HLT-NAACL.
Luam C. Totti, Prasenjit Mitra, Mourad Ouzzani, andMohammed J. Zaki. 2016. A query-oriented ap-proach for relevance in citation networks. In WWW.
Jiang Wang, Yang Song, Thomas Leung, Chuck Rosen-berg, Jingbin Wang, James Philbin, Bo Chen, andYing Wu. 2014. Learning fine-grained image simi-larity with deep ranking. In Proceedings of the 2014IEEE Conference on Computer Vision and PatternRecognition. IEEE Computer Society, Washington,DC, USA, CVPR ’14, pages 1386–1393. https://doi.org/10.1109/CVPR.2014.180.
Cheng Yang, Zhiyuan Liu, Deli Zhao, Maosong Sun,and Edward Y Chang. 2015. Network representa-tion learning with rich text information. In Pro-ceedings of the 24th International Joint Conferenceon Artificial Intelligence, Buenos Aires, Argentina.pages 2111–2117.
Xiao Yu, Quanquan Gu, Mianwei Zhou, and JiaweiHan. 2012. Citation prediction in heterogeneousbibliographic networks. In SDM.
A Hyperparameter Settings
Neural networks are complex and have a largenumber of hyperparameters to tune. This makesit challenging to reproduce experimental results.Here, we provide details of how the hyperparame-ters of the NNSelect and NNRank models werechosen or otherwise set. We chose a subset of hy-perparameters for tuning, and left the rest at man-ually set default values. Due to limited computa-tional resources, we were only able to perform hy-perparameter tuning on the development split ofthe smaller DBLP and Pubmed datasets.
For DBLP and PubMed, we first ran Hyperopt11
with 75 trials. Each trial was run for five epochsof 500,000 triplets each. The ten top performingof these models were trained for a full 50 epochs,and the best performing model’s hyperparametersare selected. Hyperparameters for NNSelectwere optimized for Recall@20 and those for theNNRank model were optimized for F1@20 on thedevelopment set. The selected values for DBLPare reported in Table 6 and for PubMed are re-ported in Table 7.OpenCorpus hyperparameters were set via in-
formal hand-tuning, and the results are in Table 9.A few miscellaneous parameters (not tuned) thatare necessary for reproducibility are in Table 8.
We briefly clarify the meaning of some param-eters below:
• Margin Multiplier - The triplet loss has vari-able margins for the three types of negatives:0.1γ, 0.2γ, and 0.3γ. We treat γ as a hyper-parameter and refer to it as the margin multi-plier.
• Use Siamese Embeddings - For the majorityof our experiments, we use a Siamese model(Bromley et al., 1993). That is, the textualembeddings for the query text and abstractshare the same weights. However, we hada significantly larger amount of data to trainNNRank on OpenCorpus, and found thatnon-Siamese embeddings are beneficial.
• Use Pretrained - We estimate word em-beddings on the titles and abstracts ofOpenCorpus using Word2Vec imple-mented by the gensim Python package12.
When computing the margins for the triplet loss,we use a boosting function for highly cited docu-ments. The full triplet loss function is as follows:
max(γα(d−)
+ s(dq, d−) +B(d−)
− s(dq, d+)−B(d+)
, 0)
where γ is the margin multiplier, and α(d−) variesbased on the type of negative document:
• α(d−) = 0.3 for random negatives
• α(d−) = 0.2 for nearest neighbor negatives
• α(d−) = 0.1 for citation-of-citation negatives.
The boosting function is defined as follows:
B(d) =σ(d[in-citations]
100
)50
where σ is the sigmoid function and d[in-citations]is the number of times document dwas cited in thecorpus. The boosting function allows the model toslightly prefer candidates that are cited more fre-quently, and the constants were set without opti-mization.
C Nearest Neighbors for Training Details
When obtaining nearest neighbors for negative ex-amples during training, we use a heuristic to finda subset of the fetched nearest neighbors that aresufficiently wrong. That is, these are non-citationsamples that look dissimilar in the original text butsimilar in the embedding space. This procedure isas follows for each training query:
1. Compute the Jaccard similarities between atraining query and all of its true citationsusing the concatenation of title and abstracttexts.
2. Compute the bottom fifth percentile Jaccardsimilarity value. I.e. the value below whichonly the bottom 5% most least textually sim-ilar true citations fall. For example, if theJaccard similarities range from 0.2 to 0.9, thefifth percentile might plausibly be 0.3.
3. Use the Annoy index computed at the end ofthe previous epoch to fetch nearest neighborsfor the query document.
4. Compute the textual Jaccard similarity be-tween all of the nearest neighbors and thequery document.
5. Retain nearest neighbors that have a smallerJaccard similarity than the fifth percentile.Using the previous example, retain the near-est neighbors that have a lower Jaccard simi-larity than 0.3.
Table 5: Results of our BM25 implementation onDBLP and Pubmed datasets.
Okapi-BM25 is a popular ranking function.We use BM25 as an IR-based baseline for thetask of citation recommendation. For the DBLPand Pubmed datasets, BM25 performance is pro-vided in Ren et al. (2014). To create a com-petitive BM25 baseline for OpenCorpus, wefirst created indexes for the DBLP and Pubmeddatasets and tuned the query to approximate theperformance reported in previous work. We usedWhoosh13 to create an index. We extract the keyterms (using Whoosh’s key terms from textmethod) from the title and abstract of each querydocument. The key terms from the document areconcatenated to form the query string. Table 5shows that our BM25 is a close approximation tothe BM25 implementation of previous work andcan be reliably used as a strong IR baseline forOpenCorpus. In Table 2, we report results on allthree datasets using our BM25 implementation.
E Key Phrases for OpenCorpus
In the OpenCorpus dataset, some documentsare accompanied by automatically extracted keyphrases. Our implementation of automatic keyphrase extraction is based on standard key phraseextraction systems – e.g. (Caragea et al., 2014a,b;Lopez and Romary, 2010). We first extract nounphrases using the Stanford CoreNLP package(Manning et al., 2014) as candidate key phrases.Next, we extract corpus level and document level
13https://pypi.python.org/pypi/Whoosh/
features (e.g. term frequency, document fre-quency, n-gram probability etc.) for each candi-date key phrase. Finally, we rank the candidatekey phrases using a ranking model that is trainedon author-provided key phrases as gold labels.
F Detailed Experimental Results
Table 10 compares NNRank with previous workin detail on DBLP and Pubmed datasets. Clus-Cite (Ren et al., 2014) clusters nodes in a het-erogeneous graph of terms, authors and venues inorder to find related documents which should becited. ClusCite obtains the previous best results onthese two datasets. L2-LR (Yu et al., 2012) uses alinear combination of meta-path based linear fea-tures to classify candidate citations. We show thatNNRank (with and without metadata) consistentlyoutperforms ClusCite and other baselines on allmetrics on both datasets.
Table 6: DBLP hyperparameter tuning results. Note that the dense dimension when using pretrained vectors isfixed to be 300. A ’-’ indicates that the variable was not tuned.
Table 7: PubMed hyperparameter tuning results. Note that the dense dimension when using pretrained GloVevectors is fixed to be 300. A ’-’ indicates that the variable was not tuned.
Hyperparameter PubMed/DBLP Value OpenCorpus Valuetitle/abstract vocabulary size 200000 200000maximum title length 50 50maximum abstract length 500 500training triplets per query 6 6min # of papers per author included 1 10min # of papers per venue included 1 10min # of papers per keyphrases included 5 10max authors per document 8 8max keyphrases per document 20 20minimum true citations per document 2/1 2maximum true citations per document 100 100optimizer LazyAdamOptimizer* Nadam**use magnitude-direction embeddings true truereduce learning rate upon plateau false true
Table 8: Per-dataset parameters. These were hand-specified. *LazyAdamOptimizer is part of TensorFlow.**Nadam is part of Keras.
Hyperparameter NNSelect Value NNRank Valuelearning rate 0.001 0.001l2 regularization 1e-5 1e-5l1 regularization 1e-7 1e-7word dropout 0.1 0.1margin multiplier 1.0 1.0dense dimension 75 75metadata dimension - 25use pretrained false falsenumber ANN neighbors 5 -triplets per batch size 256 32triplets per epoch 2500000 2500000triplets per training 25000000 100000000use Siamese embeddings true false
Table 10: Comparing NNRank with ClusCite. (Ren et al., 2014) have presented results on several other topic-based, link-based and network-based citation recommendation methods as baselines. For succinctness, we showresults for the best system, Cluscite, and two baselines BM25 and L2-LR.
![GhostLink: Latent Network Inference for Influence-aware ... · social-network based recommendation. Similarly, in the field of citation networks, [2] attempt to extract citation influence](https://static.documents.pub/doc/80x56/5f9691063dfdda6c3165dac7/ghostlink-latent-network-inference-for-influence-aware-social-network-based.jpg)
![Santosh Divvala arXiv:1506.02640v3 [cs.CV] 11 Jun 2015 · Santosh Divvala Allen Institute for Artificial Intelligence santoshd@allenai.org Ross Girshick Microsoft Research rbg@microsoft.com](https://static.documents.pub/doc/80x56/5f01b36c7e708231d4009fda/santosh-divvala-arxiv150602640v3-cscv-11-jun-2015-santosh-divvala-allen-institute.jpg)
