Relational fact extraction aims to extract se-mantic triplets from unstructured text. In thiswork, we show that all of the relational factextraction models can be organized accord-ing to a graph-oriented analytical perspective.An efficient model, aDjacency lIst oRientedrElational faCT (DIRECT), is proposed basedon this analytical framework. To alleviatechallenges of error propagation and sub-taskloss equilibrium, DIRECT employs a noveladaptive multi-task learning strategy with dy-namic sub-task loss balancing. Extensive ex-periments are conducted on two benchmarkdatasets, and results prove that the proposedmodel outperforms a series of state-of-the-art(SoTA) models for relational triplet extraction.
1 Introduction
Relational fact extraction, as an essential NLPtask, is playing an increasingly important role inknowledge graph construction (Han et al., 2019;Distiawan et al., 2019). It aims to extract rela-tional triplet from the text. A relational tripletis in the form of (subject, relation, object) or(s, r, o) (Zeng et al., 2019). While various priormodels proposed for relational fact extraction, fewof them analyze this task from the perspective ofoutput data structure.
As shown in Figure 1, the relational fact extrac-tion can be characterized as a directed graph con-struction task, where graph representation flexibil-ity and heterogeneity accompany additional bene-faction. In practice, there are three common waysto represent graphs (Gross and Yellen, 2005):
Edge List is utilized to predict a sequence oftriplets (edges). The recent sequence-to-sequencebased models, such as NovelTagging (Zheng et al.,2017), CopyRE (Zeng et al., 2018), CopyRL (Zeng
∗These two authors contributed equally to this research.†Zhuoren Jiang is the corresponding author
Figure 1: Example of exploring the relational fact ex-traction task from the perspective of directed graph rep-resentation method as output data structure.
et al., 2019), and PNDec (Nayak and Ng, 2020),fall into this category.
Edge list is a simple and space-efficient way torepresent a graph (Arifuzzaman and Khan, 2015).However, there are three problems. First, thetriplet overlapping problem (Zeng et al., 2018).For instance, as shown in Figure 1, for triplets(Obama, nationality, USA) and (Obama, presi-dent of, USA), there are two types of relations be-tween the “Obama” and “USA”. If the model onlygenerates one sequence from the text (Zheng et al.,2017), it may fail to identify the multi-relation be-tween entities. Second, to overcome the triplet over-lapping problem, the model may have to extractthe triplet element repeatedly (Zeng et al., 2018),which will increase the extraction cost. Third, therecould be an ordering problem (Zeng et al., 2019):for multiple triplets, the extraction order could in-fluence the model performance.
Adjacency Matrices are used to predict ma-trices that represent exactly which entities (ver-tices) have semantic relations (edges) betweenthem. Most early works, which take a pipeline ap-
3076
proach (Zelenko et al., 2003; Zhou et al., 2005), be-long to this category. These models first recognizeall entities in text and then perform relation classi-fication for each entity pair. The subsequent neuralnetwork-based models (Bekoulis et al., 2018; Daiet al., 2019), that attempt to extract entities andrelations jointly, can also be classified into this cat-egory.
Compared to edge list, adjacency matrices havebetter relation (edge) searching efficiency (Arifuz-zaman and Khan, 2015). Furthermore, adjacencymatrices oriented models is able to cover differ-ent overlapping cases (Zeng et al., 2018) for rela-tional fact extraction task. But the space cost ofthis approach can be expensive. For most cases, theoutput matrices are very sparse. For instance, fora sentence with n tokens, if there are m kinds ofrelations, the output space is n · n ·m, which canbe costly for graph representation efficiency. Thisphenomenon is also illustrated in Figure 1.
Adjacency List is designed to predict an arrayof linked lists that serves as a representation of agraph. As depicted in Figure 1, in the adjacencylist, each vertex v (key) points to a list (value) con-taining all other vertices connected to v by sev-eral edges. Adjacency list is a hybrid graph rep-resentation between edge list and adjacency ma-trices (Gross and Yellen, 2005), which can bal-ance space and searching efficiency1. Due to thestructural characteristic of the adjacency list, thistype of model usually adopts a cascade fashion toidentify subject, object, and relation sequentially.For instance, the recent state-of-the-art model Cas-Rel (Wei et al., 2020) can be considered as an ex-emplar. It utilizes a two-step framework to rec-ognize the possible object(s) of a given subjectunder a specific relation. However, CasRel is notfully adjacency list oriented: in the first step, ituse subject as the key; while in the second step, itpredicts (relation, object) pairs using adjacencymatrix representation.
Despite its considerable potential, the cascadefashion of adjacency list oriented model may causeproblems of sub-task error propagation (Shen et al.,2019), i.e., errors from ancestor sub-tasks may ac-cumulate to threaten downstream ones, and sub-tasks can hardly share supervision signals. Multi-task learning (Caruana, 1997) can alleviate thisproblem, however, the sub-task loss balancing prob-
1More detailed complexity analyses of different graph rep-resentations are provided in Appendix section 6.3.
lem (Chen et al., 2018; Sener and Koltun, 2018)could compromise its performance.
Based on the analysis from the perspective ofoutput data structure, we propose a novel solution,aDjacency lIst oRiented rElational faCT extractionmodel (DIRECT), with the following advantages:• For efficiency, DIRECT is a fully adjacency list
oriented model, which consists of a shared BERTencoder, the Pointer-Network based subject and ob-ject extractors, and a relation classification module.In Section 3.4, we provide a detailed comparativeanalysis2 to demonstrate the efficiency of the pro-posed method.• From the performance viewpoint, to address
sub-task error propagation and sub-task loss balanc-ing problems, DIRECT employs a novel adaptivemulti-task learning strategy with the dynamic sub-task loss balancing approach. In Section 3.2 and3.3, the empirical experimental results demonstrateDIRECT can achieve the state-of-the-art perfor-mance of relational fact extraction task, and theadaptive multi-task learning strategy did play a pos-itive role in improving the task performance.
The major contributions of this paper can besummarized as follows:
1. We refurbish the relational fact extractionproblem by leveraging an analytical frameworkof graph-oriented output structure. To the best ofour knowledge, this is a pioneer investigation toexplore the output data structure of relational factextractions.
2. We propose a novel solution, DIRECT3,which is a fully adjacency list oriented model witha novel adaptive multi-task learning strategy.
3. Through extensive experiments on two bench-mark datasets3, we demonstrate the efficiency andefficacy of DIRECT. The proposed DIRECT out-performs the state-of-the-art baseline models.
2 The DIRECT Framework
In this section, we will introduce the frameworkof the proposed DIRECT model, which includesa shared BERT encoder and three output layers:subject extraction, object extraction, and relationclassification. As shown in Figure 2, DIRECT isfully adjacency list oriented. The input sentenceis firstly fed into the subject extraction module to
2Theoretical representation efficiency analysis of graphrepresentative models are described in Appendix section 6.4.
3To help other scholars reproduce the experiment out-come, we will release the code and datasets via GitHub:https://github.com/fyubang/direct-ie.
3077
Figure 2: An overview of the proposed DIRECT framework
extract all subjects. Then each extracted subjectis concatenated with the sentence, and fed intothe object extraction module to extract all objects,which can form a set of subject-object pairs. Fi-nally, the subject-object pair is concatenated withsentence, and fed into the relation classificationmodule to get the relations between them. Forbalancing the weights of sub-task losses and to im-prove the global task performance, three modulesshare the BERT encoder layer and are trained withan adaptive multi-task learning strategy.
2.1 Shared BERT Encoder
In the DIRECT framework, the encoder is used toextract the semantic features from the inputs forthree modules. As aforementioned, we employ theBERT (Devlin et al., 2019) as the shared encoder tomake use of its pre-trained knowledge and attentionmechanism.
The architecture of the shared method is shownin Figure 2. The lower embedding layer and trans-formers (Vaswani et al., 2017) are shared across allthe three modules, while the top layers representthe task-specific outputs.
The encoding process is as follows:
ht = BERT(xt) (1)
where xt = [w1, ..., wn] is the input text of task tand ht is the hidden vector sequence of the input.Due to the limited space, the detailed architectureof BERT please refer to the original paper (Devlinet al., 2019).
2.2 Subject and Object ExtractionThe subject and object extraction modules are moti-vated by the Pointer-Network (Vinyals et al., 2015)architecture, which are widely used in MachineReading Comprehension (MRC) (Rajpurkar et al.,2016) task. Different from MRC task that onlyneeds to extract a single span, the subject and objectextractions need to extract multiple spans. There-fore, in the training phase, we replace softmaxfunction with sigmoid function for the activationfunction of the output layer, and replace cross en-tropy (CE) (Goodfellow et al., 2016) with binarycross entropy (BCE) (Luc et al., 2016) for the lossfunction. Specifically, we will perform indepen-dent binary classifications for each token twice toindicate whether the current token is the start or theend of a span. The probability of a token to be startor end is as follows:
pti,start = σ(Wtstart · hi + bt
start) (2)
pti,end = σ(Wtend · hi + bt
end) (3)
where hi represents the hidden vector of the ithtoken, t ∈ [s, o] represents subject and object ex-traction respectively, Wt ∈ Rh×1 represents thetrainable weight, bt ∈ R1 is the bias and σ issigmoid function.
During inference, we first recognize all the startpositions by checking if the probability pti,start > α,where α is the threshold of extraction. Then, weidentify the corresponding end position with thelargest probability pti,end between two neighboringstart positions. Concretely, assuming posj,start isthe start position of the jth span, the corresponding
3078
end position is:
posj,end = argmaxposj,start<=i<posj+1,start
pti,end (4)
Though the overall structure is similar, the in-puts for subject and object extraction are different.When extracting the subject, only the original sen-tence needs to be input:
x = [w1, ..., wn] (5)
inputs = [[cls],x, [sep]] (6)
where wi represents the ith token of the originalsentence.
Meanwhile, the object extraction is based on thecorresponding subject. To form the input, the sub-ject s and the original sentence x are concatenatedwith [sep] as follows:
inputo = [[cls], s, [sep],x, [sep]] (7)
2.3 Relation classification
The output layer of relation classification is rela-tively simple, which is a normal multi-label classi-fication model. The [cls] vector obtained by BERTencoder is used as the sentence embedding. A fullyconnected layer is used for the nonlinear transfor-mation, and perform multi-label classification topredict relations of the input subject-object pair.The detailed operations of relation classificationare as follows:
Pr = σ(Wr · h[cls] + br) (8)
where Pr ∈ Rc is the predicted probability vec-tor of relations, σ is sigmoid function, Wr ∈Rh×c and br ∈ Rc are the trainable weights andbias, h is the hidden size of encoder, c is the num-ber of relations, and h[cls] denotes the hidden vectorof the first token [cls]. The input for relation classi-fication task is as follows:
inputr = [[cls], s, [sep], o, [sep],x, [sep]] (9)
2.4 Adaptive Multi-task Learning
In DIRECT, subject extraction module, object ex-traction module, and relation classification modulecan be considered as three sub-tasks. As afore-mentioned, if we train each module directly andseparately, the error propagation problem would
Algorithm 1: Adaptive Multi-task Learn-ing with Dynamic Loss Balancing
Initialize model parameters Θ randomly;Load pre-trained BERT parameters forshared encoder;
Prepare the data for each task t and packthem into mini-batch: Dt, t ∈ [s, o, r] ;
Get the number of batch for each task: nt;Set the number of epoch for training:epochmax;
for epoch in 1, 2, ..., epochmax do1. Merge all the datasets:D = Ds ∪Do ∪Dr;
2. Shuffle D;3. Initialize EMA for each task vt = 1and its decay ε = 0.99 ;
for bt in D do// bt is a mini-batch of Dt ;4. Compute loss: lt(Θ) ;5. Update EMA:vt = (1− ε) ·
∑(lt) + ε · vt ;
6. Calculate and normalize theweights: wt = (vt/nt)/(vr/nr) ;
7. Update model Θ with gradient:∇(wt · l̄t) ;
endend
reduce the task performance. Meanwhile, three in-dependent encoders would consume more memory.Therefore, we use multi-task learning to alleviatethis problem, and the encoder layer is shared acrossthree modules.
However, applying multi-task learning could bechallenging in DIRECT, due to the following prob-lems:• The input and output of the three modules are
different, which means we cannot simply sum upthe loss of each task.• How should we balance the weights of losses
for three sub-task modules?These issues can affect the final results of multi-
task training (Shen et al., 2019; Sener and Koltun,2018).
In this work, based on the architecture of MT-DNN (Liu et al., 2019b), we propose a novel adap-tive multi-task learning strategy to address theabove problems. The algorithm is shown as Algo-rithm 1. Basically, the datasets are firstly split intomini-batches. A batch is then randomly sampled
3079
to calculate the loss. The parameters of the sharedencoder and its task-specific layer are updated ac-cordingly. Especially, the learning effect of eachtask t is different and dynamically changing duringtraining. Therefore, an approach of adaptively ad-justing the weights of task losses is applied. Thesum of sub-task’s loss
∑lt is utilized to approxi-
mate its optimization effect. The adaptive weightadjusting strategy ensures that the more room asub-task has to be optimized, the more weight itsloss will receive. Furthermore, an exponential mov-ing average (EMA) (Lawrance and Lewis, 1977) ismaintained to avoid the drastic fluctuations of lossweights. Last but not least, to make sure that eachtask has enough influence on the shared encoder,the weight of the sub-task will be penalized accord-ing to the training data amount of each sub-task.
3 Experiments
3.1 Dataset and Experiment Setting
Datasets. Two public datasets are used for evalu-ation: NYT (Riedel et al., 2010) is originally pro-duced by the distant supervision approach. Thereare 1.18M sentences with 24 predefined relationtypes in NYT. WebNLG (Gardent et al., 2017) isoriginally created for Natural Language Generation(NLG) tasks. (Zeng et al., 2018) adopts this datasetfor relational triplet extraction task. It contains246 predefined relation types. There are differentversions of these two datasets. To facilitate com-parison evaluation, we use the datasets releasedby (Zeng et al., 2018) and follow their data splitrules.
Besides the basic relational triplet extraction, re-cent studies are focusing on the relational tripletoverlapping problem (Zeng et al., 2018; Wei et al.,2020). Follow the overlapping pattern definitionof relational triplets (Zeng et al., 2018), the sen-tences in both datasets are divided into three cate-gories, namely, Normal, EntityPairOverlap (EPO),and SingleEntityOverlap (SEO). The statistics ofthe two datasets are described in Table 1.
Baselines: the following strong state-of-the-art(SoTA) models have been compared in the experi-ments.• NovelTagging (Zheng et al., 2017) introduces
a tagging scheme that transforms the joint entityand relation extraction task into a sequence labelingproblem. It can be considered as edge list oriented.• CopyRE (Zeng et al., 2018) is a seq2seq
Table 1: Statistics of Dataset NYT and WebNLG. Notethat a sentence can belong to both EPO class and SEOclass.
can effectively extract overlapping triplets. It hastwo variants: CopyREone employs one decoder;CopyREmul employs multiple decoders. CopyREis also edge list oriented.• GraphRel (Fu et al., 2019) is a GCN (graph
convolutional networks) (Kipf and Welling, 2017)based model, where a relation-weighted GCN is uti-lized to learn the interaction between entities andrelations. It is a two phases model: GraphRel1pdenotes 1st-phase extraction model; GraphRel2pdenotes full extraction model. GraphRel is adja-cency matrices oriented.• CopyRL (Zeng et al., 2019) combines the re-
inforcement learning with a seq2seq model to au-tomatically learn the extraction order of triplets.CopyRL is edge list oriented.• CasRel (Wei et al., 2020) is a cascade binary
tagging framework, where all possible subjects areidentified in the first stage, and then for each iden-tified subject, all possible relations and the cor-responding objects are simultaneously identifiedby a relation specific tagger. This work recentlyachieves the SoTA results. As aforementioned, Cas-Rel is partially adjacency list oriented.
Evaluation Metrics: following the previouswork (Zeng et al., 2018; Wei et al., 2020), differ-ent models are compared by using standard microPrecision (Prec.), Recall (Rec.), and F1-score4. Anextracted relational triplet (subject, relation, object)is regarded as correct only if the relation and theheads of both subject and object are all correct.
Implementation Details. The hyper-parameters are determined on the validationset. To avoid the evaluation bias, all reportedresults from our method are averaged results for 5runs. More implementation details are described in
4In this study, the results of baseline models are all self-reported results from their original papers. Meanwhile, theexperimental results of our proposed model are the average offive runs.
3080
Appendix section 6.1.
3.2 Results and AnalysisRelational Triplet Extraction Performance. Thetask performances on two datasets are summarizedin Table 2. Based on the experiment results, wehave the following observations and discussions:• The proposed DIRECT model outperformed
all baseline models in terms of all evaluation met-rics on both datasets, which proved DIRECT modelcan effectively address the relational triplet extrac-tion task.• The best-performed model (DIRECT) and
runner-up model (CasRel) were both adjacency listoriented model. These two models overwhelm-ingly outperformed other models, which indicatedthe considerable potential of adjacency list (as theoutput data structure) for improving the task per-formance.• To further compare the relation extraction abil-
ity of DIRECT and CasRel, we took a closer lookat the extraction performance of relational tripletelements from these two models. As shown inTable 35, DIRECT outperformed CasRel in termsof all relational triplet elements on both datasets.These empirical results suggested that, for rela-tional triplet extraction, a fully adjacency list ori-ented model (DIRECT) may have advantages overa partially oriented one (CasRel).
Figure 3: F1 score of extracting relational triples fromsentences with different overlapping patterns on NYTdataset.
Ability in Handling The Overlapping Prob-lem. The relational facts in sentences are oftencomplicated. Different relational triplets may haveoverlaps in a sentence. To verify the ability ofour models in handling the overlapping problem,
5More detailed results with Precision and Recall are pro-vided in Appendix section 6.2.
we conducted further experiments on NYT dataset.Figure 3 illustrated of F1 scores of extracting rela-tional triplets from sentences with different overlap-ping patterns. DIRECT outperformed all baselinemodels in terms of all overlapping patterns. Theseresults demonstrated the effectiveness of the pro-posed model in solving the overlapping problem.
Ability in Handling Multiple Relation Ex-traction. We further compared the model’s abil-ity of extracting relations from sentences that con-tain multiple triplets. The sentences in NYT andWebNLG were divided into 5 categories. Each cat-egory contained sentences that had 1,2,3,4 or ≥ 5triplets. The triplet number was denoted as N . Asshown in Table 4:• DIRECT achieved the best performance for
all triplet categories on both datasets. These ex-perimental results demonstrated our model had anexcellent ability in handling multiple relation ex-traction.• In both NYT and WebNLG datasets, when
the sentences contained more triplets, the leadingadvantage of DIRECT became greater. This obser-vation indicated that DIRECT was good at solvingcomplex relational fact extraction.
3.3 Ablation StudyTo validate the effectiveness of components in DI-RECT, We implemented several model variants forablation tests6. The results of the comparison onNYT dataset are shown in Table 5. In particular,we aim to address the following two research ques-tions:
RQ1: Is it possible to improve the model per-formance by sharing the parameters of extractionlayers?
RQ2: Did the proposed adaptive multi-tasklearning strategy improve the task performance?
Effects of Sharing Extraction Layer Parame-ters (RQ1). As described in Section 2, the struc-tures of subject extraction and object extractionoutput layers are exactly the same. To answer RQ1,we merged the subject extraction and object ex-traction layers into one entity extraction layer bysharing the parameters of output layers of these twomodules, denoted as DIRECTshared. From the re-sults of Table 5, we can observe that, sharing theparameters of output layers of two extraction mod-ules would reduce the performance of the model.
6Due to the length limitation, we list two main ablationexperiments, the rest will be provided in the Appendix section6.2.
3081
Method Category NYT WebNLGPrec. Rec. F1 Prec. Rec. F1
NovelTagging(Zheng et al., 2017) EL 62.4 31.7 42.0 52.5 193. 28.3CopyREOne(Zeng et al., 2018) EL 59.4 53.1 56.0 32.2 28.9 30.5CopyREMul(Zeng et al., 2018) EL 61.0 56.6 58.7 37.7 36.4 37.1GraphRel1p(Fu et al., 2019) AM 62.9 57.3 60.0 42.3 39.2 40.7GraphRel2p(Fu et al., 2019) AM 63.9 60.0 61.9 44.7 41.1 42.9CopyRL(Zeng et al., 2019) EL 77.9 67.2 72.1 63.3 59.9 61.6
CasRel(Wei et al., 2020) ALP 89.7 89.5 89.6 93.4 90.1 91.8
DIRECT(Ours) ALF92.3 92.8 92.5 93.6 92.7 93.2
(±0.32) (±0.26) (±0.09) (±0.1) (±0.24) (±0.07)
Table 2: Results of different methods on NYT and WebNLG datasets. EL: Edge List; AM: Adjacency Matrices;ALP: Adjacency List (Partially); ALF: Adjacency List (Fully).
Method Element NYT WebNLG
CasRels 93.5 95.7o 93.5 95.3r 94.9 94.0
DIRECT(Ours)s 95.4 97.3o 96.4 96.4r 97.8 97.4
Table 3: F1-score for extracting elements of relationaltriplets on NYT and WebNLG datasets.
A possible explanation is that, although the out-put of these two modules is similar, the semanticsof subject and object are different. Hence, directlysharing the output parameters of two modules couldlead to an unsatisfactory performance.
Effects of Adaptive Multi-task Learning(RQ2). As described in Section 2, the adaptivemulti-task learning strategy with the dynamic sub-task loss balancing approach is proposed for im-proving the task performance. To answer RQ2, wereplaced the adaptive multi-task learning strategywith an ordinary learning strategy. In this strategy,the losses of three sub-tasks were computed withequal weights, denoted as DIRECTequal. Fromthe results of Table 5, we can observe that, by usingadaptive multi-task learning, DIRECT was able toget a 1.5 percentage improvement on the F1-score.This significant improvement indicated that adap-tive multi-task learning played a positive role in thebalance of sub-task learning and can improve theglobal task performance.
3.4 Graph Representation EfficiencyAnalysis
Based on the amount estimation of predicted log-its7, we conduct a graph representation efficiency
7Numeric output (0/1) of the last layer
analysis to demonstrate the efficiency of the pro-posed method8.
For each graph representation category, wechoose one representative algorithms. Edge List:CopyRE (Zeng et al., 2018); Adjacency Matrices:MHS (Bekoulis et al., 2018); Adjacency List: Cas-Rel (partially) (Wei et al., 2020) and the proposedDIRECT (fully).
The averaged predicted logits estimation for onesample9 of different models on two datasets areshown in Table 6. MHS is adjacency matrices ori-ented, it has the most logits that need to be pre-dicted. Since CasRel is partially adjacency listoriented, it needs to predict more logits than DI-RECT. Theoretically, as an edge list oriented, thepredicted logits of CopyRE should be the least. But,as described in Section 1, it needs to extract the en-tities repeatedly to handle the overlapping problem.Hence, its graph representation efficiency could beworse than our model. The structure of our modelis simple and fully adjacency list oriented. There-fore, from the viewpoint of predicted logits estima-tion, DIRECT is the most representative-efficientmodel.
4 Related Work
Relation Fact Extraction. In this work, we showthat all of the relational fact extraction models canbe unified into a graph-oriented output structureanalytical framework. From the perspective ofgraph representation, the prior models can be di-vided into three categories. Edge List, this typeof model usually employs sequence-to-sequencefashion, such as NovelTagging (Zheng et al., 2017),
8From the graph representation perspective, when amethod requires fewer logits to represent the graph (set oftriples), it will reduce the model fitting difficulty.
9The theoretical analysis of predicted logits for differentmodels are described in Appendix section 6.4.
3082
Method NYT WebNLGN = 1 N = 2 N = 3 N = 4 N ≥ 5 N = 1 N = 2 N = 3 N = 4 N ≥ 5
Table 5: Results of model variants for ablation tests.
Method Category NYT WebNLGCopyRe EL 329 712
MHS AM 57369 26518CasRel ALP 3084 15836
DIRECT ALF 238 542
Table 6: Graph representation efficiency estimationbased on the predicted logits amount. EL: Edge List;AM: Adjacency Matrices; ALP: Adjacency List (Par-tially); ALF: Adjacency List (Fully).
CopyRE (Zeng et al., 2018), CopyRL (Zeng et al.,2019), and PNDec (Nayak and Ng, 2020). Somemodels of this category may suffer from the tripletoverlapping problem and expensive extraction cost.Adjacency Matrices, many early pipeline ap-proaches (Zelenko et al., 2003; Zhou et al., 2005;Mintz et al., 2009) and recent neural network-basedmodels (Bekoulis et al., 2018; Dai et al., 2019; Fuet al., 2019), can be classified into this category.The main problem for this type of model is thegraph representation efficiency. Adjacency List,the recent state-of-the-art model CasRel (Wei et al.,2020) is a partially adjacency list oriented model.In this work, we propose DIRECT that is a fullyadjacency list oriented relational fact extractionmodel. To the best of our knowledge, few previ-ous works analyze this task from the output datastructure perspective. GraphRel (Fu et al., 2019)employs a graph-based approach, but it is utilizedfrom an encoding perspective, while we analyze itfrom the perspective of output structure. Our work
is a pioneer investigation to analyze the output datastructure of relational fact extraction.
Multi-task Learning. Multi-task Learning(MTL) can improve the model performance. (Caru-ana, 1997) summarizes the goal succinctly: “itimproves generalization by leveraging the domain-specific information contained in the training sig-nals of related task.” It has two benefits (Van-denhende et al.): (1) multiple tasks share a sin-gle model, which can save memory. (2) Associ-ated tasks complement and constrain each other bysharing information, which can reduce overfittingand improve global performance. There are twomain types of MTL: hard parameter sharing (Bax-ter, 1997) and soft parameter sharing (Duong et al.,2015). Most of the multi-task learning is done bysumming the loses directly, this approach is notsuitable for our case. When the input and outputare different, it is impossible to get two losses inone forward propagation. MT-DNN (Liu et al.,2019b) is proposed for this problem. Furthermore,MTL is difficult for training, the magnitudes ofdifferent task-losses are different, and the directsummation of losses may lead to a bias for a partic-ular task. There are already some studies proposedto address this problem (Chen et al., 2018; Guoet al., 2018; Liu et al., 2019a). They all try todynamically adjust the weight of the loss accord-ing to the magnitude of the loss, the difficulty ofthe problem, the speed of learning, etc. In thisstudy, we adopt MT-DNN’s framework, and pro-pose an adaptive multi-task learning strategy thatcan dynamically adjust the loss weight based onthe averaged EMA (Lawrance and Lewis, 1977) ofthe training data amount, task difficulty, etc.
5 Conclusion
In this paper, we introduce a new analytical per-spective to organize the relational fact extractionmodels and propose DIRECT model for this task.
3083
Unlike existing methods, DIRECT is fully adja-cency list oriented, which employs a novel adaptivemulti-task learning strategy with dynamic sub-taskloss balancing. Extensive experiments on two pub-lic datasets, prove the efficiency and efficacy of theproposed methods.
Acknowledgments
We are thankful to the anonymous reviewers fortheir helpful comments. This work is supportedby Alibaba Group through Alibaba Research Fel-lowship Program, the National Natural ScienceFoundation of China (61876003), the Key Re-search and Development Plan of Zhejiang Province(Grant No.2021C03140), the Fundamental Re-search Funds for the Central Universities, andGuangdong Basic and Applied Basic ResearchFoundation (2019A1515010837).
ReferencesShaikh Arifuzzaman and Maleq Khan. 2015. Fast par-
allel conversion of edge list to adjacency list forlarge-scale graphs. In Proceedings of the Sympo-sium on High Performance Computing, pages 17–24.
Jonathan Baxter. 1997. A bayesian/information the-oretic model of learning to learn via multiple tasksampling. Machine learning, 28(1):7–39.
Giannis Bekoulis, Johannes Deleu, Thomas Demeester,and Chris Develder. 2018. Joint entity recogni-tion and relation extraction as a multi-head selectionproblem. Expert Systems with Applications, 114:34–45.
Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, andAndrew Rabinovich. 2018. Gradnorm: Gradientnormalization for adaptive loss balancing in deepmultitask networks. In International Conference onMachine Learning, pages 794–803. PMLR.
Dai Dai, Xinyan Xiao, Yajuan Lyu, Shan Dou, Qiao-qiao She, and Haifeng Wang. 2019. Joint ex-traction of entities and overlapping relations usingposition-attentive sequence labeling. In Proceed-ings of the AAAI Conference on Artificial Intelli-gence, volume 33, pages 6300–6308.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. Bert: Pre-training ofdeep bidirectional transformers for language under-standing. In Proceedings of the 2019 Conference ofthe North American Chapter of the Association for
Computational Linguistics: Human Language Tech-nologies, Volume 1 (Long and Short Papers), pages4171–4186.
Bayu Distiawan, Gerhard Weikum, Jianzhong Qi, andRui Zhang. 2019. Neural relation extraction forknowledge base enrichment. In Proceedings of the57th Annual Meeting of the Association for Compu-tational Linguistics, pages 229–240.
Long Duong, Trevor Cohn, Steven Bird, and Paul Cook.2015. Low resource dependency parsing: Cross-lingual parameter sharing in a neural network parser.In Proceedings of the 53rd annual meeting of the As-sociation for Computational Linguistics and the 7thinternational joint conference on natural languageprocessing (volume 2: short papers), pages 845–850.
Tsu-Jui Fu, Peng-Hsuan Li, and Wei-Yun Ma. 2019.Graphrel: Modeling text as relational graphs forjoint entity and relation extraction. In Proceedingsof the 57th Annual Meeting of the Association forComputational Linguistics, pages 1409–1418.
Claire Gardent, Anastasia Shimorina, Shashi Narayan,and Laura Perez-Beltrachini. 2017. Creating train-ing corpora for nlg micro-planners. In Proceedingsof the 55th Annual Meeting of the Association forComputational Linguistics (Volume 1: Long Papers),pages 179–188.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, andYoshua Bengio. 2016. Deep learning, volume 1.MIT press Cambridge.
Jonathan L Gross and Jay Yellen. 2005. Graph theoryand its applications. CRC press.
Michelle Guo, Albert Haque, De-An Huang, Serena Ye-ung, and Li Fei-Fei. 2018. Dynamic task prioritiza-tion for multitask learning. In European Conferenceon Computer Vision, pages 282–299. Springer.
Xu Han, Tianyu Gao, Yuan Yao, Deming Ye, ZhiyuanLiu, and Maosong Sun. 2019. Opennre: An openand extensible toolkit for neural relation extrac-tion. In Proceedings of the 2019 Conference onEmpirical Methods in Natural Language Processingand the 9th International Joint Conference on Nat-ural Language Processing (EMNLP-IJCNLP): Sys-tem Demonstrations, pages 169–174.
Diederik P Kingma and Jimmy Ba. 2015. Adam: Amethod for stochastic optimization. In InternationalConference on Learning Representations (ICLR).
Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutionalnetworks. In International Conference on LearningRepresentations (ICLR).
AJ Lawrance and PAW Lewis. 1977. An exponentialmoving-average sequence and point process (ema1).Journal of Applied Probability, pages 98–113.
3084
Shikun Liu, Edward Johns, and Andrew J Davison.2019a. End-to-end multi-task learning with atten-tion. In Proceedings of the IEEE/CVF Conferenceon Computer Vision and Pattern Recognition, pages1871–1880.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jian-feng Gao. 2019b. Multi-task deep neural networksfor natural language understanding. In Proceedingsof the 57th Annual Meeting of the Association forComputational Linguistics, pages 4487–4496.
Pauline Luc, Camille Couprie, Soumith Chintala,and Jakob Verbeek. 2016. Semantic segmenta-tion using adversarial networks. arXiv preprintarXiv:1611.08408.
Mike Mintz, Steven Bills, Rion Snow, and Dan Juraf-sky. 2009. Distant supervision for relation extrac-tion without labeled data. In Proceedings of theJoint Conference of the 47th Annual Meeting of theACL and the 4th International Joint Conference onNatural Language Processing of the AFNLP, pages1003–1011.
Tapas Nayak and Hwee Tou Ng. 2020. Effective mod-eling of encoder-decoder architecture for joint entityand relation extraction. In Proceedings of the AAAIConference on Artificial Intelligence, volume 34,pages 8528–8535.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, andPercy Liang. 2016. Squad: 100,000+ questions formachine comprehension of text. In Proceedings ofthe 2016 Conference on Empirical Methods in Natu-ral Language Processing, pages 2383–2392.
Sebastian Riedel, Limin Yao, and Andrew McCallum.2010. Modeling relations and their mentions with-out labeled text. In Joint European Conferenceon Machine Learning and Knowledge Discovery inDatabases, pages 148–163. Springer.
Ozan Sener and Vladlen Koltun. 2018. Multi-tasklearning as multi-objective optimization. In Pro-ceedings of the 32nd International Conference onNeural Information Processing Systems, pages 525–536.
Tao Shen, Xiubo Geng, QIN Tao, Daya Guo, DuyuTang, Nan Duan, Guodong Long, and Daxin Jiang.2019. Multi-task learning for conversational ques-tion answering over a large-scale knowledge base.In Proceedings of the 2019 Conference on EmpiricalMethods in Natural Language Processing and the9th International Joint Conference on Natural Lan-guage Processing (EMNLP-IJCNLP), pages 2442–2451.
Simon Vandenhende, Stamatios Georgoulis, WouterVan Gansbeke, Marc Proesmans, Dengxin Dai, andLuc Van Gool. Multi-task learning for dense predic-tion tasks: A survey.
Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N Gomez, ŁukaszKaiser, and Illia Polosukhin. 2017. Attention is allyou need. In Proceedings of the 31st InternationalConference on Neural Information Processing Sys-tems, pages 6000–6010.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly.2015. Pointer networks. In Proceedings of the28th International Conference on Neural Informa-tion Processing Systems-Volume 2, pages 2692–2700.
Zhepei Wei, Jianlin Su, Yue Wang, Yuan Tian, andYi Chang. 2020. A novel cascade binary taggingframework for relational triple extraction. In Pro-ceedings of the 58th Annual Meeting of the Asso-ciation for Computational Linguistics, pages 1476–1488.
Dmitry Zelenko, Chinatsu Aone, and AnthonyRichardella. 2003. Kernel methods for relation ex-traction. Journal of machine learning research,3(Feb):1083–1106.
Daojian Zeng, Haoran Zhang, and Qianying Liu. 2020.Copymtl: Copy mechanism for joint extraction ofentities and relations with multi-task learning. InProceedings of the AAAI Conference on Artificial In-telligence, volume 34, pages 9507–9514.
Xiangrong Zeng, Shizhu He, Daojian Zeng, Kang Liu,Shengping Liu, and Jun Zhao. 2019. Learning theextraction order of multiple relational facts in a sen-tence with reinforcement learning. In Proceedingsof the 2019 Conference on Empirical Methods inNatural Language Processing and the 9th Interna-tional Joint Conference on Natural Language Pro-cessing (EMNLP-IJCNLP), pages 367–377.
Xiangrong Zeng, Daojian Zeng, Shizhu He, Kang Liu,and Jun Zhao. 2018. Extracting relational facts byan end-to-end neural model with copy mechanism.In Proceedings of the 56th Annual Meeting of theAssociation for Computational Linguistics (Volume1: Long Papers), pages 506–514.
Ranran Haoran Zhang, Qianying Liu, Aysa Xuemo Fan,Heng Ji, Daojian Zeng, Fei Cheng, Daisuke Kawa-hara, and Sadao Kurohashi. 2020. Minimize expo-sure bias of seq2seq models in joint entity and re-lation extraction. In Proceedings of the 2020 Con-ference on Empirical Methods in Natural LanguageProcessing: Findings, pages 236–246.
Suncong Zheng, Feng Wang, Hongyun Bao, YuexingHao, Peng Zhou, and Bo Xu. 2017. Joint extractionof entities and relations based on a novel taggingscheme. In Proceedings of the 55th Annual Meet-ing of the Association for Computational Linguistics(Volume 1: Long Papers), pages 1227–1236.
GuoDong Zhou, Jian Su, Jie Zhang, and Min Zhang.2005. Exploring various knowledge in relation ex-traction. In Proceedings of the 43rd annual meet-ing of the association for computational linguistics(acl’05), pages 427–434.
3085
6 Appendix
6.1 Implementation Details
We adopted the pre-trained BERT model [BERT-Base-Cased]10 as our encoder, where the numberof Transformer layers was 12 and the hidden sizewas 768. The token types of input were always setto 0.
We used Adam as our optimizer and applied atriangular learning rate schedule as suggested byoriginal BERT paper. In addition, we adopted alazy mechanism for optimization. Different fromthe momentum mechanism of ordinary Adam opti-mizer (Kingma and Ba, 2015) that updated the out-put layer parameters for all tasks, this lazy-Adammechanism wouldn’t update the parameters of non-current tasks.
The dacay rate ε of EMA was set to 0.99 asdefault. The max sequence length was 128.
The other hyper-parameters were determined onthe validation set. Notably, considering our spe-cial decoding strategy, we raised the threshold ofextraction to 0.9 to balance the precision and therecall. The threshold of relation classification wasset to 0.5 as default. The hyper-parameter settingwas listed in Table 7.
Our mthod were implemented by Pytorch11 andrun on a server configured with a Tesla V100 GPU,16 CPU, and 64G memory.
Table 7: Hyper-parameter setting for NYT andWebNLG datasets.
6.2 Supplementary Experimental Results
Ablation Study. To validate the effectiveness ofcomponents in DIRECT, We implemented severalmodel variants for ablation tests respectively. Forexperimental fairness, we kept the other compo-nents in the same settings when modifying onemodule.
• DIRECTshared, we merged the subject ex-traction and object extraction layers into one
entity extraction layer by sharing the parame-ters of output layers of these two modules.
• DIRECTequal, we replaced the adaptivemulti-task learning strategy with an ordinarylearning strategy. In this strategy, the lossesof three sub-tasks were computed with equalweights, denoted as DIRECTequal.
• DIRECTthreshold, we simply recognized allthe start and end positions of entities by check-ing if the probability pti,start/end > α, where αwas the threshold of extraction.
Table 8: Results of model variants for ablation tests.
From the results of Table 8, we can observe that:
1. Sharing the parameters of output layers ofsubject and object extraction modules wouldreduce the performance of the model.
2. Compared to ordinary multi-task learningstrategy, by using adaptive multi-task learn-ing, DIRECT was able to get a 1.5 percentagepoint improvement on F1-score.
3. There would be a slight drop in performance,if we just used a simple threshold policy torecognize the start and end positions of anentity.
4. Despite the difference in precision and recall,there was no significant difference betweenthese two optimizers (ordinary-Adam & lazy-Adam ) for the task.
Results on Extracting Elements of RelationalTriplets. The complete extraction performanceof relational triplet elements from DIRECT andCaslRel are listed in Table 9. DIRECT outper-formed CasRel in terms of all relational triplet el-ements on both datasets. These empirical results
Table 10: Results of different methods under Exact-Match Metrics. * marks results reproduced by official imple-mentation.
suggest that, for relational triplet extraction, a fullyadjacency list oriented model (DIRECT) may haveadvantages over a partially oriented one (CasRel).
Results of Different Methods under Exact-Match Metrics. In experiment section, we fol-lowed the match metric from (Zeng et al., 2018),which only required to match the first token of en-tity span. Many previous works adopted this matchmetric (Fu et al., 2019; Zeng et al., 2019; Wei et al.,2020).
In fact, our model is capable of extracting thecomplete entities. Therefore, we collected papersthat reported the results of exact-match metrics(requiring to match the complete entity span). Thefollowing strong state-of-the-art (SoTA) modelshave been compared:• CopyMTL (Zeng et al., 2020) is a multi-task
learning framework, where conditional randomfield is used to identify entities, and a seq2seqmodel is adopted to extract relational triplets.•WDec (Nayak and Ng, 2020) fuses a seq2seq
model with a new representation scheme, whichenables the decoder to generate one word at a andcan handle full entity names of different length andoverlapping entities.• PNDec (Nayak and Ng, 2020) is a modification
of seq2seq model. Pointer networks are used in the
decoding framework to identify the entities in thesentence using their start and end locations.• Seq2UMTree (Zhang et al., 2020) is a mod-
ification of seq2seq model, which employs anunordered-multi-tree decoder to to minimize ex-posure bias.
The task performances on NYT dataset are sum-marized in Table 10. The proposed DIRECT modeloutperformed all baseline models in terms of allevaluation metrics. This experimental results fur-ther confirmed the efficacy of DIRECT for rela-tional fact extraction task.
6.3 Complexity Analysis of GraphRepresentations
For a graph G = (V,E), |V | denotes the numberof nodes/entities and |E| denotes the number ofedges/relations. Suppose there are m kinds of rela-tions, d(v) denotes the number of edges from nodev.• Edge List Complexity
− Space: O(|E|)
− Find all edges/relations from a node: O(|E|)
• Adjacency Matrices Complexity
− Space: O(|V | · |V | ·m)
3087
Category Method Theoretical NYT WebNLGEdge List CopyRe 4kl + kr 329 712
Adjacency List (Fully) DIRECT 2l + 2sl + or 238 542
Table 11: Graph representation efficiency based on the theoretical logits amount and the estimated logits amounton two benchmark datasets.
− Find all edges/relations from a node: O(|V | ·m)
• Adjacency List Complexity
− Space: O(|V |+ |E|)
− Find all edges/relations from a node: O(d(v))
6.4 Graph Representation EfficiencyAnalysis
Based on the amount estimation of predicted log-its12 (0/1), we conduct a graph representation ef-ficiency analysis to demonstrate the efficiency ofproposed method13.
For each graph representation category, wechoose one representative model algorithms. EdgeList: CopyRE (Zeng et al., 2018); Adjacency Ma-trices: MHS (Bekoulis et al., 2018); AdjacencyList: CasRel (partially) (Wei et al., 2020) and DI-RECT (fully).
Formally, for a sentence whose length is l (ltokens), there are r types of relations, k denotesthe number of triplets. Suppose there are s keys(subjects) and o values (corresponding amount ofobject-based lists) in adjacency list. The theoreti-cal logits amount and the estimated logits amounton two benchmark datasets (NYT and WebNLG)are shown in Table 11. From the viewpoint ofpredicted logits estimation, DIRECT is the mostrepresentative-efficient model.
12Numeric output of the last layer13As aforementioned, from the graph representation per-
spective, when a method requires fewer logits to represent thegraph (set of triples), it will reduce the model fitting difficulty.
