Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 1706–1721, November 16–20, 2020. c 2020 Association for Computational Linguistics 1706 Two are Better than One: Joint Entity and Relation Extraction with Table-Sequence Encoders Jue Wang 1 and Wei Lu 2 1 College of Computer Science and Technology, Zhejiang University 2 StatNLP Research Group, Singapore University of Technology and Design [email protected], [email protected]Abstract Named entity recognition and relation extrac- tion are two important fundamental problems. Joint learning algorithms have been proposed to solve both tasks simultaneously, and many of them cast the joint task as a table-filling problem. However, they typically focused on learning a single encoder (usually learning rep- resentation in the form of a table) to capture information required for both tasks within the same space. We argue that it can be beneficial to design two distinct encoders to capture such two different types of information in the learn- ing process. In this work, we propose the novel table-sequence encoders where two different encoders – a table encoder and a sequence en- coder are designed to help each other in the representation learning process. Our experi- ments confirm the advantages of having two encoders over one encoder. On several stan- dard datasets, our model shows significant im- provements over existing approaches. 1 1 Introduction Named Entity Recognition (NER, Florian et al. 2006, 2010) and Relation Extraction (RE, Zhao and Grishman 2005; Jiang and Zhai 2007; Sun et al. 2011; Plank and Moschitti 2013) are two fundamental tasks in Information Extraction (IE). Both tasks aim to extract structured information from unstructured texts. One typical approach is to first identify entity mentions, and next perform classification between every two mentions to ex- tract relations, forming a pipeline (Zelenko et al., 2002; Chan and Roth, 2011). An alternative and more recent approach is to perform these two tasks jointly (Li and Ji, 2014; Miwa and Sasaki, 2014; Miwa and Bansal, 2016), which mitigates the error propagation issue associated with the pipeline ap- 1 Our code is available at https://github.com/ LorrinWWW/two-are-better-than-one. ñ ñ Edward Thomas is from Minnesota United live_in live_in loc_in B-PER I-PER B-LOC B-LOC O O O ñ ñ ñ ñ ñ ñ ñ ñ ñ ñ ñ ñ ñ ñ ñ ñ ñ ñ ñ ñ ñ ñ ñ ñ ñ ñ ñ ñ ñ ñ ñ ñ live_in live_in loc_in , States Edward Thomas is from Minnesota , United States ñ ñ live_in live_in live_in live_in ñ ñ I-LOC live_in live_in live_in live_in loc_in loc_in ñ ñ Figure 1: An example of table filling for NER and RE. proach and leverages the interaction between tasks, resulting in improved performance. Among several joint approaches, one popular idea is to cast NER and RE as a table filling prob- lem (Miwa and Sasaki, 2014; Gupta et al., 2016; Zhang et al., 2017). Typically, a two-dimensional (2D) table is formed where each entry captures the interaction between two individual words within a sentence. NER is then regarded as a sequence labeling problem where tags are assigned along the diagonal entries of the table. RE is regarded as the problem of labeling other entries within the table. Such an approach allows NER and RE to be performed using a single model, enabling the potentially useful interaction between these two tasks. One example 2 is illustrated in Figure 1. Unfortunately, there are limitations with the ex- isting joint methods. First, these methods typically suffer from feature confusion as they use a single representation for the two tasks – NER and RE. As a result, features extracted for one task may 2 The exact settings for table filling may be different for different papers. Here we fill the entire table (rather than the lower half of the table), and assign relation tags to cells involving two complete entity spans (rather than part of such spans). We also preserve the direction of the relations.
Named entity recognition and relation extrac-tion are two important fundamental problems.Joint learning algorithms have been proposedto solve both tasks simultaneously, and manyof them cast the joint task as a table-fillingproblem. However, they typically focused onlearning a single encoder (usually learning rep-resentation in the form of a table) to captureinformation required for both tasks within thesame space. We argue that it can be beneficialto design two distinct encoders to capture suchtwo different types of information in the learn-ing process. In this work, we propose the noveltable-sequence encoders where two differentencoders – a table encoder and a sequence en-coder are designed to help each other in therepresentation learning process. Our experi-ments confirm the advantages of having twoencoders over one encoder. On several stan-dard datasets, our model shows significant im-provements over existing approaches.1
1 Introduction
Named Entity Recognition (NER, Florian et al.2006, 2010) and Relation Extraction (RE, Zhaoand Grishman 2005; Jiang and Zhai 2007; Sunet al. 2011; Plank and Moschitti 2013) are twofundamental tasks in Information Extraction (IE).Both tasks aim to extract structured informationfrom unstructured texts. One typical approach isto first identify entity mentions, and next performclassification between every two mentions to ex-tract relations, forming a pipeline (Zelenko et al.,2002; Chan and Roth, 2011). An alternative andmore recent approach is to perform these two tasksjointly (Li and Ji, 2014; Miwa and Sasaki, 2014;Miwa and Bansal, 2016), which mitigates the errorpropagation issue associated with the pipeline ap-
1Our code is available at https://github.com/LorrinWWW/two-are-better-than-one.
⊥
⊥
Edward Thomas is from Minnesota United
live_in
live_in
loc_in
B-PER
I-PER
B-LOC
B-LOC
O
O
O
⊥ ⊥ ⊥
⊥ ⊥ ⊥
⊥
⊥ ⊥ ⊥ ⊥
⊥ ⊥ ⊥
⊥
⊥
⊥
⊥ ⊥
⊥ ⊥ ⊥
⊥ ⊥ ⊥
⊥ ⊥
⊥ ⊥
⊥ ⊥
⊥
live_inlive_in loc_in
, States
Edward
Thomas
is
from
Minnesota
,
United
States ⊥ ⊥live_inlive_in
live_in
live_in
⊥
⊥
I-LOC
live_inlive_in
live_in
live_in
loc_in
loc_in ⊥
⊥
Figure 1: An example of table filling for NER and RE.
proach and leverages the interaction between tasks,resulting in improved performance.
Among several joint approaches, one popularidea is to cast NER and RE as a table filling prob-lem (Miwa and Sasaki, 2014; Gupta et al., 2016;Zhang et al., 2017). Typically, a two-dimensional(2D) table is formed where each entry captures theinteraction between two individual words withina sentence. NER is then regarded as a sequencelabeling problem where tags are assigned alongthe diagonal entries of the table. RE is regardedas the problem of labeling other entries within thetable. Such an approach allows NER and RE tobe performed using a single model, enabling thepotentially useful interaction between these twotasks. One example2 is illustrated in Figure 1.
Unfortunately, there are limitations with the ex-isting joint methods. First, these methods typicallysuffer from feature confusion as they use a singlerepresentation for the two tasks – NER and RE.As a result, features extracted for one task may
2The exact settings for table filling may be different fordifferent papers. Here we fill the entire table (rather thanthe lower half of the table), and assign relation tags to cellsinvolving two complete entity spans (rather than part of suchspans). We also preserve the direction of the relations.
coincide or conflict with those for the other, thusconfusing the learning model. Second, these meth-ods underutilize the table structure as they usuallyconvert it to a sequence and then use a sequencelabeling approach to fill the table. However, cru-cial structural information (e.g., the 4 entries atthe bottom-left corner of Figure 1 share the samelabel) in the 2D table might be lost during suchconversions.
In this paper, we present a novel approach toaddress the above limitations. Instead of predictingentities and relations with a single representation,we focus on learning two types of representations,namely sequence representations and table repre-sentations, for NER and RE respectively. On onehand, the two separate representations can be usedto capture task-specific information. On the otherhand, we design a mechanism to allow them to in-teract with each other, in order to take advantage ofthe inherent association underlying the NER andRE tasks. In addition, we employ neural networkarchitectures that can better capture the structuralinformation within the 2D table representation. Aswe will see, such structural information (in particu-lar the context of neighboring entries in the table)is essential in achieving better performance.
The recent prevalence of BERT (Devlin et al.,2019) has led to great performance gains on variousNLP tasks. However, we believe that the previoususe of BERT, i.e., employing the contextualizedword embeddings, does not fully exploit its poten-tial. One important observation here is that the pair-wise self-attention weights maintained by BERTcarry knowledge of word-word interactions. Ourmodel can effectively use such knowledge, whichhelps to better learn table representations. To thebest of our knowledge, this is the first work to usethe attention weights of BERT for learning tablerepresentations.
We summarize our contributions as follows:
• We propose to learn two separate encoders – atable encoder and a sequence encoder. Theyinteract with each other, and can capture task-specific information for the NER and RE tasks;
• We propose to use multidimensional recurrentneural networks to better exploit the structuralinformation of the table representation;
• We effectively leverage the word-word interac-tion information carried in the attention weightsfrom BERT, which further improves the perfor-mance.
Our proposed method achieves the state-of-the-art performance on four datasets, namely ACE04,ACE05, CoNLL04, and ADE. We also conductfurther experiments to confirm the effectiveness ofour proposed approach.
2 Related Work
NER and RE can be tackled by using separate mod-els. By assuming gold entity mentions are givenas inputs, RE can be regarded as a classificationtask. Such models include kernel methods (Ze-lenko et al., 2002), RNNs (Zhang and Wang, 2015),recursive neural networks (Socher et al., 2012),CNNs (Zeng et al., 2014), and Transformer models(Verga et al., 2018; Wang et al., 2019). Anotherbranch is to detect cross-sentence level relations(Peng et al., 2017; Gupta et al., 2019), and evendocument-level relations (Yao et al., 2019; Nanet al., 2020). However, entities are usually not di-rectly available in practice, so these approachesmay require an additional entity recognizer to forma pipeline.
Joint learning has been shown effective since itcan alleviate the error propagation issue and bene-fit from exploiting the interrelation between NERand RE. Many studies address the joint problemthrough a cascade approach, i.e., performing NERfirst followed by RE. Miwa and Bansal (2016) usebi-LSTM (Graves et al., 2013) and tree-LSTM (Taiet al., 2015) for the joint task. Bekoulis et al.(2018a,b) formulate it as a head selection prob-lem. Nguyen and Verspoor (2019) apply biaffineattention (Dozat and Manning, 2017) for RE. Luanet al. (2019), Dixit and Al (2019), and Wadden et al.(2019) use span representations to predict relations.
Miwa and Sasaki (2014) tackle joint NER andRE as from a table filling perspective, where the en-try at row i and column j of the table corresponds tothe pair of i-th and j-th word of the input sentence.The diagonal of the table is filled with the entitytags and the rest with the relation tags indicatingpossible relations between word pairs. Similarly,Gupta et al. (2016) employ a bi-RNN structure tolabel each word pair. Zhang et al. (2017) propose aglobal optimization method to fill the table. Tranand Kavuluru (2019) investigate CNNs on this task.
Recent work (Luan et al., 2019; Dixit and Al,2019; Wadden et al., 2019; Li et al., 2019; Ebertsand Ulges, 2019) usually leverages pre-trained lan-guage models such as ELMo (Peters et al., 2018),BERT (Devlin et al., 2019), RoBERTa (Liu et al.,
1708
7IUYIRGI�)RGSHMRK
)RGSHMRK0E]IV��
4VIHMGXMSR
� �� �K1(5
��
��
��
��
� �� �K5(
� �
� �
� �
8I\X�)QFIHHIV
5� �
8EFPI�)RGSHMRK
8EFPI�)RGSHMRK
)RGSHMRK0E]IV��
8EFPI�)RGSHMRK
)RGSHMRK0E]IV�0
4VIHMGXMSR
7IUYIRGI�)RGSHMRK
7IUYIRGI�)RGSHMRK
Figure 2: Overview of the table-sequence encoders.Dashed lines are for optional components (T `).
2019), and ALBERT (Lan et al., 2019). However,none of them use pre-trained attention weights,which convey rich relational information betweenwords. We believe it can be useful for learningbetter table representations for RE.
3 Problem Formulation
In this section, we formally formulate the NERand RE tasks. We regard NER as a sequence la-beling problem, where the gold entity tags yNER
are in the standard BIO (Begin, Inside, Outside)scheme (Sang and Veenstra, 1999; Ratinov andRoth, 2009). For the RE task, we mainly followthe work of Miwa and Sasaki (2014) to formulateit as a table filling problem. Formally, given aninput sentence x = [xi]1≤i≤N , we maintain a tagtable yRE = [yRE
i,j ]1≤i,j≤N . Suppose there is a rela-tion with type r pointing from mention xib , .., xieto mention xjb , .., xje , we have yRE
i,j = −→r andyREj,i = ←−r for all i ∈ [ib, ie] ∧ j ∈ [jb, je]. We
use ⊥ for word pairs with no relation. An examplewas given earlier in Figure 1.
4 Model
We describe the model in this section. The modelconsists of two types of interconnected encoders,a table encoder for table representation and a se-quence encoder for sequence representation, asshown in Figure 2. Collectively, we call them table-sequence encoders. Figure 3 presents the details ofeach layer of the two encoders, and how they inter-act with each other. In each layer, the table encoderuses the sequence representation to construct the
� )À�
1(�622
0MRIEV�6I09
'SRGEX�EW8EFPI
� �
%HH��2SVQ
*IIH*SV[EVH
%HH��2SVQ
�)
8EFPI�+YMHIH%XXIRXMSR
�)À�
� )
Figure 3: A layer in the table-sequence encoders.
table representation; and then the sequence encoderuses the table representation to contextualize thesequence representation. With multiple layers, weincrementally improve the quality of both represen-tations.
4.1 Text EmbedderFor a sentence containingN words x = [xi]1≤i≤N ,we define the word embeddings xw ∈ RN×d1 , aswell as character embeddings xc ∈ RN×d2 com-puted by an LSTM (Lample et al., 2016). Wealso consider the contextualized word embeddingsx` ∈ RN×d3 , which can be produced from lan-guage models such as BERT.
We concatenate those embeddings for each wordand use a linear projection to form the initial se-quence representation S0 ∈ RN×H :
S0 = Linear([xc;xw;x`]) (1)
where each word is represented as an H dimen-sional vector.
4.2 Table EncoderThe table encoder, shown in the left part of Figure3, is a neural network used to learn a table repre-sentation, an N × N table of vectors, where thevector at row i and column j corresponds to thei-th and j-th word of the input sentence.
We first construct a non-contextualized table byconcatenating every two vectors of the sequencerepresentation followed by a fully-connected layerto halve the hidden size. Formally, for the l-th layer,we have X l ∈ RN×N×H , where:
Xl,i,j = ReLU(Linear([Sl−1,i;Sl−1,j ])) (2)
Next, we use the Multi-Dimensional RecurrentNeural Networks (MD-RNN, Graves et al. 2007)
1709
� )À�
�)�&�'
MRHMVIGX�GSRXI\X�EVIE
� )
���aNC[GT TQY EQN� � �
���aNC[GT TQY EQN� � À � �aNC[GT TQY EQN� À À �!�aNC[GT TQY EQN� À �
HMVIGX�GSRXI\X�EVIE
GYVVIRX�GIPP
HMWGSRRIGXIH�EVIE
5� 5� 5� 5�
5�
5�
5�
5�
Figure 4: How the hidden states are computed in MD-RNN with 4 directions. We use D+ or D− to indicatethe direction that the hidden states flow between cellsat the D dimension (where D can be layer, row or col).For brevity, we omit the input and the layer dimensionfor cases (b), (c) and (d), as they are the same as (a).
with Gated Recurrent Unit (GRU, Cho et al. 2014)to contextualize X l. We iteratively compute thehidden states of each cell to form the contextualizedtable representation T l, where:
We provide the multi-dimensional adaptations ofGRU in Appendix A to avoid excessive formulashere.
Generally, it exploits the context along layer,row, and column dimensions. That is, it does notconsider only the cells at neighbouring rows andcolumns, but also those of the previous layer.
The time complexity of the naive implementa-tion (i.e., two for-loops) for each layer isO(N×N)for a sentence with length N . However, antidiago-nal entries3 can be calculated at the same time asthey do not depend on each other. Therefore, wecan optimize it through parallelization and reducethe effective time complexity to O(N).
The above illustration describes a unidirectionalRNN, corresponding to Figure 4(a). Intuitively,we would prefer the network to have access to thesurrounding context in all directions. However, thiscould not be done by one single RNN. For the caseof 1D sequence modeling, this problem is resolvedby introducing bidirectional RNNs. Graves et al.(2007) discussed quaddirectional RNNs to accessthe context from four directions for modeling 2Ddata. Therefore, similar to 2D-RNN, we also need
3We define antidiagonal entries to be entries at position(i, j) such that i+j = N+1+∆, where ∆ ∈ [−N+1, N−1]is the offset to the main antidiagonal entries.
#
�
�
�
WSJXQE\
jSYXTYXW
UYIVMIW
OI]W
ZEPYIW
Figure 5: The generalized form of attention. The soft-max function is used to normalize the weights of valuesV for each query Qi.
to consider RNNs in four directions4. We visualizethem in Figure 4.
Empirically, we found the setting only consider-ing cases (a) and (c) in Figure 4 achieves no worseperformance than considering four cases altogether.Therefore, to reduce the amount of computation,we use such a setting as default. The final table rep-resentation is then the concatenation of the hiddenstates of the two RNNs:
T(a)l,i,j = GRU(a)(Xl,i,j , T
(a)l−1,i,j , T
(a)l,i−1,j , T
(a)l,i,j−1) (4)
T(c)l,i,j = GRU(c)(Xl,i,j , T
(c)l−1,i,j , T
(c)l,i+1,j , T
(c)l,i,j+1) (5)
Tl,i,j = [T(a)l,i,j ;T
(c)l,i,j ] (6)
4.3 Sequence EncoderThe sequence encoder is used to learn the sequencerepresentation – a sequence of vectors, where thei-th vector corresponds to the i-th word of the inputsentence. The architecture is similar to Transformer(Vaswani et al., 2017), shown in the right portionof Figure 3. However, we replace the scaled dot-product attention with our proposed table-guidedattention. Here, we mainly illustrate why and howthe table representation can be used to computeattention weights.
First of all, given Q (queries), K (keys) and V(values), a generalized form of attention is definedin Figure 5. For each query, the output is a weightedsum of the values, where the weight assigned toeach value is determined by the relevance (givenby score function f ) of the query with all the keys.
For each query Qi and key Kj , Bahdanau et al.(2015) define f in the form of:
f(Qi,Kj) = U · g(Qi,Kj) (7)
where U is a learnable vector and g is the functionto map each query-key pair to a vector. Specifically,
4In our scenario, there is an additional layer dimension.However, as the model always traverses from the first layer tothe last layer, only one direction shall be considered for thelayer dimension.
1710
they define g(Qi,Kj) = tanh(QiW0 + KjW1),where W0,W1 are learnable parameters.
Our attention mechanism is essentially a self-attention mechanism, where the queries, keys andvalues are exactly the same. In our case, they areessentially sequence representation Sl−1 of the pre-vious layer (i.e., Q = K = V = Sl−1). Theattention weights (i.e., the output from the functionf in Figure 5) are essentially constructed from bothqueries and keys (which are the same in our case).On the other hand, we also notice the table represen-tation T l is also constructed from Sl−1. So we canconsider T l to be a function of queries and keys,such that Tl,i,j = g(Sl−1,i, Sl−1,j) = g(Qi,Kj).Then we put back this g function to Equation 7,and get the proposed table-guided attention, whosescore function is:
f(Qi,Kj) = U · Tl,i,j (8)
We show the advantages of using this table-guided attention: (1) we do not have to calculateg function since T l is already obtained from thetable encoder; (2) T l is contextualized along therow, column, and layer dimensions, which corre-sponds to queries, keys, and queries and keys in theprevious layer, respectively. Such contextual infor-mation allows the network to better capture moredifficult word-word dependencies; (3) it allows thetable encoder to participate in the sequence rep-resentation learning process, thereby forming thebidirectional interaction between the two encoders.
The table-guided attention can be extended tohave multiple heads (Vaswani et al., 2017), whereeach head is an attention with independent parame-ters. We concatenate their outputs and use a fully-connected layer to get the final attention outputs.
The remaining parts are similar to Transformer.For layer l, we use position-wise feedforward neu-ral networks (FFNN) after self-attention, and wrapattention and FFNN with a residual connection(He et al., 2016) and layer normalization (Ba et al.2016), to get the output sequence representation:
Sl = LayerNorm(Sl−1 + SelfAttn(Sl−1)) (9)
Sl = LayerNorm(Sl + FFNN(Sl)) (10)
4.4 Exploit Pre-trained Attention WeightsIn this section, we describe the dashed lines inFigures 2 and 3, which we ignored in the previousdiscussions. Essentially, they exploit informationin the form of attention weights from a pre-trainedlanguage model such as BERT.
We stack the attention weights of all heads andall layers to form T ` ∈ RN×N×(L`×A`), whereL` is the number of stacked Transformer layers,and A` is the number of heads in each layer. Weleverage T ` to form the inputs of MD-RNNs in thetable encoder. Equation 2 is now replaced with:
We keep the rest unchanged. We believe this sim-ple yet novel use of the attention weights allows usto effectively incorporate the useful word-word in-teraction information captured by pre-trained mod-els such as BERT into our table-sequence encodersfor improved performance.
5 Training and Evaluation
We use SL and T L to predict the probability distri-bution of the entity and relation tags:
Pθ(YNER) = softmax(Linear(SL)) (12)
Pθ(YRE) = softmax(Linear(T L)) (13)
where Y NER and Y RE are random variables of thepredicted tags, and Pθ is the estimated probabilityfunction with θ being our model parameters.
For training, both NER and RE adopt the preva-lent cross-entropy loss. Given the input text x andits gold tag sequence yNER and tag table yRE, wethen calculate the following two losses:
LNER =∑
i∈[1,N ]
− logPθ(YNERi = yNER
i ) (14)
LRE =∑
i,j∈[1,N ];i 6=j
− logPθ(YREi,j = yRE
i,j ) (15)
The goal is to minimize both losses LNER +LRE.During evaluation, the prediction of relations re-
lies on the prediction of entities, so we first predictthe entities, and then look up the relation proba-bility table Pθ(Y RE) to see if there exists a validrelation between predicted entities.
Specifically, we predict the entity tag of eachword by choosing the class with the highest proba-bility:
argmaxe
Pθ(YNERi = e) (16)
The whole tag sequence can be transformed intoentities with their boundaries and types.
Relations on entities are mapped to relationclasses with highest probabilities on words of theentities. We also consider the two directed tags
1711
for each relation. Therefore, for two entity spans(ib, ie) and (jb, je), their relation is given by:
argmax−→r
∑i∈[ib,ie],j∈[jb,je]
Pθ(YREi,j = −→r ) + Pθ(Y
REj,i =←−r ) (17)
where the no-relation type ⊥ has no direction, so if−→r = ⊥, we have←−r = ⊥ as well.
6 Experiments
6.1 DataWe evaluate our model on four datasets, namelyACE04 (Doddington et al., 2004), ACE05 (Walkeret al., 2006), CoNLL04 (Roth and tau Yih, 2004)and ADE (Gurulingappa et al., 2012). More detailscould be found in Appendix B.
Following the established line of work, we usethe F1 measure to evaluate the performance of NERand RE. For NER, an entity prediction is correct ifand only if its type and boundaries both match withthose of a gold entity.5 For RE, a relation predictionis considered correct if its relation type and theboundaries of the two entities match with thosein the gold data. We also report the strict relationF1 (denoted RE+), where a relation prediction isconsidered correct if its relation type as well asthe boundaries and types of the two entities allmatch with those in the gold data. Relations areasymmetric, so the order of the two entities in arelation matters.
6.2 Model SetupWe tune hyperparameters based on results on thedevelopment set of ACE05 and use the same set-ting for other datasets. GloVe vectors (Penningtonet al., 2014) are used to initialize word embed-dings. We also use the BERT variant – ALBERTas the default pre-trained language model. Bothpre-trained word embeddings and language modelare fixed without fine-tuning. In addition, we stackthree encoding layers (L = 3) with independentparameters including the GRU cell in each layer.For the table encoder, we use two separate MD-RNNs with the directions of “layer+row+col+”and “layer+row−col−” respectively. For the se-quence encoder, we use eight attention heads toattend to different representation subspaces. Wereport the averaged F1 scores of 5 runs for our mod-els. For each run, we keep the model that achieves
5Follow Li and Ji (2014); Miwa and Bansal (2016), we usehead spans for entities in ACE. And we keep the full mentionboundary for other corpora.
Data Model NER RE RE+
AC
E04
Li and Ji (2014) O 79.7 48.3 45.3Katiyar and Cardie (2017) O 79.6 49.3 45.7Bekoulis et al. (2018b) O 81.2 - 47.1Bekoulis et al. (2018a) O 81.6 - 47.5Miwa and Bansal (2016) O 81.8 - 48.4Li et al. (2019) O 83.6 - 49.4Luan et al. (2019) O 87.4 59.7 -
Ours O 88.6 63.3 59.6
AC
E05
Li and Ji (2014) O 80.8 52.1 49.5Miwa and Bansal (2016) O 83.4 - 55.6Katiyar and Cardie (2017) O 82.6 55.9 53.6Zhang et al. (2017) O 83.6 - 57.5Sun et al. (2018) O 83.6 - 59.6Li et al. (2019) O 84.8 - 60.2Dixit and Al (2019) O 86.0 62.8 -Luan et al. (2019) O 88.4 63.2 -Wadden et al. (2019) O 88.6 63.4 -
Ours O 89.5 67.6 64.3
CoN
LL
04
Miwa and Sasaki (2014)O 80.7 - 61.0Bekoulis et al. (2018a)N 83.6 - 62.0Bekoulis et al. (2018b)N 83.9 - 62.0Tran and Kavuluru (2019)N 84.2 - 62.3Nguyen and Verspoor (2019)N 86.2 - 64.4Zhang et al. (2017)O 85.6 - 67.8Li et al. (2019)O 87.8 - 68.9Eberts and Ulges (2019)O 88.9 - 71.5Eberts and Ulges (2019)N 86.3 - 72.9
OursO 90.1 73.8 73.6OursN 86.9 75.8 75.4
AD
E
Li et al. (2016) N 79.5 - 63.4Li et al. (2017) N 84.6 - 71.4Bekoulis et al. (2018b) N 86.4 - 74.6Bekoulis et al. (2018a) N 86.7 - 75.5Tran and Kavuluru (2019) N 87.1 - 77.3Eberts and Ulges (2019) N 89.3 - 79.2
Ours N 89.7 80.1 80.1
Table 1: Main results. O: micro-averaged F1; N: macro-averaged F1.
the highest averaged entity F1 and relation F1 onthe development set, and evaluate and report itsscore on the test set. Other hyperparameters couldbe found in Appendix C.
6.3 Comparison with Other Models
Table 1 presents the comparison of our model withprevious methods on four datasets. Our NER per-formance is increased by 1.2, 0.9, 1.2/0.6 and 0.4absolute F1 points over the previous best results.Besides, we observe even stronger performancegains in the RE task, which are 3.6, 4.2, 2.1/2.5(RE+) and 0.9 (RE+) absolute F1 points, respec-tively. This indicates the effectiveness of our model
Table 2: Using different pre-trained language modelson ACE05. +x` uses the contextualized word embed-dings; +T ` uses the attention weights.
for jointly extracting entities and their relations.Since our reported numbers are the average of 5runs, we can consider our model to be achievingnew state-of-the-art results.
6.4 Comparison of Pre-trained Models
In this section, we evaluate our method with differ-ent pre-trained language models, including ELMo,BERT, RoBERTa and ALBERT, with and withoutattention weights, to see their individual contribu-tion to the final performance.
Table 2 shows that, even using the relatively ear-lier contextualized embeddings without attentionweights (ELMo +x`), our system is still compara-ble to the state-of-the-art approach (Wadden et al.,2019), which was based on BERT and achievedF1 scores of 88.6 and 63.4 for NER and RE re-spectively. It is important to note that the modelof Wadden et al. (2019) was trained on the ad-ditional coreference annotations from OntoNotes(Weischedel et al., 2011) before fine-tuning onACE05. Nevertheless, our system still achievescomparable results, showing the effectiveness ofthe table-sequence encoding architecture.
The overall results reported in Table 2 confirmthe importance of leveraging the attention weights,which bring improvements for both NER and REtasks. This allows the system using vanilla BERTto obtain results no worse than RoBERTa and AL-BERT in relation extraction.
6.5 Ablation Study
We design several additional experiments to un-derstand the effectiveness of components in oursystem. The experiments are conducted on ACE05.
We also compare different table filling settings,which are included in Appendix E.
6.5.1 Bidirectional InteractionWe first focus on the understanding of the necessityof modeling the bidirectional interaction between
NER on diagonal 89.4 67.1 70.2w/o Sequence Encoder 88.6 67.0 70.2
Table 3: Ablation of the two encoders on ACE05. Goldentity spans are given in RE (gold).
the two encoders. Results are presented in Table3. “RE (gold)” is presented so as to compare withsettings that do not predict entities, where the goldentity spans are used in the evaluation.
We first try optimizing the NER and RE objec-tives separately, corresponding to “w/o RelationLoss” and “w/o Entity Loss”. Compared with learn-ing with a joint objective, the results of these twosettings are slightly worse, which indicates thatlearning better representations for one task not onlyis helpful for the corresponding task, but also canbe beneficial for the other task.
Next, we investigate the individual sequence andtable encoder, corresponding to “w/o Table En-coder” and “w/o Sequence Encoder”. We alsotry jointly training the two encoders but cut offthe interaction between them, which is “w/o Bi-Interaction”. Since no interaction is allowed inthe above three settings, the table-guided attentionis changed to conventional multi-head scaled dot-product attention, and the table encoding layer al-ways uses the initial sequence representation S0 toenrich the table representation. The results of thesesettings are all significantly worse than the defaultone, which indicates the importance of the bidi-rectional interaction between sequence and tablerepresentation in our table-sequence encoders.
We also experiment the use of the main diagonalentries of the table representation to tag entities,with results reported under “NER on diagonal”.This setup attempts to address NER and RE inthe same encoding space, in line with the originalintention of Miwa and Sasaki (2014). By exploitingthe interrelation between NER and RE, it achievesbetter performance compared with models withoutsuch information. However, it is worse than ourdefault setting. We ascribe this to the potentialincompatibility of the desired encoding space ofentities and relations. Finally, although it does not
Table 4: The performance on ACE05 with differentnumber of layers. Pre-trained word embeddings andlanguage models are not counted to the number of pa-rameters. The underlined ones are from our default set-ting.
directly use the sequence representation, removingthe sequence encoder will lead to performance dropfor NER, which indicates the sequence encoder canhelp improve the table encoder by better capturingthe structured information within the sequence.
6.5.2 Encoding LayersTable 4 shows the effect of the number of encod-ing layers, which is also the number of bidirec-tional interactions involved. We conduct one setof experiments with shared parameters for the en-coding layers and another set with independentparameters. In general, the performance increaseswhen we gradually enlarge the number of layers L.Specifically, since the shared model does not intro-duce more parameters when tuning L, we considerthat our model benefits from the mutual interactioninside table-sequence encoders. Typically, underthe same value L, the non-shared model employsmore parameters than the shared one to enhance itsmodeling capability, leading to better performance.However, when L > 3, there is no significant im-provement by using non-shared model. We believethat increasing the number of layers may bring therisk of over-fitting, which limits the performanceof the network. We choose to adopt the non-sharedmodel with L = 3 as our default setting.
6.5.3 Settings of MD-RNNTable 5 presents the comparisons of using differentdimensions and directions to learn the table repre-sentation, based on MD-RNN. Among those set-tings, “Unidirectional” refers to an MD-RNN withdirection “layer+row+col+”; “Bidirectional” usestwo MD-RNNs with directions “layer+row+col+”and “layer+row−col−” respectively; “Quaddirec-tional” uses MD-RNNs in four directions, illus-trated in Figure 4. Their results are improved whenadding more directions, showing richer contextual
Table 5: The effect of the dimensions and directionsof MD-RNNs. Experiments are conducted on ACE05.The underlined ones are from our default setting.
information is beneficial. Since the bidirectionalmodel is almost as good as the quaddirectional one,we leave the former as the default setting.
In addition, we are also curious about the con-tribution of layer, row, and column dimensionsfor MD-RNNs. We separately removed the layer,row, and column dimension. As we can see, theresults are all lower than the original model with-out removal of any dimension. “Layer-wise only”removed row and col dimensions, and is worse thanothers as it does not exploit the sentential context.
More experiments with more settings are pre-sented in Appendix D. Specifically, all unidirec-tional RNNs are consistently worse than others,while bidirectional RNNs are usually on-par withquaddirectional RNNs. Besides, we also tried touse CNNs to implement the table encoder. How-ever, since it is usually difficult for CNNs to learnlong-range dependencies, we found the perfor-mance was worse than the RNN-based models.
6.6 Attention Visualization
We visualize the table-guided attention withbertviz (Vig, 2019)6 for a better understandingof how the network works. We compare it withpre-trained Transformers (ALBERT) and human-defined ground truth, as presented in Figure 6.
Our discovery is similar to Clark et al. (2019).Most attention heads in the table-guided attentionand ALBERT show simple patterns. As shownin the left part of Figure 6, these patterns includeattending to the word itself, the next word, the lastword, and the punctuation.
The right part of Figure 6 also shows task-relatedpatterns, i.e., entities and relations. For a relation,we connect words from the head entity to the tailentity; For an entity, we connect every two wordsinside this entity mention. We can find that our pro-
Figure 6: Comparison between ground truth and selected heads of ALBERT and table-guided attention. Thesentence is randomly selected from the development set of ACE05.
Labels
PredictionLayer 1
PredictionLayer 2
PredictionLayer 3
An Abu Dhabi TV office in Baghdad also was hit
ORG FAC GPE
PART-WHOLEART
An Abu Dhabi office in Baghdad also was hit
ORG FAC GPE
PHYSORG-AFF
TV
FAC ORG
An Abu Dhabi TV office in Baghdad also was hit
ORG FAC GPE
PART-WHOLE
An Abu Dhabi TV office in Baghdad also was hit
ORG FAC GPE
PART-WHOLEART
Figure 7: Probing intermediate states
posed table-guided attention has learned more task-related knowledge compared to ALBERT. In fact,not only does it capture the entities and their rela-tions that ALBERT failed to capture, but it also hashigher confidence. This indicates that our modelhas a stronger ability to capture complex patternsother than simple ones.
6.7 Probing Intermediate States
Figure 7 presents an example picked from the de-velopment set of ACE05. The prediction layer aftertraining (a linear layer) is used as a probe to dis-play the intermediate state of the model, so we caninterpret how the model improves both representa-tions from stacking multiple layers and thus fromthe bidirectional interaction. Such probing is valid
since we use skip connection between two adjacentencoding layers, so the encoding spaces of the out-puts of different encoding layers are consistent andtherefore compatible with the prediction layer.
In Figure 7, the model made many wrong pre-dictions in the first layer, which were graduallycorrected in the next layers. Therefore, we can seethat more layers allow more interaction and thusmake the model better at capturing entities or re-lations, especially difficult ones. More cases arepresented in Appendix F.
7 Conclusion
In this paper, we introduce the novel table-sequence encoders architecture for joint extractionof entities and their relations. It learns two separateencoders rather than one – a sequence encoder anda table encoder where explicit interactions exist be-tween the two encoders. We also introduce a newmethod to effectively employ useful informationcaptured by the pre-trained language models forsuch a joint learning task where a table represen-tation is involved. We achieved state-of-the-art F1scores for both NER and RE tasks across four stan-dard datasets, which confirm the effectiveness ofour approach. In the future, we would like to inves-tigate how the table representation may be appliedto other tasks. Another direction is to generalizethe way in which the table and sequence interact toother types of representations.
1715
Acknowledgements
We would like to thank the anonymous reviewersfor their helpful comments and Lidan Shou for hissuggestions and support on this work. This workwas done during the first author’s remote internshipwith the StatNLP Group in Singapore University ofTechnology and Design. This research is supportedby Ministry of Education, Singapore, under its Aca-demic Research Fund (AcRF) Tier 2 Programme(MOE AcRF Tier 2 Award No: MOE2017-T2-1-156). Any opinions, findings and conclusions orrecommendations expressed in this material arethose of the authors and do not reflect the views ofthe Ministry of Education, Singapore.
Rasul, Stefan Schweter, and Roland Vollgraf. 2019.Flair: An easy-to-use framework for state-of-the-artnlp. In Proc. of NAACL-HLT.
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hin-ton. 2016. Layer normalization. arXiv preprint.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben-gio. 2015. Neural machine translation by jointlylearning to align and translate. In Proc. of ICLR.
Giannis Bekoulis, Johannes Deleu, Thomas Demeester,and Chris Develder. 2018a. Adversarial training formulti-context joint entity and relation extraction. InProc. of EMNLP.
Giannis Bekoulis, Johannes Deleu, Thomas Demeester,and Chris Develder. 2018b. Joint entity recogni-tion and relation extraction as a multi-head selectionproblem. Expert Systems with Applications.
Yee Seng Chan and Dan Roth. 2011. Exploitingsyntactico-semantic structures for relation extrac-tion. In Proc. of NAACL-HLT.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gul-cehre, Dzmitry Bahdanau, Fethi Bougares, HolgerSchwenk, and Yoshua Bengio. 2014. Learningphrase representations using rnn encoder–decoderfor statistical machine translation. In Proc. ofEMNLP.
Kevin Clark, Urvashi Khandelwal, Omer Levy, andChristopher D. Manning. 2019. What does bert lookat? an analysis of bert’s attention. arXiv preprint.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. Bert: Pre-training of deepbidirectional transformers for language understand-ing. In Proc. of NAACL-HLT.
Kalpit Dixit and Yaser Al. 2019. Span-level model forrelation extraction. In Proc. of ACL.
George R. Doddington, Alexis Mitchell, Mark A. Przy-bocki, Lance A. Ramshaw, Stephanie M. Strassel,and Ralph M. Weischedel. 2004. The automatic con-tent extraction (ace) program-tasks, data, and evalu-ation. In Proc. of LREC.
Timothy Dozat and Christopher D. Manning. 2017.Deep biaffine attention for neural dependency pars-ing. In Proc. of ICLR.
Markus Eberts and Adrian Ulges. 2019. Span-basedjoint entity and relation extraction with transformerpre-training. arXiv preprint.
Radu Florian, Hongyan Jing, Nanda Kambhatla, andImed Zitouni. 2006. Factorizing complex models:A case study in mention detection. In Proc. of ACL.
Radu Florian, John F. Pitrelli, Salim Roukos, and ImedZitouni. 2010. Improving mention detection robust-ness to noisy input. In Proc. of EMNLP.
Alex Graves, Santiago Fernandez, and Jurgen Schmid-huber. 2007. Multi-dimensional recurrent neural net-works. In Proc. of ICANN.
Alex Graves, Abdel-rahman Mohamed, and Geof-frey E. Hinton. 2013. Speech recognition with deeprecurrent neural networks. In Proc. of ICASSP.
Pankaj Gupta, Subburam Rajaram, Hinrich Schutze,and Thomas Runkler. 2019. Neural relation extrac-tion within and across sentence boundaries. In Proc.of AAAI.
Pankaj Gupta, Hinrich Schutze, and Bernt Andrassy.2016. Table filling multi-task recurrent neural net-work for joint entity and relation extraction. In Proc.of COLING.
Harsha Gurulingappa, Abdul Mateen Rajput, AngusRoberts, Juliane Fluck, Martin Hofmann-Apitius,and Luca Toldo. 2012. Development of a bench-mark corpus to support the automatic extraction ofdrug-related adverse effects from medical case re-ports. Journal of biomedical informatics.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and JianSun. 2016. Deep residual learning for image recog-nition. In Proc. of CVPR.
Jing Jiang and ChengXiang Zhai. 2007. A systematicexploration of the feature space for relation extrac-tion. In Proc. of HLT-NAACL.
Arzoo Katiyar and Claire Cardie. 2017. Going out ona limb: Joint extraction of entity mentions and rela-tions without dependency trees. In Proc. of ACL.
Guillaume Lample, Miguel Ballesteros, Sandeep Sub-ramanian, Kazuya Kawakami, and Chris Dyer. 2016.Neural architectures for named entity recognition.In Proc. of HLT-NAACL.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman,Kevin Gimpel, Piyush Sharma, and Radu Soricut.2019. Albert: A lite bert for self-supervised learn-ing of language representations. In Proc. of ICLR.
1716
Fei Li, Meishan Zhang, Guohong Fu, and Donghong Ji.2017. A neural joint model for entity and relationextraction from biomedical text. BMC bioinformat-ics.
Fei Li, Yue Zhang, Meishan Zhang, and DonghongJi. 2016. Joint models for extracting adverse drugevents from biomedical text. In Proc. of IJCAI.
Qi Li and Heng Ji. 2014. Incremental joint extractionof entity mentions and relations. In Proc. of ACL.
Xiaoya Li, Fan Yin, Zijun Sun, Xiayu Li, AriannaYuan, Duo Chai, Mingxin Zhou, and Jiwei Li. 2019.Entity-relation extraction as multi-turn question an-swering. In Proc. of ACL.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man-dar Joshi, Danqi Chen, Omer Levy, Mike Lewis,Luke Zettlemoyer, and Veselin Stoyanov. 2019.Roberta: A robustly optimized bert pretraining ap-proach. arXiv preprint.
Yi Luan, Dave Wadden, Luheng He, Amy Shah, MariOstendorf, and Hannaneh Hajishirzi. 2019. A gen-eral framework for information extraction using dy-namic span graphs. In Proc. of NAACL-HLT.
Makoto Miwa and Mohit Bansal. 2016. End-to-end re-lation extraction using lstms on sequences and treestructures. In Proc. of ACL.
Makoto Miwa and Yutaka Sasaki. 2014. Modelingjoint entity and relation extraction with table repre-sentation. In Proc. of EMNLP.
Guoshun Nan, Zhijiang Guo, Ivan Sekulic, and Wei Lu.2020. Reasoning with latent structure refinement fordocument-level relation extraction. In Proc. of ACL.
Dat Quoc Nguyen and Karin Verspoor. 2019. End-to-end neural relation extraction using deep biaffine at-tention. In Proc. of ECIR.
Nanyun Peng, Hoifung Poon, Chris Quirk, KristinaToutanova, and Wen-tau Yih. 2017. Cross-sentencen-ary relation extraction with graph lstms. Transac-tions of the Association for Computational Linguis-tics.
Jeffrey Pennington, Richard Socher, and Christopher D.Manning. 2014. Glove: Global vectors for word rep-resentation. In Proc. of EMNLP.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, MattGardner, Christopher Clark, Kenton Lee, and LukeZettlemoyer. 2018. Deep contextualized word repre-sentations. In Proc. of NAACL.
Barbara Plank and Alessandro Moschitti. 2013. Em-bedding semantic similarity in tree kernels for do-main adaptation of relation extraction. In Proc. ofACL.
Lev-Arie Ratinov and Dan Roth. 2009. Design chal-lenges and misconceptions in named entity recogni-tion. In Proc. of CoNLL.
Dan Roth and Wen tau Yih. 2004. A linear program-ming formulation for global inference in natural lan-guage tasks. In Proc. of CoNLL.
Erik F. Sang and Jorn Veenstra. 1999. Representingtext chunks. In Proc. of EACL.
Richard Socher, Brody Huval, Christopher D. Man-ning, and Andrew Y. Ng. 2012. Semantic composi-tionality through recursive matrix-vector spaces. InProc. of EMNLP.
Rupesh Kumar Srivastava, Klaus Greff, and JurgenSchmidhuber. 2015. Highway networks. arXivpreprint.
Ang Sun, Ralph Grishman, and Satoshi Sekine. 2011.Semi-supervised relation extraction with large-scaleword clustering. In Proc. of NAACL-HLT.
Changzhi Sun, Yuanbin Wu, Man Lan, Shiliang Sun,Wenting Wang, Kuang-Chih Lee, and Kewen Wu.2018. Extracting entities and relations with jointminimum risk training. In Proc. of EMNLP.
Kai Sheng Tai, Richard Socher, and Christopher D.Manning. 2015. Improved semantic representationsfrom tree-structured long short-term memory net-works. In Proc. of ACL.
Tung Tran and Ramakanth Kavuluru. 2019. Neuralmetric learning for fast end-to-end relation extrac-tion. arXiv preprint.
Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N. Gomez, ŁukaszKaiser, and Illia Polosukhin. 2017. Attention is allyou need. In Proc. of NIPS.
Patrick Verga, Emma Strubell, and Andrew McCallum.2018. Simultaneously self-attending to all mentionsfor full-abstract biological relation extraction. InProc. of NAACL-HLT.
Jesse Vig. 2019. A multiscale visualization of attentionin the transformer model. In Proc. of ACL.
David Wadden, Ulme Wennberg, Yi Luan, and Han-naneh Hajishirzi. 2019. Entity, relation, and eventextraction with contextualized span representations.In Proc. of EMNLP/IJCNLP.
Christopher Walker, Stephanie Strassel, Julie Medero,and Kazuaki Maeda. 2006. Ace 2005 multilingualtraining corpus. Linguistic Data Consortium.
Haoyu Wang, Ming Tan, Mo Yu, Shiyu Chang, DakuoWang, Kun Xu, Xiaoxiao Guo, and Saloni Potdar.2019. Extracting multiple-relations in one-pass withpre-trained transformers. In Proc. of ACL.
Ralph Weischedel, Eduard Hovy, Mitchell Mar-cus, Martha Palmer, Robert Belvin, Sameer Prad-han, Lance Ramshaw, and Nianwen Xue. 2011.Ontonotes: A large training corpus for enhanced pro-cessing. Handbook of Natural Language Processingand Machine Translation.
1717
Thomas Wolf, Lysandre Debut, Victor Sanh, JulienChaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtow-icz, and Jamie Brew. 2019. Huggingface’s trans-formers: State-of-the-art natural language process-ing. arXiv preprint.
Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin,Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou,and Maosong Sun. 2019. Docred: A large-scaledocument-level relation extraction dataset. In Proc.of ACL.
Dmitry Zelenko, Chinatsu Aone, and AnthonyRichardella. 2002. Kernel methods for relation ex-traction. In Proc. of EMNLP.
Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou,and Jun Zhao. 2014. Relation classification via con-volutional deep neural network. In Proc. of COL-ING.
Dongxu Zhang and Dong Wang. 2015. Relationclassification via recurrent neural network. arXivpreprint.
Meishan Zhang, Yue Zhang, and Guohong Fu. 2017.End-to-end neural relation extraction with global op-timization. In Proc. of EMNLP.
Shubin Zhao and Ralph Grishman. 2005. Extractingrelations with integrated information using kernelmethods. In Proc. of ACL.
A MD-RNN
In this section we present the detailed implementa-tion of MD-RNN with GRU.
Formally, at the time-step layer l, row i, andcolumn j, with the input Xl,i,j , the cell at layer l,row i and column j calculates the gates as follows:
T prevl,i,j = [Tl−1,i,j ;Tl,i−1,j ;Tl,i,j−1],∈ R3H
(18)
rl,i,j = σ([Xl,i,j ;Tprevl,i,j ]W r + br)),∈ RH
(19)
zl,i,j = σ([Xl,i,j ;Tprevl,i,j ]W z + bz)),∈ RH
(20)
λl,i,j,m = [Xl,i,j ;Tprevl,i,j ]W λ
m + bλm,∈ RH (21)
λl,i,j,0,λl,i,j,1, λl,i,j,2 =
softmax(λl,i,j,0, λl,i,j,1, λl,i,j,2) (22)
Figure 8: For 2D-RNNs, cells in the same color can becomputed in parallel.
where W and b are trainable parameters and pleasenote that they share parameters in different rowsand columns but not necessarily in different layers.Besides, � is the element-wise product, and σ isthe sigmoid function.
As in GRU, r is the reset gate controllingwhether to forget previous hidden states, and zis the update gate, selecting whether the hiddenstates are to be updated with new hidden states. Inaddition, we employ a lambda gate λ, which isused to weight the predecessor cells before passingthem through the update gate.
There are two slightly different ways to computethe candidate activation Tl,i,j , namely
Tl,i,j = tanh(Xl,i,jWx
+ rl,i,j � (T prevl,i,j Wp) + bhl ) (26)
and
Tl,i,j = tanh(W xl Xl,i,j
+ (rl,i,j � T prevl,i,j )W p + bhl ) (27)
And we found in our preliminary experiments thatboth of them performed as well as each other, andwe choose the former, which saves some computa-tion.
The time complexity of the naive implementa-tion (i.e., two for-loops in each layer) isO(L×N×
N) for a sentence with length N and the numberof encoding layer L. However, antidiagonal entriescan be calculated at the same time because theirvalues do not depend on each other, shown in thesame color in Figure 8. Therefore, we can optimizeit through parallelization and reduce the effectivetime complexity to O(L×N).
B Data
Table 6 shows the dataset statistics after pre-processing. We keep the same pre-processing andevaluation standards used by most previous works.
The ACE04 and ACE05 corpora are collectedfrom a variety of domains, such as newswire andonline forums. We use the same entity and relationtypes, data splits, and pre-processing as Li and Ji(2014) and Miwa and Bansal (2016)7. Specifically,they use head spans for entities but not use the fullmention boundary.
The CoNLL04 dataset provides entity and re-lation labels. We use the same train-test split asGupta et al. (2016)8, and we use the same 20% trainset as development set as Eberts and Ulges (2019)9.Both micro and macro average F1 are used in previ-ous work, so we will specify this while comparingwith other systems.
The ADE dataset is constructed from medicalreports that describe the adverse effects arisingfrom drug use. It contains a single relation type“Adverse-Effect” and the two entity types “Adverse-Effect” and “Drug”. Similar to previous work, wefilter out instances containing overlapping entities,only accounting for 2.8% of total.
Following prior work, we perform 5-fold cross-validation for ACE04 and 10-fold for ADE. Be-sides, we use 15% of the training set as the devel-opment set. We report the average score of 5 runs
7We use the prepocess script provided by Luanet al. (2019): https://github.com/luanyi/DyGIE/tree/master/preprocessing
batch size 24optimizer Adamlearning rate (lr) 1e-3warm-up steps 1000dropout rate 0.5# layers (L) 3# attention heads (A) 8hidden dim (H) 200token emb dim 100char emb dim 30gradient clipping 5.0
Table 7: Hyperparameters used in our experiments.
for every dataset. For each run, we use the modelthat achieves the best performance (averaged entitymetric score and relation metric score) on the de-velopment set, and evaluate and report its score onthe test set.
C Hyperparameters and Pre-trainedLanguage Models
The detailed hyperparameters are present in Table 7.For the word embeddings, we use 100-dimensionalGloVe word embeddings trained on 6B tokens10
as initialization. We disable updating the wordembeddings during training. We set the hidden sizeto 200, and since we use bidirectional MD-RNNs,the hidden size for each MD-RNN is 100. We useinverse time learning rate decay: lr = lr/(1 +decay rate × steps/decay steps), with decay rate0.05 and decay steps 1000.
Besides, the tested pre-trained language modelsare shown as follows:
• [ELMo] (Peters et al., 2018): Character-based pre-trained language model. We usethe large checkpoint, with embeddings ofdimension 3072.
• [BERT] (Devlin et al., 2019): Pre-trained Transformer. We use thebert-large-uncased checkpoint,with embeddings of dimension 1024 andattention weight feature of dimension 384 (24layers × 16 heads).
• [RoBERTa] (Liu et al., 2019): Pre-trainedTransformer. We use the roberta-largecheckpoint, with embeddings of dimension
Table 8: Comparisons with different methods to learnthe table representation. For MD-RNN, D+, D− andD are indicators representing the direction, in whichthe hidden state flows forward, backward, or unable toflow at dimension D (D could be layer, row, or col).When using multiple MD-RNNs, we separate the indi-cators by “;”.
1024 and attention weight feature of dimen-sion 384 (24 layers × 16 heads).
• [ALBERT] (Lan et al., 2019): A lite versionof BERT with shared layer parameters. Weuse the albert-xxlarge-v1 checkpoint,with embeddings of dimension 4096 and at-tention weight feature of dimension 768 (12layers × 64 heads). We by default use thispre-trained model.
We use the implementation provided by Wolfet al. (2019)11 and Akbik et al. (2019)12 to generatecontextualized embeddings and attention weights.Specifically, we generate the contextualized wordembedding by averaging all sub-word embeddingsin the last four layers; we generate the attentionweight feature (if available) by summing all sub-word attention weights for each word, which arethen concatenated for all layers and all heads. Bothof them are fixed without fine-tuning.
D Ways to Leverage the Table Context
Table 8 presents the comparisons of different waysto learn the table representation.
11https://github.com/huggingface/Transformers
12https://github.com/flairNLP/flair
Importance of context Setting “layer+row col”does not exploit the table context when learning thetable representation, instead, only layer-wise opera-tions are used. As a result, it performs much worsethan the ones exploiting the context, confirming theimportance to leverage the context information.
Context along row and column Neighbors alongboth the row and column dimensions are impor-tant. setting “layer+row+col ; layer+row−col”and “layer+row col+; layer+row col−” removethe row and column dimensions respectively, andtheir performance is though better than “layer+rowcol”, but worse than setting “layer+row+col+;layer+row−col−”.
Multiple dimensions Since in setting“layer+row+col+”, the cell at row i and columnj only knows the information before the i-th andj-th word, causing worse performance than bidirec-tional (“layer+row+col+; layer+row−col−” and“layer+row+col−; layer+row−col+”) and quad-directional (“layer+row+col+; layer+row−col−;layer+row+col−; layer+row−col+”) settings.Besides, the quaddirectional model does not showsuperior performance than bidirectional ones, sowe use the latter by default.
Layer dimension Different from the row and col-umn dimensions, the layer dimension does notcarry more sentential context information. Instead,it carries the information from previous layers, sothe model can reason high-level relations based onlow-level dependencies captured by predecessorlayers, which may help recognize syntactically andsemantically complex relations. Moreover, recur-ring along the layer dimension can also be viewedas a layer-wise short-cut, serving similarly to highway (Srivastava et al., 2015) and residual connec-tion (He et al., 2016) and making it possible for thenetworks to be very deep. By removing it (resultsunder “layer row+col+; layer row−col−”), the per-formance is harmed.
Other network Our model architecture can beadapted to other table encoders. We try CNN toencode the table representation. For each layer l,given inputs X l, we have:
Table 9: Comparisons of different table filling formula-tions. When not filling the entire table, L only fills thelower-triangular part, and U fills the upper-triangularpart.
ever, despite its advantages in training time, itsperformance is worse than the MD-RNN basedones.
E Table Filling Formulations
Our table filling formulation does not exactly fol-low Miwa and Sasaki (2014). Specifically, we fillthe entire table instead of only the lower (or higger)triangular part, and we assign relation tags to cellswhere entity spans intersect instead of where lastwords intersect. To maintain the ratio of positiveinstances to negative instances, although the entiretable can express directed relations by undirectedtags, we still keep the directed relation tags. I.e, ifyREi,j = −→r then yRE
j,i =←−r , and vice versa. Table 9ablates our formulation (last row), and comparesit with the original one (Miwa and Sasaki, 2014)(first row).
F Probing Intermediate States
Figure 9 presents examples picked from the de-velopment set of ACE05. The prediction layer (alinear layer) after training is used as a probe to dis-play the intermediate state of the model, so we caninterpret how the model improves both representa-tions from stacking multiple layers and thus fromthe bidirectional interaction.
Such probing is valid since for the table encoder,the encoding spaces of different cells are consistentas they are connected through gate mechanism,including cells in different encoding layers; for thesequence encoder, we used residual connection sothe encoding spaces of the inputs and outputs areconsistent. Therefore, they are all compatible withthe prediction layer. Empirically, the intermediatelayers did give valid predictions, although they arenot directly trained for prediction.
In Figure 9a, the model made a wrong prediction
PHYS
Labels
PredictionLayer 1
PredictionLayer 2
PredictionLayer 3
He
PER
wants to call his mom in Houston
PER PER
PER-SOC
PHYS
He wants to call his mom in Houston
PER PER
PER-SOC
PHYS
He wants to call his mom in Houston
PER PER
PER-SOC
PART-WHOLE
He wants to call his mom in Houston
PER PER
(a) Correct the prediction at the 2nd layer
Labels
PredictionLayer 1
PredictionLayer 2
PredictionLayer 3
An Abu Dhabi TV office in Baghdad also was hit
ORG FAC GPE
PART-WHOLEART
An Abu Dhabi office in Baghdad also was hit
ORG FAC GPE
PHYSORG-AFF
TV
FAC ORG
An Abu Dhabi TV office in Baghdad also was hit
ORG FAC GPE
PART-WHOLE
An Abu Dhabi TV office in Baghdad also was hit
ORG FAC GPE
PART-WHOLEART
(b) Correct the prediction at the 3rd layer
Labels
PredictionLayer 1
PredictionLayer 2
PredictionLayer 3
... more than 200 marines and sailors from the war in iraq
PER PER GPE
PHYS
PHYS
... more than 200 marines and sailors from the war in iraq
PER PER GPE
... more than 200 marines and sailors from the war in iraq
PER PER GPE
... more than 200 marines and sailors from the war in iraq
PER PER GPE
PHYS
PHYS
PHYS
(c) A mistake at the last layer
Figure 9: Comparisons of predictions by different en-coding layers. We predict relations and entities withthe intermediate sequence and table representation, sothat we can figure out how the model improves bothrepresentations by stacking multiple encoding layers.
with the representation learned by the first encod-ing layer. But after the second encoding layer, thismistake has been corrected by the model. This isalso the case that happens most frequently, indi-cating that two encoding layers are already goodenough for most situations. For some more compli-
1721
cated cases, the model needs three encoding layersto determine the final decision, shown in Figure 9b.Nevertheless, more layers do not always push theprediction towards the correct direction, and Fig-ure 9c shows a negative example, where the modelmade a correct prediction in the second encoding
layer, but in the end it decided not to output onerelation, resulting in a false-negative error. Butwe must note that such errors rarely occur, andthe more common errors are that entities or rela-tionships are not properly captured at all encodinglayers.
