Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 139–148July 5 - 10, 2020. c©2020 Association for Computational Linguistics

139

TransS-Driven Joint Learning Architecture for Implicit DiscourseRelation Recognition

Ruifang He1,2,3, Jian Wang1,2, Fengyu Guo1,2∗, and Yugui Han1,2

1College of Intelligence and Computing, Tianjin University, Tianjin, China2Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin, China

3State Key Laboratory of Cognitive Intelligence, iFLYTEK, China{rfhe,jian wang,fengyuguo,yghan}@tju.edu.cn

Abstract

Implicit discourse relation recognition is achallenging task due to the lack of connectivesas strong linguistic clues. Previous methodsprimarily encode two arguments separately orextract the specific interaction patterns for thetask, which have not fully exploited the anno-tated relation signal. Therefore, we propose anovel TransS-driven joint learning architectureto address the issues. Specifically, based on themulti-level encoder, we 1) translate discourserelations in low-dimensional embedding space(called TransS), which could mine the latentgeometric structure information of argument-relation instances; 2) further exploit the seman-tic features of arguments to assist discourse un-derstanding; 3) jointly learn 1) and 2) to mutu-ally reinforce each other to obtain the betterargument representations, so as to improve theperformance of the task. Extensive experimen-tal results on the Penn Discourse TreeBank(PDTB) show that our model achieves compet-itive results against several state-of-the-art sys-tems.

1 Introduction

Discourse relation describes how two adjacent textunits (e.g., clauses, sentences, and larger sentencegroups) are connected logically to one another.A discourse relation instance is usually definedas a connective taking two arguments (as Arg1and Arg2, respectively). Implicit discourse rela-tion recognition without explicit connectives (Pitleret al., 2009) is still a challenging problem of dis-course analysis, which needs to infer the discourserelation from a specific context. It is beneficialto many downstream natural language processing(NLP) applications, such as machine translation(Meyer and Popescu-Belis, 2012) and text summa-rization (Gerani et al., 2014).

∗Corresponding author.

The existing neural network-based models haveshown great success in recognizing implicit dis-course relations. It mainly includes 1) Basic neuralnetworks (Braud and Denis, 2015; Zhang et al.,2015; Liu et al., 2016) can learn the dense vectorrepresentations of discourse arguments, which cancapture the semantic information to some extent.Further studies exploit different attention or mem-ory mechanisms (Liu and Li, 2016; Zhang et al.,2016) to capture the critical information of argu-ment pairs. 2) Complex neural models (Chen et al.,2016; Lei et al., 2017; Guo et al., 2018) utilizegated relevance networks or neural tensor networksto capture the deeper interactions between two dis-course arguments. 3) Joint learning architectures(Qin et al., 2017; Bai and Zhao, 2018; Xu et al.,2019) exploit implicit connective cues, differentgranularity of text, or topic-level relevant informa-tion to improve the discourse relation prediction.However, these approaches still have the followingdrawbacks: 1) do not make full use of the annotateddiscourse relation signal to explore the argument-relation features; 2) neglect the extra information inthe low-dimensional continuous embedding space,i.e., the direction or structure information of thevectors.

Notice that Translating Embeddings (TransE) isa method for the prediction of entities’ missingrelations in knowledge graphs. Bordes et al. (2013)model relations by interpreting them as translatingoperation not on the graph structure directly butin a learned low-dimensional embedding of theknowledge graph entities: if (he, le, te) holds, thenthe embedding of the tail entity te should be closeto the embedding of the head entity he plus somevector that depends on the relation le. Similar to theentity relation extraction, our task aims to identifythe semantic relations between two arguments (i.e.,sentences).

Inspired by TransE, we design a new method

140

(TransS), which translates discourse relations insentence embedding spaces to mine the argument-relation features. Intuitively, these features reflectthe latent geometric structure among the argumentsand their discourse relation by performing the al-gebraic operation, and the argument-relation in-stances with the same discourse relation may havesimilar direction and position information in theembedding space. Therefore, we propose a novelTransS-driven joint learning neural network frame-work that leverages the latent geometric structureinformation of argument-relation instances, in ad-dition to using the semantic features to improvethe comprehension of discourse argument. Amongthem, we adopt a multi-level encoder to furtherenrich the argument representations, which couldobtain the deeper semantics of discourse.

In summary, the main contributions of this paperare as follows:

• Propose a novel TransS-driven joint learningarchitecture, including the latent geometricstructure information learning (GSL) and se-mantic feature learning (SFL);

• Design TransS approach to translate dis-course relations in low-dimensional embed-ding space from the sentence-level perspec-tive, which could induce the geometric struc-ture of argument-relation instances to someextent;

• Employ the mutual reinforcing between theGSL and SFL to optimize the argument rep-resentations: 1) the GSL adopts its geometricstructure clues to facilitate the SFL; 2) theSFL utilizes its semantic cues to improve thelearning capability of GSL;

• The experimental results on the PDTB demon-strate the effectiveness of our model.

2 The Proposed Model

The implicit discourse relation recognition task isusually formalized as a classification problem. Inthis section, we give an overview of the TransS-driven joint learning framework, which consists offour parts: embedding layer, multi-level encoder,latent geometric structure learning, and semanticfeature learning, as shown in Figure 1.

2.1 Embedding LayerIn order to model two discourse arguments withneural networks, we transform the one-hot repre-

sentations of arguments and their discourse relationinto the distributed representations. Formally, theembedding layer could be seen as a simple projec-tion layer where the word embedding is achievedby lookup table operation according to the indexes.All words of two arguments Arg1, Arg2, and theirrelation will be mapped into low dimensional vec-tor representations, which are taken as the input ofour model.

2.2 Multi-level Encoder

To enrich the discourse argument representations,we exploit multi-level encoder shown in Figure 2 tolearn the argument representations at the differentlevels. Particularly, the higher-level states of multi-level encoder could capture context-dependent as-pects of words while the lower-level states couldmodel aspects of syntax (Peters et al., 2018). Themulti-level encoder is composed of stacked encoderlayers.

2.2.1 Encoder LayerReferring to the previous work, we implement thebidirectional LSTM (BiLSTM) neural network tomodel the argument sequences, which could pre-serve both the historical and future information inforward and reverse directions. Therefore, we canobtain two representations

−→ht and

←−ht at each time

step t of the sequence. Then we concatenate themto get the intermediate state ht = [

−→ht ,←−ht ].

Attention Controller. Due to the limitations oftreating each word equally in the general represen-tations, we use attention mechanism to point out thewords particularly useful for our task. Let H be thematrix consisting of output vectors [h1, h2, ..., hn]of the last layer produced, where n is the length ofthe argument. The new representation h of the ar-gument is formed by a weighted sum of the outputvectors:

M = tanh(H), (1)

α = softmax(wTM), (2)

h = HαT . (3)

where H ∈ Rn×d, d is the dimension of wordembedding,w is a parameter vector. Then we couldobtain the argument representation with importantinformation from Eq. (4) for the next step.

h∗ = tanh(h) (4)

141

Geometric Structure Learning (GSL)

Semantic information

Latent geometric structure information

Arg1 Arg2Relation

Update Update

Discourse Relation

rs

Relation

Multi-level Encoder

Joint Loss

Embedding LayerSemantic embedding

space

re

te

The idea of TransE

he

he + re » te

hs + rs » tsrs(Relation) hs(h

*Arg1)

ts(h*Arg2)

GSL Loss

Translating operation

+

h*Arg1

(hs)Relation

Gro

un

d T

ruth

...

...

h*Arg2

(ts)

Softmax

SFL Loss

GSL Module SFL Module

Semantic Feature Learning (SFL)

Update Update

TransS

Arg1 Arg2

(Arg1, Arg2) (Arg1, Arg2)

Figure 1: TransS-driven joint learning architecture of our proposed model.

Multi-level Encoder Module( Taking Arg1 as an example )

Encoder Layer(1)

Encoder Layer(2)

Pooling layer

...

BiLSTM

...

a1a2 a3

an

a1 ana2 a3

Encoder Layer

... ...

... ... ...

Arg1

Figure 2: The illustration of multi-level encoder.

2.2.2 Pooling LayerFinally, we can receive the overall argument repre-sentations by averaging pooling operation for theword embedding sequence, defined as:

h∗Arg =1

n

n∑i=1

h∗(m)i (5)

where h∗Arg is the argument representation, h∗(m)i

is the representation of the i-th word in the wordembedding sequence of the m-th encoder layer, nis the number of words in an argument.

2.3 Latent Geometric Structure Learning

TransE, as a model for learning low-dimensionalembeddings of entities, is to enforce the structureof embedding space in which different relations be-tween entities of different types may be representedby translation (Bordes et al., 2013). Discourse rela-tion recognition and entity relation extraction are

similar to some extent. Intuitively, the argument-relation instances with the same discourse relationmay also have similar direction and position infor-mation in embedding space. However, discourseargument embedding is a sentence-level represen-tation, which is different from the reuse of entitiesin other sentences, and more diverse and complexthan entity representation. Therefore, we designTransS, a method which models discourse rela-tions by interpreting them as translations operat-ing in the low-dimensional embedding space fromthe sentence perspective. Moreover, it could minethe latent geometric structure of argument-relationinstances. Specifically, to define two argumentsas head vector hs and tail vector ts respectively,their annotated relation signal as relation vectorrs, the latent geometric structure is reflected byhs + rs ≈ ts, their score function is defined asfollows:

ds(hs, ts) = ||hs + rs − ts||22. (6)

where hs, ts denote the representations of Arg1 andArg2 respectively; rs ∈ Rd is the embedding ofdiscourse relation and d is the dimension of wordembedding.

GSL Loss. Under the framework of TransS, givena training set T of triplets (hs, rs, ts) composed oftwo arguments hs, ts ∈ V (the set of sentence vec-tors) and a relation rs ∈ R (the set of relation), ourmodel would learn the embeddings of the wordsin arguments and the discourse relation. The GSL

142

loss function is defined as:

LGSL =∑

(hs,rs,ts)∈T

∑(h′s,rs,t

′s)∈T ′(hs,rs,ts)

[γ + ds(hs

+ rs, ts)− ds(h′s + rs, t′s)]+ + λGSL‖θ‖22.

(7)

where [·]+ denotes the positive instances, γ > 0is a margin hyper-parameter, and the set of neg-ative triplets, constructed according to Eq.(8), inwhich the head or tail is replaced by a random argu-ment vector (but not simultaneously). θ denotes theother parameters of the network. L2 regularizationis used to penalize the size of all parameters forpreventing overfitting, weighted by λGSL.

T ′(hs,rs,ts)={(h′s, rs, ts)|h′s ∈ V }∪

{(hs, rs, t′s)|t′s ∈ V )}.(8)

By optimizing the GSL loss, we could ob-tain the latent geometric structure informationabout argument-relation instances. Different fromTransE, we could not directly utilize TransS to rec-ognize discourse relations, for that each argumentcould not be reused in discourse. Therefore, we ex-ploit TransS to mine the latent geometric structureinformation and further guide the semantic featurelearning.

2.4 Semantic Feature LearningThe new argument representations (h∗Arg1, h

∗Arg2)

with latent geometric structure information learnedby the GSL are as inputs of the semantic featurelearning (SFL). The h∗Arg1(i.e., hs) and h∗Arg2(i.e.,ts) are obtained from the multi-level encoder. Wefurther stack a softmax layer upon the representa-tions:

y = f(Wf

[h∗Arg1,

h∗Arg2

]+ bf ). (9)

where f is the softmax function, Wf ∈ RC×2d,bf ∈ RC are the weights and bias term respectively,d denotes the dimension of word embedding andC denotes the number of relation classes.SFL Loss. Under the framework of basic neuralnetworks for our task, given training set T , twoargument vectors hs, ts in the triplet (hs, rs, ts) areconcatenated to a new sentence vector during thetraining process, and then the generated vector isused for relation recognition. The SFL loss is across-entropy style shown as:

LSFL = −C∑

j=1

yjlog(yj) (10)

where y is the one-hot representation of the ground-truth relation; y is the predicted probabilities ofrelations; C is the number of relation class.

2.5 Joint LearningAfter obtaining the new representations Arg1 ashead vector hs, Arg2 as tail vector ts, and the re-lation vector rs, our model is trained using jointlearning mechanism. The goal of our model is tominimize the loss function (Eq.(11))

L = LGSL + λLSFL. (11)

where, LGSL and LSFL are from Eq.(7) and (10),respectively; λ is the trade-off parameter control-ling the balance between GSL and SFL.

Our model jointly learns the GSL and SFL tooptimize the argument representations. On the onehand, the GSL maps the discourse relation betweentwo arguments to the low-dimensional embeddingspace and obtains the vectors hs, rs, ts with geo-metric structure information to constrain the SFL.On the other hand, the SFL alternately optimizesthe discourse representations and provides the nec-essary semantic clues for geometric structure in-formation mining. Generally, the GSL and SFLreinforce with each other, and finally get the betterargument representations containing the semanticsand the latent geometric structure information ofargument-relation.

3 Experiments

3.1 DatasetsThe PDTB 2.0, a large scale corpus annotated on2,312 Wall Street Journal articles, is utilized for allexperiments. It contains three hierarchies: Level-1Class, Level-2 Type, and Level-3 Subtype. We fo-cus on the first level, which contains four classes:Comparison (Comp.), Contingency (Cont.), Expan-sion (Exp.), and Temporal (Temp.). As (Rutherfordand Xue, 2014), we use Sections 2-21 as the train-ing set, Section 22 as the development set, Section23 as the test set.

Relation Train Dev TestComp. 1945 196 152Cont. 3242 284 272Exp. 6794 646 546Temp. 709 61 79Total 12690 1187 1049

Table 1: The statistical distribution of PDTB.

143

3.2 Experimental Settings

All the arguments are padded at the same lengthof 100. Word embedding is randomly initializedby uniformly distributed samples [-0.1, 0.1] with300-dimension. The learning rate is set to 0.001,the batch size is 128, and the number of iterationis 100. For the GSL, the margin of loss is set to0.5, the trade-off parameter λ in Eq.(11) is set to1.0, and we use L2 distance as dissimilarity; Forthe SFL, the sizes of the input and the hidden layerof the BiLSTMs are both 300; we choose threeencoder layers, and set the dimension of pre-trainedembeddings from ELMo (Peters et al., 2018) to300.

3.3 The Comparison Models

3.3.1 The State-of-the-art Systems

To validate the effectiveness of our model, we se-lect some state-of-the-art systems from the follow-ing three aspects to compare with our model:• Discourse Argument Representation1) Ji2015: Ji and Eisenstein (2015) computed dis-tributed representations for each discourse argu-ment by composition up the syntactic parse tree.2) Zhang2015: Zhang et al. (2015) proposed pureneural networks with three different pooling opera-tions to learn shallow representations in tasks.3) Liu2016a: Liu and Li (2016) combined atten-tion mechanism and external memory to focus onspecific words that helps determine discourse rela-tions.4) Lan2017: Lan et al. (2017) designed anattention-based neural network for learning dis-course argument representations and a multi-taskframework for learning knowledge from annotatedand unannotated corpora.• Complex Neural Models5) Chen2016: Chen et al. (2016) adopted a gatedrelevance network to capture interaction informa-tion between two arguments to enhance relationrecognition.6) Qin2016: Qin et al. (2016a) adopted context-aware character-enhanced embeddings to addressimplicit discourse relation recognition task.7) Lei2017: Lei et al. (2017) devised the SimpleWord Interaction Model (SWIM) to learn the inter-actions between word pairs.8) Dai2018: Dai and Huang (2018) modeled inter-dependencies between discourse units as well asdiscourse relation continuity and patterns, and pre-

dict a sequence of discourse relations in a para-graph.• Joint Learning9) Liu2016b: Liu et al. (2016) designed related dis-course classification tasks specific to a corpus, andproposed a novel Convolutional Neural Networkembedded multi-task learning system to synthe-size these tasks by learning both unique and sharedrepresentations for each task.10) Bai2018: Bai and Zhao (2018) employed dif-ferent grained text representations, including char-acter, subword, word, sentence, and sentence pairlevels, and transfered the knowledge from the im-plicit connectives to support discourse relation pre-diction.

3.3.2 The Ablation Methods

In order to validate the effectiveness of each com-ponent of our model, we present the following ab-lation methods:

• Baseline (Including SFL) We use three encoderlayers to encode the argument pairs separately,then concatenate them together, and feed themto the SFL module for relation recognition.

• +GSL We encode two arguments based on theBaseline, and then feed them into GSL and SFLmodules, respectively. Finally, we use the twomodules to help recognize the discourse relation.

• +ELMo We utilize the Baseline to receive theargument representations, and then we use thepre-trained ELMo vector to enhance the argu-ment representations. Finally, we feed them tothe SFL module for relation recognition.

• +GSL & ELMo (Ours) We feed the two argu-ment representations, encoded by the Baselineand enhanced by the pre-trained ELMo vector,into GSL and SFL modules, respectively. Andthen, we utilize the integrated representation torecognize the discourse relation.

3.4 Results and Discussion

Consistent with previous studies, we choose F1

score and accuracy as evaluation metrics. For bi-nary classification, the result is computed by F1

score, and for 4-way classification, the result iscomputed by macro average F1 score.

144

Model Comp. Cont. Exp. Temp. 4-way Acc.Ji2015 35.93 52.78 - 27.63 - -Zhang2015 33.22 52.04 69.59 30.54 - -Liu2016a 32.13 46.09 69.88 31.82 44.98 57.27Lan2017 40.73 58.96 72.47 38.50 47.80 57.39Chen2016 40.17 54.76 - 31.32 - -Qin2016 38.67 54.91 80.66 32.76 - -Lei2017 40.47 55.36 69.50 35.34 46.46 -Dai2018 37.72 49.39 67.45 40.70 48.82 59.75Liu2016b 39.86 54.48 70.43 38.84 46.29 57.57Bai2018 47.85 54.47 70.60 36.87 51.06 -Ours 47.98 55.62 69.37 38.94 51.24 59.94

Table 2: F1 score (%) and Accuracy(Acc., %) of different comparison models on binary and 4-way classification.

Model Comp. Cont. Exp. Temp. 4-way Acc.Baseline 32.32 49.53 65.91 34.86 46.46 54.02+ GSL 44.88 53.17 67.91 37.38 48.91 57.65+ ELMo 46.85 54.57 68.44 38.71 50.07 58.89+ GSL & ELMo (Ours) 47.98 55.62 69.37 38.94 51.24 59.94

Table 3: F1 score (%) and Accuracy(Acc., %) of ablation models on binary and 4-way classification.

3.4.1 Comparison with the state-of-the-artSystems

Table 2 shows the results of the compared state-of-the-art systems on binary and 4-way classification.We could make the following observations:

• Overall, i) our model achieves state-of-the-artperformance, i.e., the F1 score and accuracyare 51.24% and 59.94% on the 4-way classi-fication, respectively; ii) the results of binaryclassification are keeping a similar tendencywith the 4-way classification. In particular,our model gains the best F1 score on Compar-ison relation. The main reasons may be thatthe instances with different discourse relationshave different directions and position (geomet-ric structure) features in the low-dimensionalcontinuous embedding space, and the Compar-ison instances have more obvious indicativestructure features.

• Comparing our model with Chen2016 andLei2017, the F1 scores of our model arehigher than those of the latter two. It provesthat our model is better than the two meth-ods only considering the content interactions,since we jointly leverage the geometric struc-ture information and the semantic information

of the argument-relation instances to obtaindeeper interactions.

• In the comparison models, Bai2018 with jointlearning framework achieves the best perfor-mance, which illustrates that jointly utilizingthe discourse relation and the implicit connec-tives are helpful to the task. Moreover, theperformance of our model is better than thatof Bai2018. It not only indicates that the ef-fectiveness of joint learning, but also provesconsidering the geometric structure is benefi-cial to our task.

3.4.2 Ablation ModelsFor the ablation models, we can make the observa-tions from Table 3:

Overall:1) Our model gains state-of-the-art per-formance than that of the other ablation models.This demonstrates that the geometric structure in-formation could enrich the argument representationand promote implicit discourse relation recognition.2) All models have a higher F1 values on the Ex-pansion relation than those of the other relations.The unbalanced data may cause that.

GSL: The F1 score of our model using the GSLmodule is 48.91%, higher than the performanceof Baseline. In addition, compared with ELMo,

145

(a) without geometric structure features. (b) with geometric structure features.

Figure 3: Visualization of the interaction information of argument representation.

although the performance of GSL does not exceedELMo’s, GSL obtain comparable results. This man-ifests that the two modules (GSL and SFL) couldreinforce with each other, which utilizes the geo-metric structure information by the algebraic opera-tion. Moreover, we exploit the geometric structureclues to augment the semantic understanding of dis-course from a new aspect, which is different fromthe ELMo only focusing on the semantic informa-tion of the text itself.

ELMo: The third row of Table 3 is the result ofour model, which only uses the pre-trained ELMovector to enhance argument representations. TheF1 score and accuracy are 50.07% and 58.89%,respectively, which achieve 3.61% and 4.87% im-provements than those of the Baseline. It verifiesthat ELMo, as pre-trained contextualized word em-beddings, could contain more contextual informa-tion.

GSL & ELMo: Compared with ELMo, GSL& ELMo gains better performance, which demon-strates that inducing spatial geometry structure in-formation based on argument enhancement couldunderstand the semantics of discourse better.

3.4.3 Impact of TransSTo illustrate the effectiveness of the latent geomet-ric structure information of argument-relation in-stances gotten by TransS, we visualize the heatmaps of the interaction information of argumentrepresentations shown in Figure3. Every wordcomes with various background colors. The darkerpatches denote the correlations of word pairs arehigher. The example of Comparison relation islisted below:

Arg1: I was prepared to be in a very bad moodtonight.

Arg2: Now, I feel maybe there’s a little bit of eu-phoria.

From the semantics of perspective, this examplecould be identified as Comparison or Temporal re-lation. Since argument pairs may have distinct dis-tinguishing features in geometric space, we couldconsider the geometric structure of argument pairsto help identify the discourse relation. We can ob-tain the following observations:

• Seen from Figure3(a), without introducinggeometric structure information, the modelhas a high correlation around the word “Now”which might indicate the Temporal relationdirectly. This demonstrates that only consid-ering the semantic information of argumentsmay suffer from issues such as polysemy, am-biguity, as well as fuzziness.

• Figure3(b) shows the result of the interac-tion information of argument representations,which introduces the GSL. From the results,we can see that the model has a high correla-tion around the word “little” and “very” withthe comparative information. The possiblereason is that our model utilizing GSL shiftsthe higher attention from the word “Now”with Temporal information to the word pairs(little, very), (euphoria, bad) and (euphoria,mood) with Comparison relation. Our modelwith GSL introduces the geometric structureinformation and jointly utilizes these featuresand semantic information to help identify thediscourse relation.

3.4.4 Impact of Encoder Layer NumberIn order to illustrate the impact of the encoder layernumber, we select different sizes of encoder layer

146

Figure 4: The effect of encoder layers’ number.

as comparison experiments on the 4-way classi-fication. Figure 4 shows that the F1 scores areincreasing until three encoder layers. And whenthe size of the encoder layer is four or five, theperformance of our model is decreasing obviously.

With the increasing of the number of encoder lay-ers, the model could capture the richer semantic in-formation. However, the results imply that with themore encoder layers considered, the model couldincur the over-fitting problem due to adding moreparameters. Therefore, we adopt three encoderlayers to encode the arguments as our Baseline insection 3.3.

4 Related Work

Neural network-based models have shown greateffectiveness in implicit discourse relation recog-nition. We give the analysis of mainly relevantwork:

4.1 Discourse Argument Representation

Proper argument representation is a core factorof our task. Most previous researches encode ar-guments as dense and continuous representationbased on various neural networks, from basic neu-ral networks (such as CNN, RNN) to complex neu-ral networks (Zhang et al., 2015; Qin et al., 2016b;Rutherford et al., 2016). Some studies adopt dif-ferent attention or memory mechanisms to catchthe emphasis on discourse arguments (Mnih et al.,2014; Liu and Li, 2016; Zhang et al., 2016). Liet al. (2016) exploit the hierarchical attention tocapture the focus of different granularities. Zhanget al. (2016) build upon a semantic memory to storeknowledge in the distributed fashion for the task.However, these models have only considered thetwo arguments independently without the interac-tion information.

4.2 Argument Pair Interactions

Further studies tend to discover more semantic in-teractions between two arguments by complex neu-ral networks (Qin et al., 2016c; Cai and Zhao, 2017;Lan et al., 2017; Guo et al., 2018). Chen et al.(2016) develop a novel gated relevance network tocapture semantic interactions between arguments.Lei et al. (2017) conduct word pair interaction scoreto capture both linear and quadratic relation for ar-gument representation. However, these methodsutilize the pre-trained embeddings for mining theinteraction features and ignore the geometric struc-ture information entailed in discourse argumentsand their relation.

4.3 Joint Learning Perspective

Recently, some researches adopt joint learningframework to capture more discourse clues for thetask. Bai and Zhao (2018) jointly predict connec-tives and relations, assuming the shared parame-ters of the deep learning models. Xu et al. (2019)propose a topic tensor network (TTN) to modelthe sentence-level interactions and topic-level rel-evance among arguments for this task. However,few studies model discourse relations by translat-ing them in the low-dimensional embedding spaceas we do in this work.

TransE effectively maps the relation to the em-bedding space of entities by performing the alge-braic operation. Bordes et al. (2013) model entityrelations by interpreting them as translating op-eration in the low-dimensional embedding of theentities. Inspired by TransE, we design a TransSmethod to mine the latent geometric structure infor-mation, which could enhance the argument repre-sentations for promoting discourse relation recog-nition. To our knowledge, this is the first attemptto mine the latent geometric structure of argument-relation. Meanwhile, the embeddings of argumentand relation by TransS could be used to the otherhigh-level NLP tasks.

5 Conclusion

In this paper, we propose a novel TransS-drivenjoint learning neural network framework by op-timizing the discourse argument representationsto improve implicit discourse relation recognition.We interpret the discourse relations as translation inlow-dimensional embedding space, which reflectsthe geometric structure of argument-relation, andalso can obtain the richer argument representations

147

based on the multi-level encoder. Different fromthe conventional approaches only considering thesemantic features, we jointly leverage the latentgeometric structure information and the semanticfeatures to optimize the argument representations,which could improve the semantic understandingof discourse. Experimental results on the PDTBshow the effectiveness of our model.

Acknowledgments

We thank the anonymous reviewers for theirvaluable feedback. Our work is supportedby the National Key R&D Program of China(2019YFC1521200), the National Natural Sci-ence Foundation of China (61976154,U1736103),the Tianjin Natural Science Foundation (18JCY-BJC15500), and the Foundation of State Key Labo-ratory of Cognitive Intelligence, iFLYTEK (CIOS-20190001).

ReferencesHongxiao Bai and Hai Zhao. 2018. Deep enhanced

representation for implicit discourse relation recog-nition. In Proceedings of the 27th COLING, pages571–583.

Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko.2013. Translating embeddings for modeling multi-relational data. In Proceedings of the 26th NIPS,pages 2787–2795.

Chloe Braud and Pascal Denis. 2015. Comparing wordrepresentations for implicit discourse relation classi-fication. In Proceedings of the 2015 EMNLP, pages2201–2211.

Deng Cai and Hai Zhao. 2017. Pair-aware neural sen-tence modeling for implicit discourse relation classi-fication. In International Conference on Industrial,Engineering and Other Applications of Applied In-telligent Systems, pages 458–466. Springer.

Jifan Chen, Qi Zhang, Pengfei Liu, Xipeng Qiu, andXuanjing Huang. 2016. Implicit discourse relationdetection via a deep architecture with gated rele-vance network. In Proceedings of the 54th ACL,pages 1726–1735.

Zeyu Dai and Ruihong Huang. 2018. Improving im-plicit discourse relation classification by modelinginter-dependencies of discourse units in a paragraph.In Proceedings of the 2018 NAACL, pages 141–151.

Shima Gerani, Yashar Mehdad, Giuseppe Carenini,Raymond T. Ng, and Bita Nejat. 2014. Abstractivesummarization of product reviews using discoursestructure. In Proceedings of the 2014 EMNLP,pages 1602–1613.

Fengyu Guo, Ruifang He, Di Jin, Jianwu Dang, Long-biao Wang, and Xiangang Li. 2018. Implicit dis-course relation recognition using neural tensor net-work with interactive attention and sparse learning.In Proceedings of the 27th COLING, pages 547–558.

Yangfeng Ji and Jacob Eisenstein. 2015. One vector isnot enough: Entity-augmented distributional seman-tics for discourse relations. Transactions of the Asso-ciation for Computational Linguistics, 3:329–344.

Man Lan, Jianxiang Wang, Yuanbin Wu, Zheng-YuNiu, and Haifeng Wang. 2017. Multi-task attention-based neural networks for implicit discourse rela-tionship representation and identification. In Pro-ceedings of the 2017 EMNLP, pages 1299–1308.

Wenqiang Lei, Xuancong Wang, Meichun Liu, IlijaIlievski, Xiangnan He, and Min-Yen Kan. 2017.Swim: A simple word interaction model for implicitdiscourse relation recognition. In Proceedings of the26th IJCAI, pages 4026–4032.

Qi Li, Tianshi Li, and Baobao Chang. 2016. Discourseparsing with attention-based hierarchical neural net-works. In Proceedings of the 2016 EMNLP, pages362–371.

Yang Liu and Sujian Li. 2016. Recognizing implicitdiscourse relations via repeated reading: Neural net-works with multi-level attention. In Proceedings ofthe 2016 EMNLP, pages 1224–1233.

Yang Liu, Sujian Li, Xiaodong Zhang, and Zhifang Sui.2016. Implicit discourse relation classification viamulti-task neural networks. In Proceedings of the30th AAAI, pages 2750–2756.

Thomas Meyer and Andrei Popescu-Belis. 2012. Us-ing sense-labeled discourse connectives for statisti-cal machine translation. In Proceedings of the 13thEACL, pages 129–138.

Volodymyr Mnih, Nicolas Heess, Alex Graves, et al.2014. Recurrent models of visual attention. In Pro-ceedings of the 27th NIPS, pages 2204–2212.

Matthew Peters, Mark Neumann, Mohit Iyyer, MattGardner, Christopher Clark, Kenton Lee, and LukeZettlemoyer. 2018. Deep contextualized word rep-resentations. In Proceedings of the 2018 NAACL,pages 2227–2237.

Emily Pitler, Annie Louis, and Ani and Nenkova. 2009.Automatic sense prediction for implicit discourse re-lations in text. In Proceedings of the AFNLP, pages683–691.

Lianhui Qin, Zhisong Zhang, and Hai Zhao. 2016a.Implicit discourse relation recognition with context-aware character-enhanced embeddings. In Proceed-ings of the 26th COLING, pages 1914–1924.

Lianhui Qin, Zhisong Zhang, and Hai Zhao. 2016b.Shallow discourse parsing using convolutional neu-ral network. In CoNLL Shared Task, pages 70–77.

148

Lianhui Qin, Zhisong Zhang, and Hai Zhao. 2016c. Astacking gated neural architecture for implicit dis-course relation classification. In Proceedings of the2016 EMNLP, pages 2263–2270.

Lianhui Qin, Zhisong Zhang, Hai Zhao, Zhiting Hu,and Eric P. Xing. 2017. Adversarial connective-exploiting networks for implicit discourse relationclassification. In Proceedings of the 55th ACL,pages 1006–1017.

Attapol Rutherford and Nianwen Xue. 2014. Discover-ing implicit discourse relations through brown clus-ter pair representation and coreference patterns. InProceedings of the 14th EACL, pages 645–654.

Attapol T Rutherford, Vera Demberg, and NianwenXue. 2016. Neural network models for implicitdiscourse relation classification in english and chi-nese without surface features. arXiv preprintarXiv:1606.01990.

Sheng Xu, Peifeng Li, Fang Kong, Qiaoming Zhu, andGuodong Zhou. 2019. Topic tensor network for im-plicit discourse relation recognition in chinese. InProceedings of the 57th ACL, pages 608–618.

Biao Zhang, Jinsong Su, Deyi Xiong, Yaojie Lu, HongDuan, and Junfeng Yao. 2015. Shallow convolu-tional neural network for implicit discourse relationrecognition. In Proceedings of the 2015 EMNLP,pages 2230–2235.

Biao Zhang, Deyi Xiong, and Jinsong Su. 2016. Neuraldiscourse relation recognition with semantic mem-ory. arXiv preprint arXiv:1603.03873.