GILE: A Generalized Input-Label Embedding for Text Classification Nikolaos Pappas James Henderson Idiap Research Institute, Martigny 1920, Switzerland {nikolaos.pappas,[email protected]} Abstract Neural text classification models typically treat output labels as categorical variables that lack description and semantics. This forces their parametrization to be dependent on the label set size, and, hence, they are unable to scale to large label sets and generalize to unseen ones. Existing joint input-label text models overcome these issues by exploiting label descriptions, but they are unable to capture complex label relationships, have rigid parametrization, and their gains on unseen labels happen often at the expense of weak performance on the labels seen during training. In this paper, we propose a new input-label model that generalizes over previous such models, addresses their limitations, and does not compromise performance on seen labels. The model consists of a joint nonlinear input- label embedding with controllable capacity and a joint-space-dependent classification unit that is trained with cross-entropy loss to opti- mize classification performance. We evaluate models on full-resource and low- or zero- resource text classification of multilingual news and biomedical text with a large label set. Our model outperforms monolingual and multilingual models that do not leverage label semantics and previous joint input-label space models in both scenarios. 1 Introduction Text classification is a fundamental NLP task with numerous real-world applications such as topic recognition (Tang et al., 2015; Yang et al., 2016), sentiment analysis (Pang and Lee, 2005; Yang et al., 2016), and question answering (Chen et al., 2015; Kumar et al., 2015). Classification also appears as a sub-task for sequence prediction tasks such as neural machine translation (Cho et al., 2014; Luong et al., 2015) and summarization (Rush et al., 2015). Despite numerous studies, existing models are trained on a fixed label set using k-hot vectors, and therefore treat target labels as mere atomic symbols without any particular structure to the space of labels, ignoring potential linguistic knowledge about the words used to describe the output labels. Given that semantic representations of words have been shown to be useful for representing the input, it is reasonable to expect that they are going to be useful for representing the labels as well. Previous work has leveraged knowledge from the label texts through a joint input-label space, initially for image classification (Weston et al., 2011; Mensink et al., 2012; Frome et al., 2013; Socher et al., 2013). Such models generalize to labels both seen and unseen during training, and scale well on very large label sets. However, as we explain in Section 2, existing input-label models for text (Yazdani and Henderson, 2015; Nam et al., 2016) have the following limitations: (i) their embedding does not capture complex label relationships due to its bilinear form, (ii) their output layer parametrization is rigid because it depends on the dimensionality of the encoded text and labels, and (iii) they are outperformed on seen labels by classification baselines trained with cross-entropy loss (Frome et al., 2013; Socher et al., 2013). In this paper, we propose a new joint input- label model that generalizes over previous such models, addresses their limitations, and does not compromise performance on seen labels (see Figure 1). The proposed model is composed of a joint nonlinear input-label embedding with controllable capacity and a joint-space- dependent classification unit which is trained 139 Transactions of the Association for Computational Linguistics, vol. 7, pp. 139–155, 2019. Action Editor: Eneko Agirre. Submission batch: 9/2018; Revision batch: 1/2019; Published 4/2019. c 2019 Association for Computational Linguistics. Distributed under a CC-BY 4.0 license.
Transcript
GILE: A Generalized Input-Label Embedding for Text Classification
Nikolaos Pappas James Henderson
Idiap Research Institute, Martigny 1920, Switzerland{nikolaos.pappas,[email protected]}
Abstract
Neural text classification models typicallytreat output labels as categorical variables thatlack description and semantics. This forcestheir parametrization to be dependent on thelabel set size, and, hence, they are unableto scale to large label sets and generalize tounseen ones. Existing joint input-label textmodels overcome these issues by exploitinglabel descriptions, but they are unable tocapture complex label relationships, have rigidparametrization, and their gains on unseenlabels happen often at the expense of weakperformance on the labels seen during training.In this paper, we propose a new input-labelmodel that generalizes over previous suchmodels, addresses their limitations, and doesnot compromise performance on seen labels.The model consists of a joint nonlinear input-label embedding with controllable capacityand a joint-space-dependent classification unitthat is trained with cross-entropy loss to opti-mize classification performance. We evaluatemodels on full-resource and low- or zero-resource text classification of multilingualnews and biomedical text with a large labelset. Our model outperforms monolingual andmultilingual models that do not leverage labelsemantics and previous joint input-label spacemodels in both scenarios.
1 Introduction
Text classification is a fundamental NLP taskwith numerous real-world applications such astopic recognition (Tang et al., 2015; Yang et al.,2016), sentiment analysis (Pang and Lee, 2005;Yang et al., 2016), and question answering (Chenet al., 2015; Kumar et al., 2015). Classification
also appears as a sub-task for sequence predictiontasks such as neural machine translation (Choet al., 2014; Luong et al., 2015) and summarization(Rush et al., 2015). Despite numerous studies,existing models are trained on a fixed label setusing k-hot vectors, and therefore treat targetlabels as mere atomic symbols without anyparticular structure to the space of labels, ignoringpotential linguistic knowledge about the wordsused to describe the output labels. Given thatsemantic representations of words have beenshown to be useful for representing the input,it is reasonable to expect that they are going to beuseful for representing the labels as well.
Previous work has leveraged knowledge fromthe label texts through a joint input-label space,initially for image classification (Weston et al.,2011; Mensink et al., 2012; Frome et al., 2013;Socher et al., 2013). Such models generalize tolabels both seen and unseen during training, andscale well on very large label sets. However,as we explain in Section 2, existing input-labelmodels for text (Yazdani and Henderson, 2015;Nam et al., 2016) have the following limitations:(i) their embedding does not capture complexlabel relationships due to its bilinear form, (ii)their output layer parametrization is rigid becauseit depends on the dimensionality of the encodedtext and labels, and (iii) they are outperformed onseen labels by classification baselines trained withcross-entropy loss (Frome et al., 2013; Socheret al., 2013).
In this paper, we propose a new joint input-label model that generalizes over previous suchmodels, addresses their limitations, and doesnot compromise performance on seen labels(see Figure 1). The proposed model is composedof a joint nonlinear input-label embeddingwith controllable capacity and a joint-space-dependent classification unit which is trained
139
Transactions of the Association for Computational Linguistics, vol. 7, pp. 139–155, 2019. Action Editor: Eneko Agirre.Submission batch: 9/2018; Revision batch: 1/2019; Published 4/2019.
with cross-entropy loss to optimize classificationperformance.1 The need for capturing complexlabel relationships is addressed by two nonlineartransformations that have the same target jointspace dimensionality. The parametrization of theoutput layer is not constrained by the dimen-sionality of the input or label encoding, but isinstead flexible with a capacity that can be easilycontrolled by choosing the dimensionality of thejoint space. Training is performed with cross-entropy loss, which is a suitable surrogate loss forclassification problems, as opposed to a rankingloss such as WARP loss (Weston et al., 2010),which is more suitable for ranking problems.
Evaluation is performed on full-resource andlow- or zero-resource scenarios of two text clas-sification tasks, namely, on biomedical semanticindexing (Nam et al., 2016) and on multilingualnews classification (Pappas and Popescu-Belis,2017), against several competitive baselines. Inboth scenarios, we provide a comprehensive abla-tion analysis that highlights the importance ofeach model component and the difference withprevious embedding formulations when using thesame type of architecture and loss function.
Our main contributions are the following:
(i) We identify key theoretical and practical lim-itations of existing joint input-label models.
(ii) We propose a novel joint input-label embed-ding with flexible parametrization that gen-eralizes over the previous such models andaddresses their limitations.
(iii) We provide empirical evidence of the supe-riority of our model over monolingual andmultilingual models that ignore label se-mantics, and over previous joint input-labelmodels on both seen and unseen labels.
The remainder of this paper is organized asfollows. Section 2 provides background knowl-edge and explains limitations of existing models.Section 3 describes the model components, train-ing, and relation to previous formulations. Sec-tion 4 describes our evaluation results andanalysis, while Section 5 provides an overviewof previous work and Section 6 concludes thepaper and provides future research directions.
1Our code is available at:github.com/idiap/gile.
2 Background: Neural TextClassification
We are given a collection D = {(xi, yi), i =1, . . . , N} made of N documents, where eachdocument xi is associated with labels yi ={yij ∈ {0, 1} | j = 1, . . . , k}, and k is thetotal number of labels. Each document xi ={w11, w12, . . . , wKiTKi
} is a sequence of wordsgrouped into sentences, withKi being the numberof sentences in document i and Tj being thenumber of words in sentence j. Each label j hasa textual description composed of multiple words,cj = {cj1, cj2, . . . , cjLj | j = 1, . . . , k} with Lj
being the number of words in each description.Given the input texts and their associated labelsseen during the training portion of D, our goalis to learn a text classifier that is able to predictlabels both in the seen, Ys, or unseen, Yu, labelsets, defined as the sets of unique labels that havebeen seen or not during training, respectively, and,hence, Y ∩ Yu = ∅ and Y = Ys ∪ Yu.2
2.1 Input Text Representation
To encode the input text, we focus on hierar-chical attention networks (HANs), which arecompetitive for monolingual (Yang et al., 2016)and multilingual text classification (Pappas andPopescu-Belis, 2017). The model takes as inputa document x and outputs a document vector h.The input words and label words are representedby vectors in IRd from the same3 embeddingsE ∈ IR|V|×d, where V is the vocabulary and d isthe embedding dimension; E can be pre-trainedor learned jointly with the rest of the model. Themodel has two levels of abstraction, word andsentence. The word level is made of an encodernetwork gw and an attention network aw, whilethe sentence level similarly includes an encoderand an attention network.
Encoders. The function gw encodes the se-quence of input words {wit | t = 1, . . . , Ti} foreach sentence i of the document, noted as:
h(it)w = gw(wit), t ∈ [1, Ti] (1)
2Note that depending on the number of labels per docu-ment the problem can be a multi-label or multi-class problem.
3This statement holds true for multilingual classifica-tion problems, too, if the embeddings are aligned acrosslanguages.
and at the sentence level, after combining theintermediate word vectors {h(it)
w | t = 1, . . . , Ti}to a sentence vector si ∈ IRdw (see below), wheredw is the dimension of the word encoder, thefunction gs encodes the sequence of sentencevectors {si | i = 1, . . . ,K}, noted as h(i)
s . The gwand gs functions can be any feed-forward (DENSE)or recurrent networks, for example, GRU (Choet al., 2014).
Attention. The αw and αs attention mechanisms,which estimate the importance of each hiddenstate vector, are used to obtain the sentence siand document representation h, respectively. Thesentence vector is thus calculated as follows:
si =
Ti∑t=1
α(it)w h(it)
w =
Ti∑t=1
exp(v>ituw)∑j exp(v>ijuw)
h(it)w
(2)
where vit = fw(h(it)w ) is a fully connected net-
work with Ww parameters. The document vectorh ∈ IRdh , where dh is the dimension of the sen-tence encoder, is calculated similarly, by replacinguit with vi = fs(h
(i)s ) which is a fully connected
network with Ws parameters, and uw with us,which are parameters of the attention functions.
2.2 Label Text Representation
To encode the label text we use an encoderfunction that takes as input a label descriptioncj and outputs a label vector ej ∈ IRdc ∀j =1, . . . , k. For efficiency reasons, we use a simple,parameter-free function to compute ej , namely,the average of word vectors which describe labelj, namely, ej = 1
Lj
∑Lj
t=1 cjt, and hence dc = d
in this case. By stacking all these label vectorsinto a matrix, we obtain the label embeddingE ∈ IR|Y|×d. In principle, we could also use thesame encoder functions as the ones for inputtext, but this would increase the computationsignificantly; hence, we keep this direction asfuture work.
2.3 Output Layer Parametrizations
2.3.1 Typical Linear UnitThe most typical output layer consists of a linearunit with a weight matrix W ∈ IRdh×|Y| and abias vector b ∈ IR|Y| followed by a softmax orsigmoid activation function. Given the encoder’shidden representation h with dimension size dh,
the probability distribution of output y given inputx is proportional to the following quantity:
p(y|x) ∝ exp(W>h+ b) (3)
The parameters in W can be learned separately orbe tied with the parameters of the embedding Eby settingW = ET if the input dimension ofW isrestricted to be the same as that of the embeddingE (d = dh) and each label is represented by asingle word description (i.e., when Y correspondsto V and E = E). In the latter case, Equation (3)becomes:
p(y|x) ∝ exp(Eh+ b) (4)
Either way, the parameters of such models aretypically learned with cross-entropy loss, whichis suitable for classification problems. However,in both cases they cannot be applied to labels thatare not seen during training, because each labelhas learned parameters which are specific to thatlabel, so the parameters for unseen labels cannotbe learned. We now turn our focus to a class ofmodels that can handle unseen labels.
2.3.2 Bilinear Input-Label UnitJoint input–output embedding models can gen-eralize from seen to unseen labels because theparameters of the label encoder are shared. Thepreviously proposed joint input–output embed-ding models by Yazdani and Henderson (2015)and Nam et al. (2016) are based on the followingbilinear ranking function f(·):
f(x, y) = EWh (5)
where E ∈ IR|Y|×d is the label embedding andW ∈ IRd×dh is the bilinear embedding. Thisfunction allows one to define the rank of a givenlabel y with respect to x and is trained using hingeloss to rank positive labels higher than negativeones. But note that the use of this ranking lossmeans that they do not model the conditionalprobability, as do the traditional models above.
Limitations. Firstly, Equation (5) can only cap-ture linear relationships between encoded text(h) and label embedding (E) through W . Weargue that the relationships between differentlabels are nonlinear because of the complexinteractions of the semantic relations across labelsbut also between labels and different encodedinputs. A more appropriate form for this purpose
141
would include a nonlinear transformation σ(·), forexample, with either:
(a) σ(EW)︸ ︷︷ ︸Label structure
h or (b) E σ(Wh)︸ ︷︷ ︸Input structure
(6)
Secondly, it is hard to control their output layercapacity because of their bilinear form, whichuses a matrix of parameters (W) whose size isbounded by the dimensionalities of the labelembedding and the text encoding. Thirdly, theirloss function optimizes ranking instead of clas-sification performance and thus treats the ground-truth as a ranked list when in reality it consists ofone or more independent labels.
Summary. We hypothesize that these are thereasons why these models do not yet perform wellon seen labels compared to models that make useof the typical linear unit, and they do not take fulladvantage of the structure of the problem whentested on unseen labels. Ideally, we would like tohave a model that will address these issues andwill combine the benefits from both the typicallinear unit and the joint input-label models.
3 The Proposed Output LayerParametrization for TextClassification
We propose a new output layer parametrizationfor neural text classification which is composed ofa generalized input-label embedding that capturesthe structure of the labels, the structure of theencoded texts and the interactions between thetwo, followed by a classification unit which isindependent of the label set size. The resultingmodel has the following properties: (i) it is ableto capture complex output structure, (ii) it has aflexible parametrization that allows its capacityto be controlled, and (iii) it is trained with aclassification surrogate loss such as cross-entropy.The model is depicted in Figure 1. In this section,we describe the model in detail, showing how itcan be trained efficiently for arbitrarily large labelsets and how it is related to previous models.
3.1 A Generalized Input-Label Embedding
Let gin(h) and gout(ej) be two nonlinear projec-tions of the encoded input, namely, the documenth, and any encoded label ej , where ej is the jth
h' e2
U
V∧h' e1
Classification unit
w
y1∧y2
…
Joint space
h' ek yk∧…
LabelEncoder
cj1cj2
cjLj
...ei
InputEncoder
w11w12
wKiTKi
...h
Encodersdh x dj
d x dj
Wor
d em
bedd
ings
''
'
T
Figure 1: Each encoded text and label are projectedto a joint input-label multiplicative space, the outputof which is processed by a classification unit withlabel-set-size independent parametrization.
row vector from the label embedding matrix E ,which have the following form:
e′j = gout(ej) = σ(ejU + bu) (7)
h′ = gin(h) = σ(V h+ bv) (8)
where σ(·) is a nonlinear activation function suchas ReLU or Tanh, the matrix U ∈ IRd×dj and biasbu ∈ IRdj are the linear projection of the labels,and the matrix V ∈ IRdj×dh and bias bv ∈ IRdj arethe linear projection of the encoded input. Notethat the projections for h′ and e′j could behigh-rank or low-rank depending on their initialdimensions and the target joint space dimension.Also let E ′ ∈ IR|Y|×dj be the matrix resulting fromprojecting all the outputs ej to the joint space, thatis, gout(E).
The conditional output probability distributioncan now be rewritten as:
p(y|x) ∝ exp(E ′h′
)∝ exp
(gout(E)gin(h)
)∝ exp
(σ(EU + bu)︸ ︷︷ ︸Label Structure
σ(V h+ bv)︸ ︷︷ ︸Input Structure
)(9)
Crucially, this function has no label-set-sizedependent parameters, unlike W and b in Equa-tion (3). In principle, this parametrization can beused for both multi-class and multi-label problemsby defining the exponential in terms of a softmaxand sigmoid functions, respectively. However, inthis paper we will focus on the latter.
3.2 Classification Unit
We require that our classification unit parametersdepend only on the joint input-label space above.To represent the compatibility between any en-coded input text hi and any encoded label ejfor this task, we define their joint representation
142
based on multiplicative interactions in the jointspace:
g(ij)joint = gin(hi)� gout(ej) (10)
where � is component-wise multiplication.The probability for hi to belong to one of the
k known labels is modeled by a linear unit thatmaps any point in the joint space into a scorewhich indicates the validity of the combination:
p(ij)val = g
(ij)jointw + b (11)
where w ∈ IRdj and b are a scalar variables. Wecompute the output of this linear unit for eachknown label which we would like to predict for agiven document i, namely:
P(i)val =
p(i1)val
p(i2)val
. . .
p(ik)val
=
g(i1)jointw + b
g(i2)jointw + b
. . .
g(ik)jointw + b
(12)
For each row, the higher the value the more likelythe label is to be assigned to the document. Toobtain valid probability estimates and be able totrain with binary cross-entropy loss for multi-label classification, we apply a sigmoid functionas follows:
yi = p(yi|xi) =1
1 + e−P(i)
val
(13)
Summary. By adding the above changes tothe general form of Equation (9) the conditionalprobability p(yi|xi) is now proportional to thefollowing quantity:
exp(σ(EU + bu)(σ(V h+ bv)� w) + b
)(14)
Note that the number of parameters in this equationis independent of the size of the label set, giventhat U , V , w, and b depend only on dj , and k canvary arbitrarily. This allows the model to scale upto large label sets and generalize to unseen labels.Lastly, the proposed output layer addresses all thelimitations of the previous models, as follows:(i) it is able to capture complex structure in thejoint input–output space, (ii) it provides a meansto easily control its capacity dj , and (iii) it istrainable with cross-entropy loss.
3.3 Training Objectives
The training objective for the multi-label classifi-cation task is based on binary cross-entropy loss.Assuming θ contains all the parameters of themodel, the training loss is computed as follows:
L(θ) = − 1
Nk
N∑i=1
k∑j=1
H(yij , yij) (15)
where H is the binary cross-entropy between thegold label yij and predicted label yij for a docu-ment i and a candidate label j.
We handle multiple languages according toFirat et al. (2016) and Pappas and Popescu-Belis(2017). Assuming that Θ = {θ1, θ2, ..., θM} areall the parameters required for each of the Mlanguages, we use a joint multilingual objectivebased on the sum of cross-entropy losses:
L(Θ) = − 1
Z
Ne∑i
M∑l
k∑j=1
H(y(l)ij , y
(l)ij ) (16)
where Z = NeMk with Ne being the numberof examples per epoch. At each iteration, adocument-label pair for each language is sampled.In addition, multilingual models share a certainsubset of the encoder parameters during train-ing while the output layer parameters are keptlanguage-specific, as described by Pappas andPopescu-Belis (2017). In this paper, we sharemost of the output layer parameters, namely, theones from the input-label space (U, V, bv, bu), andwe keep only the classification unit parameters(w, b) language-specific.
3.4 Scaling Up to Large Label Sets
For a very large number dj of joint-space di-mensions in our parametrization, the computa-tional complexity increases prohibitively becauseour projection requires a large matrix multiplica-tion between U and E, which depends on |Y|. Insuch cases, we resort to sampling-based trainingby adopting the commonly used negative samplingmethod proposed by Mikolov et al. (2013). Letxi ∈ IRd and yik ∈ {0, 1} be an input-label pairand yik the output probabilities from our model(Equation (14)). By introducing the sets kpi andkni , which contain the indices of the positive andnegative labels respectively for the i-th input, the
143
loss L(θ) in Equation (15) can be re-written asfollows:
= − 1
Z
N∑i=1
k∑j=1
[yij log yij + yij log (1− yij)
]
= − 1
Z
N∑i=1
[ kpi∑j=1
log yij +
kni∑j=1
log (1− yij)](17)
where Z = Nk and yij is (1 − yij). To reducethe computational cost needed to evaluate yijfor all the negative label set kni , we sample k∗
labels from the negative label set with probabilityp = 1
|kni |to create the set kni . This enables training
on arbitrarily large label sets without increasingthe computation required. By controlling thenumber of samples we can drastically speed upthe training time, as we demonstrate empiricallyin Section 4.2.2. Exploring more informativesampling methods (e.g., importance sampling)would be an interesting direction of future work.
3.5 Relation to Previous Parametrizations
The proposed embedding form can be seen asa generalization over the input-label embeddingswith a bilinear form, because its degenerate formis equivalent to the bilinear form of Equation (5).In particular, this can be simply derived if we setone of the two nonlinear projection functions inthe second line of Equation (9) to be the identityfunction (e.g., gout(·) = I), set all biases to zero,and make the σ(.) activation function linear, asfollows:
σ(EU + bu)σ(V h+ bv) = (EI) (V h)
= EV h (18)
where V by consequence has the same numberof dimensions as W ∈ IRd×dh from the bilinearinput-label embedding model of Equation (5).
�
4 Experiments
The evaluation is performed on large-scalebiomedical semantic indexing using the BioASQdata set, obtained by Nam et al. (2016), and onmultilingual news classification using the DWcorpus, which consists of eight language data setsobtained by Pappas and Popescu-Belis (2017). Thestatistics of these data sets are listed in Table 1.
Data set Documents Labelsabbrev. # count # words wd # count wl
BioASQ 11,705,534 528,156 214 26,104 35.0DW 598,304 884,272 436 5,637 2.3– en 112,816 110,971 516 1,385 2.1– de 132,709 261,280 424 1,176 1.8– es 75,827 130,661 412 843 4.7– pt 39,474 58,849 571 396 1.8– uk 35,423 105,240 342 288 1.7– ru 108,076 123,493 330 916 1.8– ar 57,697 58,922 357 435 2.4– fa 36,282 34,856 538 198 2.5
Table 1: Data set statistics. #count is the number ofdocuments, #words are the number of unique words inthe vocabulary V , wd and wl are the average numberof words per document and label, respectively.
4.1 Biomedical Text Classification
We evaluate on biomedical text classificationto demonstrate that our generalized input-labelmodel scales to very large label sets and performsbetter than previous joint input-label models onboth seen and unseen label prediction scenarios.
4.1.1 SettingsWe follow the exact evaluation protocol, data,and settings of Nam et al. (2016), as describedbelow. We use the BioASQ Task 3a data set,which is a collection of scientific publications inbiomedical research. The data set contains about12M documents labeled with around 11 labelsout of 27,455, which are defined according tothe Medical Subject Headings (MESH) hierarchy.The data were minimally pre-processed withtokenization, number replacements (NUM), rareword replacements (UNK), and split with theprovided script by year so that the training setincludes all documents until 2004 and the onesfrom 2005 to 2015 were kept for the test set; thiscorresponded to 6,692,815 documents for trainingand 4,912,719 for testing. For validation, a set of100,000 documents were randomly sampled fromthe training set. We report the same ranking-basedevaluation metrics as Nam et al. (2016), namely,rank loss (RL), average precision (AvgPr), andone-error loss (OneErr).
Our hyper-parameters were selected on valida-tion data based on average precision as follows:100-dimensional word embeddings, encoder, at-tention (same dimensions as the baselines), jointinput-label embedding of 500, batch size of 64,
144
Model Layer form Dim Seen labels Unseen labels Paramsabbrev. output #count RL AvgPr OneErr RL AvgPr OneErr #count
Table 2: Biomedical semantic indexing results computed over labels seen and unseen during training, i.e., thefull-resource versus zero-resource settings. Best scores among the competing models are marked in bold.
maximum number of 300 words per documentand 50 words per label, ReLU activation, 0.3%negative label sampling, and optimization withADAM until convergence. The word embeddingswere learned end-to-end on the task.4
The baselines are the joint input-label modelsfrom Nam et al. (2016), noted as [N16], namely:
• WSABIE+: This model is an extension ofthe original WSABIE model by Weston et al.(2011), which, instead of learning a rankingmodel with fixed document features, jointlylearns features for documents and words, andis trained with the WARP ranking loss.
• AiTextML: This model is the one proposedby Nam et al. (2016) with the purpose oflearning joint representations of documents,labels, and words, along with a joint input-label space that is trained with the WARPranking loss.
The scores of the WSABIE+ and AiTextMLbaselines in Table 2 are the ones reported byNam et al. (2016). In addition, we report scoresof a word-level attention neural network (WAN)with DENSE encoder and attention followed by asigmoid output layer, trained with binary cross-entropy loss.5 Our model replaces WAN’s output
4Here, the word embeddings are included in the parameterstatistics because they are variables of the network.
5In our preliminary experiments, we also trained theneural model with a hinge loss as WSABIE+ and AiTextML,but it performed similarly to them and much worse thanWAN, so we did not further experiment with it.
layer with a generalized input-label embeddinglayer and its variations, noted GILE-WAN. Forcomparison, we also compare to bilinear input-label embedding versions of WAN for the modelby Yazdani and Henderson (2015), noted asBIL-WAN [YH16], and the one by Nam et al.(2016), noted as BIL-WAN [N16]. Note that theAiTextML parameter space is huge and makeslearning difficult for our models (linear withrespect to labels and documents). Instead, wemake sure that our models have far fewer pa-rameters than the baselines (Table 2).
4.1.2 Results
The results on biomedical semantic indexing onseen and unseen labels are shown in Table 2.We observe that the neural baseline, WAN, out-performs WSABIE+ and AiTextML on the seenlabels, by +5.73 and +9.59 points in terms ofAvgPr, respectively. The differences are evenmore pronounced when considering the rankingloss and one error metrics. This result is com-patible with previous findings that existing jointinput-label models are not able to outperformstrong supervised baselines on seen labels. How-ever, WAN is not able to generalize at all to unseenlabels, hence the WSABIE+ and AiTextML havea clear advantage in the zero-resource setting.
In contrast, our generalized input-label model,GILE-WAN, outperforms WAN even on seenlabels, where our model has higher averageprecision by +2.02 points, better ranking lossby +43% and comparable OneErr (−3%). Andthis gain is not at the expense of performance
145
on unseen labels. GILE-WAN outperformsWSABIE+ and AiTextML variants6 by a largemargin in both cases—for example, by +7.75,+11.61 points on seen labels and by +12.58,+10.29 points in terms of average precision onunseen labels, respectively. Interestingly, our GILE-WAN model also outperforms the two previousbilinear input-label embedding formulations ofYazdani and Henderson (2015) and Nam et al.(2016), namely, BIL-WAN [YH15] and BIL-WAN[N16], by +3.71, +2.48 points on seen labelsand +3.45 and +2.39 points on unseen labels,respectively, even when they are trained with thesame encoders and loss as ours. These models arenot able to outperform the WAN baseline whenevaluated on the seen labels, that is they have−1.68 and −0.46 points lower average precisionthan WAN, but they outperform WSABIE+ andAiTextML on both seen and unseen labels.Overall, the results show a clear advantage of ourgeneralized input-label embedding model againstprevious models on both seen and unseen labels.
4.1.3 Ablation Analysis
To evaluate the effectiveness of individual com-ponents of our model, we performed an ablationstudy (last three rows in Table 2). Note that whenwe use only the label or only the input embeddingin our generalized input-label formulation, thedimensionality of the joint space is constrainedto be the dimensionality of the encoded labelsand inputs respectively (i.e., dj=100 in ourexperiments).
All three variants of our model outperformprevious embedding formulations of Nam et al.(2016) and Yazdani and Henderson (2015) in allmetrics except for AvgPr on seen labels, wherethey score slightly lower. The decrease in AvgPrecfor our model variants with dj=100 comparedwith the neural baselines could be attributedto the difficulty in learning the parameters ofa highly nonlinear space with only a few hid-den dimensions. Indeed, when we increase thenumber of dimensions (dj=500), our full modeloutperforms them by a large margin. Recall thatthis increase in capacity is only possible with ourfull model definition in Equation (9) and noneof the other variants allow us to do this without
6Namely, avg when using the average of word vectorsand inf when using inferred label vectors to make predictions.
interfering with the original dimensionality of theencoded labels (E) and input (ht). In addition, ourmodel variants with dj=100 exhibit consistentlyhigher scores than baselines in terms of mostmetrics on both seen and unseen labels, whichsuggests that they are able to capture morecomplex relationships across labels and betweenencoded inputs and labels.
Overall, the best performance among our modelvariants is achieved when using only the labelembedding and, hence, it is the most significantcomponent of our model. Surprisingly, our modelwith only the label embedding achieves higherperformance than our full model on unseen labelsbut it is far behind our full model when weconsider performance on both seen and unseenlabels. When we constrain our full model to havethe same dimensionality with the other variants(i.e., dj=100), it outperforms the one that usesonly the input embedding in most metrics and it isoutperformed by the one that uses only the labelembedding.
4.2 Multilingual News Text Classification
We evaluate on multilingual news text clas-sification to demonstrate that our output layerbased on the generalized input-label embeddingoutperforms previous models with a typical outputlayer in a wide variety of settings, even for labelsthat have been seen during training.
4.2.1 SettingsWe follow the exact evaluation protocol, data, andsettings of Pappas and Popescu-Belis (2017), asdescribed below. The data set is split per languageinto 80% for training, 10% for validation, and10% for testing. We evaluate on both types oflabels (general Yg, and specific Ys) in a full-resource scenario, and we evaluate only on thegeneral labels (Yg) in a low-resource scenario.Accuracy is measured with the micro-averagedF1 percentage scores.
The word embeddings for this task are thealigned pre-trained 40-dimensional multi-CCAmultilingual word embeddings by Ammar et al.(2016) and are kept fixed during training.7 Thesentences are already truncated at a length of30 words and the documents at a length of 30sentences. The hyper-parameters were selected
7The word embeddings are not included in the parametersstatistics because they are not variables of the network.
146
Models Languages (en + aux→ en) Languages (en + aux→ aux) Stat.Yg abbrev. de es pt uk ru ar fa de es pt uk ru ar fa avg
Table 3: Full-resource classification results on general (upper half) and specific (lower half) labels usingmonolingual and bilingual models with DENSE encoders on English as target (left) and the auxiliary language astarget (right). The average bilingual F1-score (%) is noted avg and the top ones per block are underlined. Themonolingual scores on the left come from a single model, hence a single score is repeated multiple times; therepetition is marked with consecutive dots.
on validation data as follows: 100-dimensionalencoder and attention, ReLU activation, batchsize 16, epoch size 25k, no negative sampling (alllabels are used), and optimization with ADAMuntil convergence. To ensure equal capacity tobaselines, we use approximately the same numberof parameters ntot with the baseline classificationlayers, by setting:
dj 'dh ∗ |k(i)|dh + d
, i = 1, . . . ,M (19)
in the monolingual case, and similarly, dj '(dh ∗
∑Mi=1 |k(i)|)/(dh + d) in the multilingual
case, where k(i) is the number of labels in lan-guage i.
The hierarchical models have Dense encodersin all scenarios (Tables 3, 6, and 7), exceptfrom the varying encoder experiment (Table 4).For the low-resource scenario, the levels of dataavailability are: tiny from 0.1% to 0.5%, smallfrom 1% to 5% and medium from 10% to 50%of the original training set. For each level, theaverage F1 across discrete increments of 0.1, 1
and 10 are reported respectively. The decisionthresholds, which were tuned on validation databy Pappas and Popescu-Belis (2017), are set asfollows: for the full-resource scenario it is set to0.4 for |Ys| < 400 and 0.2 for |Ys| ≥ 400, andfor the low-resource scenario it is set to 0.3 forall sets.
The baselines are all the monolingual andmultilingual neural networks from Pappas andPopescu-Belis (2017),8 noted as [PB17], namely:
• NN: A neural network that feeds the av-erage vector of the input words directly toa classification layer, as the one used byKlementiev et al. (2012).
• HNN: A hierarchical network with encodersand average pooling at every level, followedby a classification layer, as the one used byTang et al. (2015).
8For reference, in Table 4 we also compare to a logisticregression trained with unigrams over the full vocabulary andover the top-10% most frequent words by Mrini et al. (2017),noted as [M17], which use the same settings and data.
147
Models Languages Statisticsabbrev. en de es pt uk ru ar fa nl fl
Table 4: Full-resource classification results on general (Yg) topic labels with DENSE and GRU encoders. Reportedare also the average number of parameters per language (nl) and the average F1 per language (fl).
• HAN: A hierarchical network with encodersand attention, followed by a classificationlayer, as the one used by Yang et al. (2016).
• MHAN: Three multilingual hierarchical net-works with shared encoders, noted MHAN-Enc, shared attention, noted MHAN-Att,and shared attention and encoders, notedMHAN-Both, as the ones used by Pappasand Popescu-Belis (2017).
To ensure a controlled comparison to theabove baselines, for each model we evaluatea version where their output layer is replacedby our generalized input-label embedding outputlayer using the same number of parameters; thesehave the abbreviation ‘‘GILE’’ prepended in theirname (e.g., GILE-HAN). The scores of HAN andMHAN models in Tables 3, 6, and 7 are the onesreported by Pappas and Popescu-Belis (2017),while for Table 4 we train them ourselves usingtheir code. Lastly, the best score for each pairwisecomparison between a joint input-label model andits counterpart is marked in bold.
4.2.2 ResultsTable 3 displays the results of full-resource docu-ment classification using DENSE encoders for bothgeneral and specific labels. On the left, we displaythe performance of models on the English sub-corpus when English and an auxiliary language areused for training, and on the right, the performanceon the auxiliary language sub-corpus when thatlanguage and English are used for training.
The results show that in 98% of comparisonson general labels (top half of Table 3) the jointinput-label models improve consistently over the
corresponding models using a typical sigmoidclassification layer. This finding validates ourmain hypothesis that the joint input-label modelssuccessfully exploit the semantics of the labels,which provide useful cues for classification, asopposed to models which are agnostic to labelsemantics. The results for specific labels (bottomhalf of Table 3) demonstrate the same trend, withthe joint input-label models performing better in87% of comparisons.
In Table 5, we also directly compare our embed-ding to previous bilinear input-label embeddingformulations when using the best monolingualconfiguration (HAN) from Table 3, exactly asdone in Section 4.1. The results on the generallabels show that GILE outperforms the previousbilinear input-label models, BIL [YH15] and BIL[N16], by +1.62 and +3.3 percentage pointson average, respectively. This difference is muchmore pronounced on the specific labels, wherethe label set is much larger, namely, +6.5 and+13.5 percentage points, respectively. Similarly,our model with constrained dimensionality is alsoas good or better on average than the bilinearinput-label models, by +0.9 and +2.2 on generallabels and by −0.5 and +6.1 on specific labelsrespectively, which highlights the importance oflearning nonlinear relationships across encodedlabels and documents. Among our ablated modelvariants, as in the previous section, the best isthe one with only the label projection but it stillworse than our full model by −5.2 percentagepoints. The improvements of GILE against eachbaseline is significant and consistent on both datasets. Hence, in the following experiments we willonly consider the best of these alternatives.
148
HAN LanguagesYg output layer en de es pt uk ru ar faLinear [PB17] 71.2 71.8 82.8 71.3 85.3 79.8 80.5 76.6BIL [YH15] 71.7 70.5 82.0 71.1 86.6 80.6 80.4 76.0BIL [N16] 69.8 69.1 80.9 67.4 87.5 79.9 78.4 75.1GILE (Ours) 76.5 74.2 83.4 71.9 86.1 82.7 82.6 77.2- constrained dj 73.6 73.1 83.3 71.0 87.1 81.6 80.4 76.4- only label 71.4 69.6 82.1 70.3 86.2 80.6 81.1 76.2- only input 55.1 54.2 80.6 66.5 85.6 60.8 78.9 74.0
Ys output layer en de es pt uk ru ar faLinear[PB17] 43.4 44.8 46.3 41.9 46.4 45.8 41.2 49.4BIL [YH15] 40.7 37.8 38.1 33.5 44.6 38.1 39.1 42.6BIL [N16] 34.4 30.2 34.4 33.6 31.4 22.8 35.6 38.9GILE (Ours) 45.9 47.3 47.4 42.6 46.6 46.9 41.9 48.6- constrained dj 38.5 38.0 36.8 35.1 42.1 36.1 36.7 48.7- only label 38.4 41.5 42.9 38.3 44.0 39.3 37.2 43.4- only input 12.1 10.8 8.8 20.5 11.8 7.8 12.0 24.6
Table 5: Direct comparison with previous bilinear input-label models, namely, BIL [YH15] and BIL [N16], andwith our ablated model variants using the best monolingual configuration (HAN) from Table 3 on both general(upper half) and specific (lower half) labels. Best scores among the competing models are marked in bold.
The best bilingual performance on average isthat of the GILE-MHAN-Att model, for bothgeneral and specific labels. This improvementcan be attributed to the effective sharing betweenlabel semantics across languages through the jointmultilingual input-label output layer. Effectively,this model has the same multilingual sharingscheme with the best model reported by Pappasand Popescu-Belis (2017), MHAN-Att, namely,sharing attention at each level of the hierarchy,which agrees well with their main finding.
Interestingly, the improvement holds whenusing different types of hierarchical encoders,namely, DENSE GRU, and biGRU, as shown inTable 4, which demonstrate the generality of theapproach. In addition, our best models outperformlogistic regression trained either on top-10% mostfrequent words or on the full vocabulary, eventhough our models utilize many fewer parameters,that is, 377K/138K vs. 26M/5M. Increasing thecapacity of our models should lead to even furtherimprovements.
Multilingual learning. So far, we have shownthat the proposed joint input-label models out-perform typical neural models when training withone and two languages. Does the improvementremain when increasing the number of languageseven more? To answer the question we report inTable 6 the average F1-score per language forthe best baselines from the previous experiment(HAN and MHAN-Att) with the proposed joint
Models General labelsSpecific labelsabbrev. # lang. nl fl nl fl
Table 6: Multilingual learning results. The columnsare the average number of parameters per language(nl), average F1 per language (fl).
input-label versions of them (GILE-HAN andGILE-MHAN-Att) when increasing the numberof languages (1, 2, and 8) that are used for train-ing. Overall, we observe that the joint input-labelmodels outperform all the baselines independentlyof the number of languages involved in the train-ing, while having the same number of parameters.We also replicate the previous result that asecond language helps but beyond that there isno improvement.
Low-resource transfer. We investigate herewhether joint input-label models are useful forlow-resource languages. Table 7 shows the low-resource classification results from English toseven other languages when varying the amountof their training data. Our model with both sharedencoders and attention, GILE-MHAN, outper-forms previous models in average, namely, HAN
149
Levels [PB17] Oursrange HAN MHAN GILE-MHAN
en→
de 0.1-0.5% 29.9 39.4 42.91-5% 51.3 52.6 51.6
10-50% 63.5 63.8 65.9
en→
es 0.1-0.5% 39.5 41.5 39.01-5% 45.6 50.1 50.9
10-50% 74.2 75.2 76.4
en→
pt 0.1-0.5% 30.9 33.8 39.61-5% 44.6 47.3 48.9
10-50% 60.9 62.1 62.3
en→
uk 0.1-0.5% 60.4 60.9 61.11-5% 68.2 69.0 69.4
10-50% 76.4 76.7 76.5
en→
ru 0.1-0.5% 27.6 29.1 27.91-5% 39.3 40.2 40.2
10-50% 69.2 69.4 70.4
en→
ar 0.1-0.5% 35.4 36.6 46.11-5% 45.6 46.6 49.5
10-50% 48.9 47.8 61.8
en→
fa 0.1-0.5% 36.0 41.3 42.51-5% 55.0 55.5 55.4
10-50% 69.2 70.0 69.7
Table 7: Low-resource classification results withvarious sizes of training data using the general labels.
(Yang et al., 2016) and MHAN (Pappas andPopescu-Belis, 2017), for low-resource classifi-cation in the majority of the cases.
The shared input-label space appears to behelpful especially when transferring from Englishto German, Portuguese, and Arabic languages.GILE-MHAN is significantly behind MHAN ontransferring knowledge from English to Spanishand to Russian in the 0.1% to 0.5% resourcesetting, but in the rest of the cases they have verysimilar scores.
Label sampling. To speed up computation it ispossible to train our model by sampling labels,instead of training over the whole label set. Howmuch speed-up can we achieve from this labelsampling approach and still retain good levels ofperformance? In Figure 2, we attempt to answerthis question by reporting the performance of ourGILE-HNN model when varying the amount oflabels (%) that it uses for training over Englishgeneral and specific labels of the DW data set.In both cases, the performance of GILE-HNNtends to increase as the percentage of labelssampled increases, but it levels off for the higherpercentages.
Figure 2: Varying sampling percentage for general andspecific English labels. (Top) GILE-HNN is comparedagainst HNN in terms of F1 (%). (Bottom) The runtimespeed-up over GILE-HNN trained on the full label set.
For general labels, top performance is reachedwith a 40% to 50% sampling rate, which translatesto a 22% to 18% speed-up, whereas for the specificlabels, it is reached with a 60% to 70% samplingrate, which translates to a 40% to 36% speed-up.The speed-up is correlated to the size of the labelset, since there are many fewer general labelsthan specific labels, namely, 327 vs. 1,058 here.Hence, we expect even higher speedups for biggerlabel sets. Interestingly, GILE-HNN with labelsampling reaches the performance of the baselinewith a 25% and 60% sample for general andspecific labels respectively. This translates to aspeed-up of 30% and 50%, respectively, comparedwith a GILE-HNN trained over all labels. Overall,these results show that our model is effectiveand that it can also scale to large label sets.The label sampling should also be useful in taskswhere the computation resources may be limitedor budgeted.
5 Related Work
5.1 Neural text ClassificationResearch in neural text classification was initiallybased on feed-forward networks, which requiredunsupervised pre-training (Collobert et al., 2011;Mikolov et al., 2013; Le and Mikolov, 2014) andlater on they focused on networks with hierarchicalstructure. Kim (2014) proposed a convolutionalneural network (CNN) for sentence classification.Johnson and Zhang (2015) proposed a CNN forhigh-dimensional data classification, while Zhanget al. (2015) adopted a character-level CNNfor text classification. Lai et al. (2015) pro-posed a recurrent CNN to capture sequential
150
information, which outperformed simpler CNNs.Lin et al. (2015) and Tang et al. (2015) pro-posed hierarchical recurrent neural networksand showed that they were superior to CNN-based models. Yang et al. (2016) demonstratedthat a hierarchical attention network with bi-directional gated encoders outperforms previousalternatives. Pappas and Popescu-Belis (2017)adapted such networks to learn hierarchicaldocument structures with shared componentsacross different languages.
The issue of scaling to large label sets hasbeen addressed previously by output layer approx-imations (Morin and Bengio, 2005) and withthe use of sub-word units or character-levelmodeling (Sennrich et al., 2016; Lee et al.,2017) which is mainly applicable to structuredprediction problems. Despite the numerous stud-ies, most of the existing neural text classificationmodels ignore label descriptions and semantics.Moreover, they are based on typical output layerparametrizations that are dependent on the labelset size, and thus are not able to scale well tolarge label sets nor to generalize to unseen labels.Our output layer parametrization addresses theselimitations and could potentially improve suchmodels.
5.2 Output Representation LearningThere exist studies that aim to learn output rep-resentations directly from data without any seman-tic grounding to word embeddings (Srikumar andManning, 2014; Yeh et al., 2018; Augensteinet al., 2018). Such methods have a label-set-sizedependent parametrization, which makes themdata hungry, less scalable on large label sets, andincapable of generalizing to unseen classes. Wanget al. (2018) addressed the lack of semanticgrounding to word embeddings by proposing anefficient method based on label-attentive text rep-resentations which are helpful for text clas-sification. However, in contrast to our study, theirparametrization is still label-set-size dependentand thus their model is not able to scale well tolarge label sets nor to generalize to unseen labels.
5.3 Zero-shot Text ClassificationSeveral studies have focused on learning jointinput-label representations grounded to wordsemantics for unseen label prediction for images(Weston et al., 2011; Socher et al., 2013; Norouziet al., 2014; Zhang et al., 2016; Fu et al., 2018),
called zero-shot classification. However, there arefewer such studies for text classification. Dauphinet al. (2014) predicted semantic utterances of textby mapping them in the same semantic space withthe class labels using an unsupervised learn-ing objective. Yazdani and Henderson (2015) pro-posed a zero-shot spoken language understandingmodel based on a bilinear input-label model able togeneralize to previously unseen labels. Nam et al.(2016) proposed a bilinear joint document-labelembedding that learns shared word representationsbetween documents and labels. More recently, Shuet al. (2017) proposed an approach for open-worldclassification that aims to identify novel docu-ments during testing but it is not able to generalizeto unseen classes. Perhaps the model most similarto ours is from the recent study by Pappas et al.(2018) on neural machine translation, with thedifference that they have single-word label des-criptions and they use a label-set-dependent bias ina softmax linear prediction unit, which is designedfor structured prediction. Hence, their modelcan neither handle unseen labels nor multi-labelclassification, as we do here.
Compared with previous joint input-labelmodels, the proposed model has a more generaland flexible parametrization, which allows theoutput layer capacity to be controlled. Moreover,it is not restricted to linear mappings, which havelimited expressivity, but uses nonlinear mappings,similar to energy-based learning networks (LeCunet al., 2006; Belanger and McCallum, 2016). Thelink to the latter can be made if we regard P (ij)
val
in Equation (11) as an energy function for the i-thdocument and the j-th label, the calculation ofwhich uses a simple multiplicative transformation(Equation (10)). Lastly, the proposed modelperforms well on both seen and unseen label setsby leveraging the binary cross-entropy loss, whichis the standard loss for classification problems,instead of a ranking loss.
6 Conclusion
We proposed a novel joint input-label embeddingmodel for neural text classification that gener-alizes over existing input-label models and ad-dresses their limitations while preserving highperformance on both seen and unseen labels. Com-pared with baseline neural models with a typicaloutput layer, our model is more scalable and hasbetter performance on the seen labels. Compared
151
with previous joint input-label models, it performssignificantly better on unseen labels withoutcompromising performance on the seen labels.These improvements can be attributed to the theability of our model to capture complex input-label relationships, to its controllable capacity,and to its training objective, which is based oncross-entropy loss.
As future work, the label representation couldbe learned by a more sophisticated encoder, andthe label sampling could benefit from importancesampling to avoid revisiting uninformative labels.Another interesting direction would be to find amore scalable way of increasing the output layercapacity—for instance, using a deep rather than awide classification network. Moreover, adaptingthe proposed model to structured prediction, forinstance by using a softmax classification unitinstead of a sigmoid one, would benefit tasks suchas neural machine translation, language modeling,and summarization in isolation but also whentrained jointly with multi-task learning.
Acknowledgments
We are grateful for the support from the EuropeanUnion through its Horizon 2020 program inthe SUMMA project n. 688139, see http://www.summa-project.eu. We would also like to thank ouraction editor, Eneko Agirre, and the anonymousreviewers for their invaluable suggestions andfeedback.
References
Waleed Ammar, George Mulcaire, Yulia Tsvetkov,Guillaume Lample, Chris Dyer, and Noah A.Smith. 2016. Massively multilingual word em-beddings. CoRR, abs/1602.01925.v2.
Isabelle Augenstein, Sebastian Ruder, and AndersSøgaard. 2018. Multi-task learning of pairwisesequence classification tasks over disparatelabel spaces. In Proceedings of the 2018Conference of the North American Chapter ofthe Association for Computational Linguistics:Human Language Technologies, Volume 1(Long Papers), pages 1896–1906. New Orleans,Louisiana.
David Belanger and Andrew McCallum. 2016.Structured prediction energy networks. In Pro-ceedings of The 33rd International Conference on
Machine Learning, volume 48 of Proceedingsof Machine Learning Research, pages 983–992,New York, New York, USA. PMLR.
Jianshu Chen, Ji He, Yelong Shen, Lin Xiao,Xiaodong He, Jianfeng Gao, Xinying Song,and Li Deng. 2015. End-to-end learning of LDAby mirror-descent back propagation over a deeparchitecture. In Advances in Neural InformationProcessing Systems 28, pages 1765–1773.Montreal, Canada.
Kyunghyun Cho, Bart van Merrienboer, CaglarGulcehre, Dzmitry Bahdanau, Fethi Bougares,Holger Schwenk, and Yoshua Bengio. 2014.Learning phrase representations using RNNencoder–decoder for statistical machine trans-lation. In Proceedings of the 2014 Conferenceon Empirical Methods in Natural LanguageProcessing, pages 1724–1734, Doha, Qatar.
Ronan Collobert, Jason Weston, Leon Bottou,Michael Karlen, Koray Kavukcuoglu, and PavelKuksa. 2011. Natural language processing(almost) from scratch. Journal of MachineLearning Research, 12:2493–2537.
Yann N. Dauphin, Gokhan Tur, Dilek Hakkani-Tur, and Larry P. Heck. 2014. Zero-shotlearning and clustering for semantic utteranceclassification. In International Conference onLearning Representations. Banff, Canada.
Orhan Firat, Baskaran Sankaran, Yaser Al-Onaizan, Fatos T. Yarman Vural, andKyunghyun Cho. 2016. Zero-resource trans-lation with multi-lingual neural machine trans-lation. In Proceedings of the 2016 Conferenceon Empirical Methods in Natural LanguageProcessing, pages 268–277, Austin, USA.
Andrea Frome, Greg S. Corrado, Jon Shlens, SamyBengio, Jeff Dean, Marc Aurelio Ranzato,and Tomas Mikolov. 2013. DeViSE: A deepvisual-semantic embedding model, In C. J. C.Burges, L. Bottou, M. Welling, Z. Ghahramani,and K. Q. Weinberger, editors, Advances inNeural Information Processing Systems 26,pages 2121–2129. Curran Associates, Inc.
Yanwei Fu, Tao Xiang, Yu-Gang Jiang,Xiangyang Xue, Leonid Sigal, and ShaogangGong. 2018. Recent advances in zero-shot
recognition: Toward data-efficient understand-ing of visual content. IEEE Signal ProcessingMagazine, 35(1):112–125.
Rie Johnson and Tong Zhang. 2015. Effectiveuse of word order for text categorization withconvolutional neural networks. In Proceedingsof the 2015 Conference of the North AmericanChapter of the Association for ComputationalLinguistics: Human Language Technologies,pages 103–112, Denver, Colorado.
Yoon Kim. 2014. Convolutional neural networksfor sentence classification. In Proceedings ofthe 2014 Conference on Empirical Methods inNatural Language Processing, pages 1746–1751,Doha, Qatar.
Alexandre Klementiev, Ivan Titov, and BinodBhattarai. 2012. Inducing crosslingual dis-tributed representations of words. In Pro-ceedings of COLING 2012, pages 1459–1474,Mumbai, India.
Ankit Kumar, Ozan Irsoy, Jonathan Su, JamesBradbury, Robert English, Brian Pierce, PeterOndruska, Ishaan Gulrajani, and RichardSocher. 2015. Ask me anything: Dynamic mem-ory networks for natural language processing.In Proceedings of The 33rd International Con-ference on Machine Learning, pages 334–343,New York City, USA.
Siwei Lai, Liheng Xu, Kang Liu, and JunZhao. 2015. Recurrent convolutional neuralnetworks for text classification. In Proceedingsof the 29th AAAI Conference on ArtificialIntelligence, pages 2267–2273, Austin, USA.
Quoc V. Le and Tomas Mikolov. 2014. Distributedrepresentations of sentences and documents. InProceedings of The 31st International Confer-ence on Machine Learning, pages 1188–1196,Beijing, China.
Yann LeCun, Sumit Chopra, Raia Hadsell, Fu JieHuang, and et al. 2006. A tutorial on energy-based learning. In Predicting Structured Data.MIT Press.
Jason Lee, Kyunghyun Cho, and Thomas Hofmann.2017. Fully character-level neural machinetranslation without explicit segmentation. Tran-sactions of the Association for ComputationalLinguistics, 5:365–378.
Rui Lin, Shujie Liu, Muyun Yang, Mu Li, MingZhou, and Sheng Li. 2015. Hierarchical re-current neural network for document model-ing. In Proceedings of the 2015 Conferenceon Empirical Methods in Natural LanguageProcessing, pages 899–907. Lisbon, Portugal.
Thang Luong, Hieu Pham, and Christopher D.Manning. 2015. Effective approaches toattention-based neural machine translation. InProceedings of the 2015 Conference on Empir-ical Methods in Natural Language Processing,pages 1412–1421, Lisbon, Portugal.
Thomas Mensink, Jakob Verbeek, FlorentPerronnin, and Gabriela Csurka. 2012. Metriclearning for large scale image classification:Generalizing to new classes at near-zero cost. InComputer Vision – ECCV 2012, pages 488–501,Berlin, Heidelberg. Springer Berlin Heidelberg.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg SCorrado, and Jeff Dean. 2013. Distributedrepresentations of words and phrases andtheir compositionality. In C. J. C. Burges,L. Bottou, M. Welling, Z. Ghahramani,and K. Q. Weinberger, editors, Advances inNeural Information Processing Systems 26,pages 3111–3119. Curran Associates, Inc.
Frederic Morin and Yoshua Bengio. 2005. Hier-archical probabilistic neural network languagemodel. In Proceedings of the Tenth Interna-tional Workshop on Artificial Intelligence andStatistics, pages 246–252.
Khalil Mrini, Nikolaos Pappas, and AndreiPopescu-Belis. 2017. Cross-lingual transfer fornews article labeling: Benchmarking statisticaland neural models. In Idiap Research Report,Idiap-RR-26-2017.
Jinseok Nam, Eneldo Loza Mencıa, and JohannesFurnkranz. 2016. All-in text: Learning docu-ment, label, and word representations jointly.In Proceedings of the 13th AAAI Confer-ence on Artificial Intelligence, AAAI’16,pages 1948–1954, Phoenix, Arizona.
Mohammad Norouzi, Tomas Mikolov, SamyBengio, Yoram Singer, Jonathon Shlens, AndreaFrome, Greg Corrado, and Jeffrey Dean.2014. Zero-shot learning by convex combina-tion of semantic embeddings. In International
153
Conference on Learning Representations, Banff,Canada.
Bo Pang and Lillian Lee. 2005. Seeing stars:Exploiting class relationships for sentimentcategorization with respect to rating scales.In Proceedings of the 43rd Annual Meetingon Association for Computational Linguistics,pages 115–124, Ann Arbor, Michigan.
Nikolaos Pappas, Lesly Miculicich, and JamesHenderson. 2018. Beyond weight tying: Learn-ing joint input-output embeddings for neuralmachine translation. In Proceedings of the ThirdConference on Machine Translation: ResearchPapers, pages 73–83, Belgium, Brussels.Association for Computational Linguistics.
Nikolaos Pappas and Andrei Popescu-Belis. 2017.Multilingual hierarchical attention networksfor document classification. In Proceedings ofthe Eighth International Joint Conference onNatural Language Processing (Volume 1: LongPapers), pages 1015–1025.
Alexander M. Rush, Sumit Chopra, and JasonWeston. 2015. A neural attention model forabstractive sentence summarization. In Pro-ceedings of the 2015 Conference on EmpiricalMethods in Natural Language Processing,pages 379–389. Lisbon, Portugal.
Rico Sennrich, Barry Haddow, and AlexandraBirch. 2016. Neural machine translation of rarewords with subword units. In Proceedings ofthe 54th Annual Meeting of the Associationfor Computational Linguistics (Volume 1: LongPapers), pages 1715–1725, Berlin, Germany.
Lei Shu, Hu Xu, and Bing Liu. 2017. DOC:Deep open classification of text documents.In Proceedings of the 2017 Conference onEmpirical Methods in Natural LanguageProcessing, pages 2911–2916, Copenhagen,Denmark. Association for ComputationalLinguistics.
Richard Socher, Milind Ganjoo, Christopher D.Manning, and Andrew Y. Ng. 2013. Zero-shot learning through cross-modal transfer.In Proceedings of the 26th InternationalConference on Neural Information ProcessingSystems, NIPS’13, pages 935–943, Lake Tahoe,Nevada.
Vivek Srikumar and Christopher D. Manning.2014. Learning distributed representations forstructured output prediction. In Proceedings ofthe 27th International Conference on NeuralInformation Processing Systems - Volume 2,NIPS’14, pages 3266–3274, Cambridge, MA,USA. MIT Press.
Duyu Tang, Bing Qin, and Ting Liu. 2015.Document modeling with gated recurrentneural network for sentiment classification.In Proceedings of the 2015 Conference onEmpirical Methods in Natural Language Pro-cessing, pages 1422–1432, Lisbon, Portugal.Association for Computational Linguistics.
Guoyin Wang, Chunyuan Li, Wenlin Wang,Yizhe Zhang, Dinghan Shen, Xinyuan Zhang,Ricardo Henao, and Lawrence Carin. 2018.Joint embedding of words and labels fortext classification. In Proceedings of the56th Annual Meeting of the Association forComputational Linguistics (Volume 1: LongPapers), pages 2321–2331. Association forComputational Linguistics.
Jason Weston, Samy Bengio, and Nicolas Usunier.2010. Large scale image annotation: Learning torank with joint word-image embeddings. Mach.Learn., 81(1):21–35.
Jason Weston, Samy Bengio, and NicolasUsunier. 2011. WSABIE: Scaling up to largevocabulary image annotation. In Proceedingsof the Twenty-Second International Joint Con-ference on Artificial Intelligence (Volume 3),pages 2764–2770, Barcelona, Spain.
Zichao Yang, Diyi Yang, Chris Dyer, XiaodongHe, Alex Smola, and Eduard Hovy. 2016.Hierarchical attention networks for documentclassification. In Proceedings of the 2016 Con-ference of the North American Chapter of the-Association for Computational Linguistics:HumanLanguageTechnologies, pages1480–1489,San Diego, California.
Majid Yazdani and James Henderson. 2015. Amodel of zero-shot learning of spoken languageunderstanding. In Proceedings of the 2015Conference on Empirical Methods in NaturalLanguage Processing, pages 244–249, Lisbon,Portugal.
154
Chih-Kuan Yeh, Wei-Chieh Wu, Wei-Jen Ko, andYu-Chiang Frank Wang. 2018. Learning deeplatent spaces for multi-label classification. InProceedings of the 32nd AAAI Conference onArtificial Intelligence, New Orleans, USA.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.Character-level convolutional networks for text
classification. In Advances in Neural Infor-mation Processing Systems 28, pages 649–657,Montreal, Canada.
Yang Zhang, Boqing Gong, and Mubarak Shah.2016. Fast zero-shot image tagging. In Proceed-ings of the IEEE Conference on Computer Visionand Pattern Recognition, Las Vegas, USA.
