Proceedings of NAACL-HLT 2018, pages 852–860New Orleans, Louisiana, June 1 - 6, 2018. c©2018 Association for Computational Linguistics

Multimodal Named Entity Recognition for Short Social Media Posts

Seungwhan Moon1,2, Leonardo Neves2, Vitor Carvalho3

1 Language Technologies Institute, Carnegie Mellon University2 Snap Research

3 Intuitseungwhm@cs.cmu.edu, lneves@snap.com, vitor carvalho@intuit.com

AbstractWe introduce a new task called MultimodalNamed Entity Recognition (MNER) for noisyuser-generated data such as tweets or Snapchatcaptions, which comprise short text with ac-companying images. These social media postsoften come in inconsistent or incomplete syn-tax and lexical notations with very limitedsurrounding textual contexts, bringing signif-icant challenges for NER. To this end, we cre-ate a new dataset for MNER called SnapCap-tions (Snapchat image-caption pairs submittedto public and crowd-sourced stories with fullyannotated named entities). We then build uponthe state-of-the-art Bi-LSTM word/characterbased NER models with 1) a deep image net-work which incorporates relevant visual con-text to augment textual information, and 2)a generic modality-attention module whichlearns to attenuate irrelevant modalities whileamplifying the most informative ones to ex-tract contexts from, adaptive to each sam-ple and token. The proposed MNER modelwith modality attention significantly outper-forms the state-of-the-art text-only NER mod-els by successfully leveraging provided visualcontexts, opening up potential applications ofMNER on myriads of social media platforms.

1 Introduction

Social media with abundant user-generated postsprovide a rich platform for understanding events,opinions and preferences of groups and individ-uals. These insights are primarily hidden in un-structured forms of social media posts, such asin free-form text or images without tags. Namedentity recognition (NER), the task of recognizingnamed entities from free-form text, is thus a criti-cal step for building structural information, allow-ing for its use in personalized assistance, recom-mendations, advertisement, etc.

While many previous approaches (Lampleet al., 2016; Ma and Hovy, 2016; Chiu and

(a) (b)

Figure 1: Multimodal NER + modality attention.(a) Visual contexts help recognizing polysemous entitynames (‘Monopoly’ as in a board game versus an eco-nomics term). (b) Modality attention successfully sup-presses word embeddings of a unknown token (‘Marsh-melloooo’ with erroneously trailing ‘o’s), and focuseson character-based context (e.g. capitalized first letter,and lexical similarity to a known named entity (‘Marsh-mello’, a music producer)) for correct prediction.

Nichols, 2015; Huang et al., 2015; Lafferty et al.,2001) on NER have shown success for well-formed text in recognizing named entities viaword context resolution (e.g. LSTM with wordembeddings) combined with character-level fea-tures (e.g. CharLSTM/CNN), several additionalchallenges remain for recognizing named enti-ties from extremely short and coarse text foundin social media posts. For instance, short so-cial media posts often do not provide enough tex-tual contexts to resolve polysemous entities (e.g.“monopoly is da best ”, where ‘monopoly’ mayrefer to a board game (named entity) or a termin economics). In addition, noisy text includesa huge number of unknown tokens due to in-consistent lexical notations and frequent mentionsof various newly trending entities (e.g. “xoxoMarshmelloooo ”, where ‘Marshmelloooo’ is amis-spelling of a known entity ‘Marshmello’, a

852

music producer), making word embeddings basedneural networks NER models vulnerable.

To address the challenges above for social me-dia posts, we build upon the state-of-the-art neu-ral architecture for NER with the following twonovel approaches (Figure 1). First, we propose toleverage auxiliary modalities for additional con-text resolution of entities. For example, many pop-ular social media platforms now provide ways tocompose a post in multiple modalities - specifi-cally image and text (e.g. Snapchat captions, Twit-ter posts with image URLs), from which we canobtain additional context for understanding posts.While “monopoly” in the previous example is am-biguous in its textual form, an accompanying snapimage of a board game can help disambiguateamong polysemous entities, thereby correctly rec-ognizing it as a named entity.

Second, we also propose a general modal-ity attention module which chooses per decod-ing step the most informative modality amongavailable ones (in our case, word embeddings,character embeddings, or visual features) to ex-tract context from. For example, the modalityattention module lets the decoder attenuate theword-level signals for unknown word tokens (e.g.“Marshmellooooo” with trailing ‘o’s) and ampli-fies character-level features intsead (e.g. capital-ized first letter, lexical similarity to other knownnamed entity token ‘Marshmello’, etc.), therebysuppressing noise information (“UNK” token em-bedding) in decoding steps. Note that most ofthe previous literature in NER or other NLPtasks combine word and character-level informa-tion with naive concatenation, which is vulnerableto noisy social media posts. When an auxiliaryimage is available, the modality attention moduledetermines to amplify this visual context e.g. indisambiguating polysemous entities, or to atten-uate visual contexts when they are irrelevant totarget named entities, e.g. selfies, etc. Note thatthe proposed modality attention module is distinctfrom how attention is used in other sequence-to-sequence literature (e.g. attending to a specific to-ken within an input sequence). Section 2 providesthe detailed literature review.

Our contributions are three-fold: we propose(1) an LSTM-CNN hybrid multimodal NER net-work that takes as input both image and text forrecognition of a named entity in text input. To thebest of our knowledge, our approach is the first

work to incorporate visual contexts for named en-tity recognition tasks. (2) We propose a generalmodality attention module that selectively choosesmodalities to extract primary context from, max-imizing information gain and suppressing irrele-vant contexts from each modality (we treat words,characters, and images as separate modalities). (3)We show that the proposed approaches outperformthe state-of-the-art NER models (both with andwithout using additional visual contexts) on ournew MNER dataset SnapCaptions, a large collec-tion of informal and extremely short social mediaposts paired with unique images.

2 Related Work

Neural models for NER have been recently pro-posed, producing state-of-the-art performance onstandard NER tasks. For example, some of theend-to-end NER systems (Passos et al., 2014; Chiuand Nichols, 2015; Huang et al., 2015; Lampleet al., 2016; Ma and Hovy, 2016) use a recur-rent neural network usually with a CRF (Laf-ferty et al., 2001; McCallum and Li, 2003) forsequence labeling, accompanied with feature ex-tractors for words and characters (CNN, LSTMs,etc.), and achieve the state-of-the-art performancemostly without any use of gazetteers information.Note that most of these work aggregate textualcontexts via concatenation of word embeddingsand character embeddings. Recently, several workhave addressed the NER task specifically on noisyshort text segments such as Tweets, etc. (Baldwinet al., 2015; Aguilar et al., 2017). They report per-formance gains from leveraging external sourcesof information such as lexical information (e.g.POS tags, etc.) and/or from several preprocessingsteps (e.g. token substitution, etc.). Our modelbuilds upon these state-of-the-art neural modelsfor NER tasks, and improves the model in twocritical ways: (1) incorporation of visual contextsto provide auxiliary information for short mediaposts, and (2) addition of the modality attentionmodule, which better incorporates word embed-dings and character embeddings, especially whenthere are many missing tokens in the given wordembedding matrix. Note that we do not explorethe use of gazetteers information or other auxiliaryinformation (POS tags, etc.) (Ratinov and Roth,2009) as it is not the focus of our study.

Attention modules are widely applied in sev-eral deep learning tasks (Xu et al., 2015; Chan

853

et al., 2015; Sukhbaatar et al., 2015; Yao et al.,2015). For example, they use an attention mod-ule to attend to a subset within a single input (apart/region of an image, a specific token in an in-put sequence of tokens, etc.) at each decoding stepin an encoder-decoder framework for image cap-tioning tasks, etc. (Rei et al., 2016) explore var-ious attention mechanisms in NLP tasks, but donot incorporate visual components or investigatethe impact of such models on noisy social mediadata. (Moon and Carbonell, 2017) propose to useattention for a subset of discrete source samplesin transfer learning settings. Our modality atten-tion differs from the previous approaches in thatwe attenuate or amplifies each modality input asa whole among multiple available modalities, andthat we use the attention mechanism essentially tomap heterogeneous modalities in a single joint em-bedding space. Our approach also allows for re-use of the same model for predicting labels evenwhen some of the modalities are missing in input,as other modalities would still preserve the samesemantics in the embeddings space.

Multimodal learning is studied in various do-mains and applications, aimed at building a jointmodel that extracts contextual information frommultiple modalities (views) of parallel datasets.

The most relevant task to our multimodal NERsystem is the task of multimodal machine transla-tion (Elliott et al., 2015; Specia et al., 2016), whichaims at building a better machine translation sys-tem by taking as input a sentence in a source lan-guage as well as a corresponding image. Sev-eral standard sequence-to-sequence architecturesare explored (e.g. a target-language LSTM de-coder that takes as input an image first).

Other previous literature include study ofCanonical Correlation Analysis (CCA) (Dhillonet al., 2011) to learn feature correlations amongmultiple modalities, which is widely used in manyapplications. Other applications include imagecaptioning (Xu et al., 2015), audio-visual recogni-tion (Moon et al., 2015), visual question answer-ing systems (Antol et al., 2015), etc.

To the best of our knowledge, our approach isthe first work to incorporate visual contexts fornamed entity recognition tasks.

3 Proposed Methods

Figure 2 illustrates the proposed multimodal NER(MNER) model. First, we obtain word embed-

Figure 2: The main architecture for our multimodalNER (MNER) network with modality attention. Ateach decoding step, word embeddings, character em-beddings, and visual features are merged with modalityattention. Bi-LSTM/CRF takes as input each token andproduces an entity label.

dings, character embeddings, and visual features(Section 3.1). A Bi-LSTM-CRF model then takesas input a sequence of tokens, each of which com-prises a word token, a character sequence, and animage, in their respective representation (Section3.2). At each decoding step, representations fromeach modality are combined via the modality at-tention module to produce an entity label for eachtoken (3.3). We formulate each component of themodel in the following subsections.

Notations: Let x = {xt}Tt=1 a sequence of in-put tokens with length T , with a corresponding la-bel sequence y = {yt}Tt=1 indicating named en-tities (e.g. in standard BIO formats). Each in-put token is composed of three modalities: xt =

{x(w)t ,x

(c)t ,x

(v)t } for word embeddings, character

embeddings, and visual embeddings representa-tions, respectively.

3.1 Features

Similar to the state-of-the-art NER approaches(Lample et al., 2016; Ma and Hovy, 2016; Aguilaret al., 2017; Passos et al., 2014; Chiu and Nichols,2015; Huang et al., 2015), we use both word em-beddings and character embeddings.

Word embeddings are obtained from an unsu-pervised learning model that learns co-occurrencestatistics of words from a large external corpus,yielding word embeddings as distributional se-mantics (Mikolov et al., 2013). Specifically, weuse pre-trained embeddings from GloVE (Pen-nington et al., 2014).

854

Character embeddings are obtained from a Bi-LSTM which takes as input a sequence of char-acters of each token, similarly to (Lample et al.,2016). An alternative approach for obtaining char-acter embeddings is using a convolutional neuralnetwork as in (Ma and Hovy, 2016), but we findthat Bi-LSTM representation of characters yieldsempirically better results in our experiments.

Visual embeddings: To extract features froman image, we take the final hidden layer represen-tation of a modified version of the convolutionalnetwork model called Inception (GoogLeNet)(Szegedy et al., 2014, 2015) trained on the Ima-geNet dataset (Russakovsky et al., 2015) to clas-sify multiple objects in the scene. Our implemen-tation of the Inception model has deep 22 lay-ers, training of which is made possible via “net-work in network” principles and several dimen-sion reduction techniques to improve computingresource utilization. The final layer representa-tion encodes discriminative information describ-ing what objects are shown in an image, whichprovide auxiliary contexts for understanding tex-tual tokens and entities in accompanying captions.

Incorporating this visual information onto thetraditional NER system is an open challenge, andmultiple approaches can be considered. For in-stance, one may provide visual contexts only asan initial input to decoder as in some encoder-decoder image captioning systems (Vinyals et al.,2015). However, we empirically observe thatan NER decoder which takes as input the vi-sual embeddings at every decoding step (Section3.2), combined with the modality attention mod-ule (Section 3.3), yields better results.

Lastly, we add a transform layer foreach feature e.g. x

(w)t ,x

(c)t ,x

(v)t :=

σw(x(w)t ), σc(x

(c)t ), σv(x

(v)t ) before it is fed

to the NER entity LSTM.

3.2 Bi-LSTM + CRF for Multimodal NEROur MNER model is built on a Bi-LSTM and CRFhybrid model. We use the following implementa-tion for the entity Bi-LSTM.

it = σ(Wxiht−1 +Wcict−1)

ct = (1− it)� ct−1+ it � tanh(Wxcxt +Whcht−1)

ot = σ(Wxoxt +Whoht−1 +Wcoct)

ht = LSTM(xt) (1)

= ot � tanh(ct)

where xt is a weighted average of three modalitiesxt = {x(w)

t ;x(c)t ;x

(v)t } via the modality attention

module, which will be defined in Section 3.3. Biasterms for gates are omitted here for simplicity ofnotation.

We then obtain bi-directional entity token rep-resentations

←→ht = [

−→ht;←−ht] by concatenating its

left and right context representations. To enforcestructural correlations between labels in sequencedecoding,

←→ht is then passed to a conditional ran-

dom field (CRF) to produce a label for each tokenmaximizing the following objective.

y∗ = argmaxy

p(y|←→h ;WCRF) (2)

p(y|←→h ;WCRF) =

∏t ψt(yt−1,yt;

←→h )

∑y′∏

t ψt(y′t−1,y′t;←→h )

where ψt(y′,y′;

←→h ) is a potential function,

WCRF is a set of parameters that defines the po-tential functions and weight vectors for label pairs(y′,y′). Bias terms are omitted for brevity of for-mulation.

The model can be trained via log-likelihoodmaximization for the training set {(xi,yi)}:

L(WCRF) =∑

i

log p(y|←→h ;W) (3)

3.3 Modality AttentionThe modality attention module learns a unifiedrepresentation space for multiple available modal-ities (e.g. words, characters, images, etc.), andproduces a single vector representation with ag-gregated knowledge among multiple modalities,based on their weighted importance. We motivatethis module from the following observations.

A majority of the previous literature combinethe word and character-level contexts by sim-ply concatenating the word and character em-beddings at each decoding step, e.g. ht =

LSTM([x(w)t ;x

(c)t ]) in Eq.1. However, this naive

concatenation of two modalities (word and char-acters) results in inaccurate decoding, specificallyfor unknown word token embeddings (e.g. anall-zero vector x

(w)t = 0 or a random vector

x(w)t = ε ∼ U(−σ,+σ) is assigned for any un-

known token xt, thus ht = LSTM([0;x(c)t ]) or

LSTM([ε;x(c)t ])). While this concatenation ap-

proach does not cause significant errors for well-formatted text, we observe that it induces per-formance degradation for our social media post

855

datasets which contain a significant number ofmissing tokens.

Similarly, naive merging of textual and visualinformation (e.g. ht = LSTM([x

(w)t ;x

(c)t ;x

(v)t ]))

yields suboptimal results as each modality istreated equally informative, whereas in ourdatasets some of the images may contain irrele-vant contexts to textual modalities. Hence, ideallythere needs a mechanism in which the model caneffectively turn the switch on and off the modali-ties adaptive to each sample.

To this end, we propose a general modalityattention module, which adaptively attenuates oremphasizes each modality as a whole at each de-coding step t, and produces a soft-attended contextvector xt as an input token for the entity LSTM.

[a(w)t ,a

(c)t ,a

(v)t ] = σ

(Wm · [x(w)

t ;x(c)t ;x

(v)t ] + bm

)

α(m)t =

exp(a(m)t )

∑m′∈{w,c,v}

exp(a(m′)t )

∀m ∈ {w, c, v}

xt =∑

m∈{w,c,v}α(m)t x

(m)t (4)

where αt = [α(w)t ;α

(c)t ;α

(v)t ] ∈ R3 is an atten-

tion vector at each decoding step t, and xt is afinal context vector at t that maximizes informa-tion gain for xt. Note that the optimization of theobjective function (Eq.1) with modality attention(Eq.4) requires each modality to have the samedimension (e.g. x

(w)t ,x

(c)t ,x

(v)t ∈ Rp), and that

the transformation via Wm essentially enforceseach modality to be mapped into the same unifiedsubspace, where the weighted average of whichencodes discrimitive features for recognition ofnamed entities.

When visual context is not provided with eachtoken (as in the traditional NER task), we can de-fine the modality attention for word and characterembeddings only in a similar way:

[a(w)t ,a

(c)t ] = σ

(Wm · [x(w)

t ;x(c)t ] + bm

)(5)

α(m)t =

exp(a(m)t )

∑m′∈{w,c}

exp(a(m′)t )

∀m ∈ {w, c}

xt =∑

m∈{w,c}α(m)t x

(m)t

Note that while we apply this modality attentionmodule to the Bi-LSTM+CRF architecture (Sec-tion 3.2) for its empirical superiority, the module

itself is flexible and thus can work with other NERarchitectures or for other multimodal applications.

4 Empirical Evaluation

4.1 SnapCaptions DatasetThe SnapCaptions dataset is composed of 10Kuser-generated image (snap) and textual captionpairs where named entities in captions are man-ually labeled by expert human annotators (en-tity types: PER, LOC, ORG, MISC). These cap-tions are collected exclusively from snaps sub-mitted to public and crowd-sourced stories (akaSnapchat Live Stories or Our Stories). Examplesof such public crowd-sourced stories are “NewYork Story” or “Thanksgiving Story”, which com-prise snaps that are aggregated for various publicevents, venues, etc. All snaps were posted be-tween year 2016 and 2017, and do not containraw images or other associated information (onlytextual captions and obfuscated visual descriptorfeatures extracted from the pre-trained Inception-Net are available). We split the dataset into train(70%), validation (15%), and test sets (15%). Thecaptions data have average length of 30.7 char-acters (5.81 words) with vocabulary size 15,733,where 6,612 are considered unknown tokens fromStanford GloVE embeddings (Pennington et al.,2014). Named entities annotated in the Snap-Captions dataset include many of new and emerg-ing entities, and they are found in various sur-face forms (various nicknames, typos, etc.) To thebest of our knowledge, SnapCaptions is the onlydataset that contains natural image-caption pairswith expert-annotated named entities.

4.2 BaselinesTask: given a caption and a paired image (if used),the goal is to label every token in a caption in BIOscheme (B: beginning, I: inside, O: outside) (Sangand Veenstra, 1999). We report the performance ofthe following state-of-the-art NER models as base-lines, as well as several configurations of our pro-posed approach to examine contributions of eachcomponent (W: word, C: char, V: visual).

• Bi-LSTM/CRF (W only): only takes word to-ken embeddings (Stanford GloVE) as input.The rest of the architecture is kept the same.

• Bi-LSTM/CRF + Bi-CharLSTM (C only):only takes a character sequence of each wordtoken as input. (No word embeddings)

856

Modalities Model4 Entity Types (%) Segmentation (%)

Prec. Recall F1 Prec. Recall F1

C Bi-LSTM/CRF + Bi-CharLSTM 5.0 28.1 8.5 68.6 10.8 18.6W Bi-LSTM/CRF 38.2 53.3 44.6 82.5 50.1 62.4

W + C (Aguilar et al., 2017) 45.9 48.9 47.4 74.0 61.7 67.3W + C (Ma and Hovy, 2016) 46.0 51.9 48.7 76.8 61.0 68.0W + C (Lample et al., 2016) 47.7 49.9 48.8 74.4 63.3 68.4W + C Bi-LSTM/CRF + Bi-CharLSTM w/ Modality Attention 49.4 51.7 50.5 75.7 63.3 68.9

W + C + V Bi-LSTM/CRF + Bi-CharLSTM + Inception 50.5 52.3 51.4 71.9 66.5 69.1W + C + V Bi-LSTM/CRF + Bi-CharLSTM + Inception w/ Modality Attention 48.7 58.7 52.4 77.4 60.6 68.0

Table 1: NER performance on the SnapCaptions dataset with varying modalities (W: word, C: char, V: visual).We report precision, recall, and F1 score for both entity types recognition (PER, LOC, ORG, MISC) and entitysegmentation (untyped recognition - named entity or not) tasks.

• Bi-LSTM/CRF + Bi-CharLSTM (W+C)(Lample et al., 2016): takes as input bothword embeddings and character embed-dings extracted from a Bi-CharLSTM. EntityLSTM takes concatenated vectors of wordand character embeddings as input tokens.

• Bi-LSTM/CRF + CharCNN (W+C) (Ma andHovy, 2016): uses character embeddings ex-tracted from a CNN instead.

• Bi-LSTM/CRF + CharCNN (W+C) + Multi-task (Aguilar et al., 2017): trains the model toperform both recognition (into multiple entitytypes) as well as segmentation (binary) tasks.

• (proposed) Bi-LSTM/CRF + Bi-CharLSTMwith modality attention (W+C): uses themodality attention to merge word and char-acter embeddings.

• (proposed) Bi-LSTM/CRF + Bi-CharLSTM+ Inception (W+C+V): takes as input visualcontexts extracted from InceptionNet as well,concatenated with word and char vectors.

• (proposed) Bi-LSTM/CRF + Bi-CharLSTM+ Inception with modality attention(W+C+V): uses the modality attentionto merge word, character, and visualembeddings as input to entity LSTM.

4.3 Results: SnapCaptions Dataset

Table 1 shows the NER performance on the SnapCaptions dataset. We report both entity typesrecognition (PER, LOC, ORG, MISC) and namedentity segmentation (named entity or not) results.

Parameters: We tune the parameters of eachmodel with the following search space (bold in-dicate the choice for our final model): characterembeddings dimension: {25, 50, 100, 150, 200,300}, word embeddings size: {25, 50, 100, 150,200, 300}, LSTM hidden states: {25, 50, 100,150, 200, 300}, and x dimension: {25, 50, 100,150, 200, 300}. We optimize the parameters withAdagrad (Duchi et al., 2011) with batch size 10,learning rate 0.02, epsilon 10−8, and decay 0.0.

Main Results: When visual context is available(W+C+V), we see that the model performancegreatly improves over the textual models (W+C),showing that visual contexts are complimentaryto textual information in named entity recognitiontasks. In addition, it can be seen that the modalityattention module further improves the entity typerecognition performance for (W+C+V). This re-sult indicates that the modality attention is ableto focus on the most effective modality (visual,words, or characters) adaptive to each sample tomaximize information gain. Note that our text-only model (W+C) with the modality attentionmodule also significantly outperform the state-of-the-art baselines (Aguilar et al., 2017; Ma andHovy, 2016; Lample et al., 2016) that use the sametextual modalities (W+C), showing the effective-ness of the modality attention module for textualmodels as well.

Error Analysis: Table 2 shows example caseswhere incorporation of visual contexts affects pre-diction of named entities. For example, the token‘curry’ in the caption “The curry’s ” is poly-semous and may refer to either a type of food ora famous basketball player ‘Stephen Curry’, andthe surrounding textual contexts do not provide

857

Caption (target) Visual Tags GTPrediction

(W+C+V) (W+C)

+

“The curry’s ” parade, marching, urban area, ... B-PER B-PER O“Grandma w dat lit Apple Crisp” funnel cake, melting, frozen, ... O O B-ORG“Okay duke dumont ” DJ, guitarist, circus, ... B,I-PER B,I-PER O,O“CSI with my hubby” TV, movie, television, ... B-MISC B-MISC B-ORG“Twin day at angel stadium” stadium, arena, stampede, ... B,I-LOC B,I-LOC O,O“LETS GO CID” drum, DJ, drummer, ... B-PER B-PER O“MARSHMELLOOOOOOOOS” DJ, night, martini, ... B-PER B-PER O

-“Y’all come see me at bojangles. ” floor, tile, airport terminal, ... B-ORG O B-ORG“If u’re not watching this season of

monitor, suite, cubicle, ... B-MISC O B-MISCbachelorette ur doing LIFE WRONG”

Table 2: Error analysis: when do images help NER? Ground-truth labels (GT) and predictions of our model withvision input (W+C+V) and the one without (W+C) for the underlined named entities (or false positives) are shown.For interpretability, visual tags (label output of InceptionNet) are presented instead of actual feature vectors used.

enough information to disambiguate it. On theother hand, visual contexts (visual tags: ‘parade’,‘urban area’, ...) provide similarities to the token’sdistributional semantics from other training exam-ples (e.g. snaps from “NBA Championship Pa-rade Story”), and thus the model successfully pre-dicts the token as a named entity. Similarly, whilethe text-only model erroneously predicts ‘Apple’in the caption “Grandma w dat lit Apple Crisp”as an organization (e.g. Apple Inc.), the visualcontexts (describing objects related to food) helpdisambiguate the token, making the model pre-dict it correctly as a non-named entity (a fruit).Trending entities (musicians or DJs such as ‘CID’,‘Duke Dumont’, ‘Marshmello’, etc.) are also rec-ognized correctly with strengthened contexts fromvisual information (describing concert scenes) de-spite lack of surrounding textual contexts. A fewcases where visual contexts harmed the perfor-mance mostly include visual tags that are unre-lated to a token or its surrounding textual contexts.

Visualization of Modality Attention: Figure 3visualizes the modality attention module at eachdecoding step (each column), where amplifiedmodality is represented with darker color, and at-tenuated modality is represented with lighter color.

For the image-aided model (W+C+V; upper rowin Figure 3), we confirm that the modality at-tention successfully attenuates irrelevant signals(e.g. selfies, etc.) and amplifies relevant modality-based contexts in prediction of a given token.In the example of “disney word essential = cof-fee” with visual tags selfie, phone, person, themodality attention successfully attenuates distract-

ing visual signals and focuses on textual modali-ties, consequently making correct predictions. Thenamed entities in the examples of “Beautiful nightatop The Space Needle” and “Splash Mountain”are challenging to predict because they are com-posed of common nouns (space, needle, splash,mountain), and thus they often need additionalcontexts to correctly predict. In the training data,visual contexts make stronger indicators for thesenamed entities (space needle, splash mountain),and the modality attention module successfully at-tends more to stronger signals.

For text-only model (W+C), we observe thatperformance gains mostly come from the modal-ity attention module better handling tokens unseenduring training or unknown tokens from the pre-trained word embeddings matrix. For example,while WaRriOoOrs and Kooler Matic are missingtokens in the word embeddings matrix, it success-fully amplifies character-based contexts (e.g. cap-italized first letters, similarity to known entities‘Golden State Warriors’) and suppresses word-based contexts (word embeddings for unknown to-kens e.g. ‘WaRriOoOrs’), leading to correct pre-dictions. This result is significant because it showsperformance of the model, with an almost identi-cal architecture, can still improve without havingto scale the word embeddings matrix indefinitely.

Figure 3 (b) shows the cases where the modalityattention led to incorrect predictions. For example,the model predicts missing tokens HUUUGE andShampooer incorrectly as named entities by am-plifying misleading character-based contexts (e.g.capitalized first letters) or visual contexts (e.g.

858

Figure 3: Visualization of modality attention (a) successful cases and (b) unsuccessful ones from SnapCaptionstest data. For each decoding step of a token (column), the modality attention module amplifies the most relevantmodality (darker) while attenuating irrelevant modalities (lighter). The model makes final predictions based onweighted signals from all modalities. For interpretability, visual tags (label output of InceptionNet) are presentedinstead of actual feature vectors used. GT: ground-truth, Pred: prediction by our model. Modalities- W: words, C:characters, V: visual.

Vocab Size w/o M.A. w/ M.A.

100% 48.8 50.5

75% 48.7 50.150% 47.8 49.625% 46.4 48.7

Table 3: NER performance (F1) on SnapCaptions withvarying word embeddings vocabulary size. Mod-els being compared: (W+C) Bi-LSTM/CRF + Bi-CharLSTM w/ and w/o modality attention (M.A.)

concert scenes, associated contexts of which ofteninclude named entities in the training dataset).

Sensitivity to Word Embeddings VocabularySize: In order to isolate the effectiveness of themodality attention module on textual models inhandling missing tokens, we report the perfor-mance with varying word embeddings vocabularysizes in Table 3. By increasing the number ofmissing tokens artificially by randomly removingwords from the word embeddings matrix (originalvocab size: 400K), we observe that while the over-all performance degrades, the modality attentionmodule is able to suppress the peformance degra-dation. Note also that the performance gap gen-erally gets bigger as we decrease the vocabularysize of the word embeddings matrix. This result is

significant in that the modality attention is able toimprove the model more robust to missing tokenswithout having to train an indefinitely large wordembeddings matrix for arbitrarily noisy social me-dia text datasets.

5 Conclusions

We proposed a new multimodal NER (MNER: im-age + text) task on short social media posts. Wedemonstrated for the first time an effective MNERsystem, where visual information is combinedwith textual information to outperform traditionaltext-based NER baselines. Our work can be ap-plied to myriads of social media posts or other arti-cles across multiple platforms which often includeboth text and accompanying images. In addition,we proposed the modality attention module, a newneural mechanism which learns optimal integra-tion of different modes of correlated information.In essence, the modality attention learns to attenu-ate irrelevant or uninformative modal informationwhile amplifying the primary modality to extractbetter overall representations. We showed that themodality attention based model outperforms otherstate-of-the-art baselines when text was the onlymodality available, by better combining word andcharacter level information.

859

ReferencesGustavo Aguilar, Suraj Maharjan, A. Pastor Lopez-

Monroy, and Thamar Solorio. 2017. A multi-taskapproach for named entity recognition in social me-dia data. ACL WNUT Workshop .

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar-garet Mitchell, Dhruv Batra, C Lawrence Zitnick,and Devi Parikh. 2015. Vqa: Visual question an-swering. In ICCV .

Timothy Baldwin, Marie-Catherine de Marneffe,Bo Han, Young-Bum Kim, Alan Ritter, and WeiXu. 2015. Shared tasks of the 2015 workshop onnoisy user-generated text: Twitter lexical normaliza-tion and named entity recognition. In Proceedingsof the Workshop on Noisy User-generated Text.

William Chan, Navdeep Jaitly, Quoc V Le, andOriol Vinyals. 2015. Listen, attend and spell.arXiv:1508.01211 .

Jason PC Chiu and Eric Nichols. 2015. Namedentity recognition with bidirectional lstm-cnns.arXiv:1511.08308 .

Paramveer Dhillon, Dean P Foster, and Lyle H Ungar.2011. Multi-view learning of word embeddings viacca. In NIPS. pages 199–207.

John Duchi, Elad Hazan, and Yoram Singer. 2011.Adaptive subgradient methods for online learningand stochastic optimization. JMLR .

Desmond Elliott, Stella Frank, and Eva Hasler. 2015.Multi-language image description with neural se-quence models. CoRR, abs/1510.04709 .

Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidi-rectional lstm-crf models for sequence tagging.arXiv:1508.01991 .

John Lafferty, Andrew McCallum, and Fernando CNPereira. 2001. Conditional random fields: Prob-abilistic models for segmenting and labeling se-quence data .

Guillaume Lample, Miguel Ballesteros, Sandeep Sub-ramanian, Kazuya Kawakami, and Chris Dyer. 2016.Neural architectures for named entity recognition.NAACL .

Xuezhe Ma and Eduard Hovy. 2016. End-to-endsequence labeling via bi-directional lstm-cnns-crf.arXiv preprint arXiv:1603.01354 .

Andrew McCallum and Wei Li. 2003. Early results fornamed entity recognition with conditional randomfields, feature induction and web-enhanced lexicons.In NAACL.

Tomas Mikolov, Kai Chen, Greg Corrado, and JeffreyDean. 2013. Efficient estimation of word represen-tations in vector space. ICLR .

Seungwhan Moon and Jaime Carbonell. 2017. Com-pletely heterogeneous transfer learning with atten-tion: What and what not to transfer. IJCAI .

Seungwhan Moon, Suyoun Kim, and Haohan Wang.2015. Multimodal transfer deep learning with appli-cations in audio-visual recognition. In NIPS MMMLWorkshop.

Alexandre Passos, Vineet Kumar, and Andrew McCal-lum. 2014. Lexicon infused phrase embeddings fornamed entity resolution. aarXiv:1404.5367 .

Jeffrey Pennington, Richard Socher, and Christo-pher D. Manning. 2014. Glove: Global vectors forword representation. In EMNLP.

Lev Ratinov and Dan Roth. 2009. Design challengesand misconceptions in named entity recognition. InCoNLL.

Marek Rei, Gamal KO Crichton, and Sampo Pyysalo.2016. Attending to characters in neural sequencelabeling models. COLING .

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause,Sanjeev Satheesh, Sean Ma, Zhiheng Huang, An-drej Karpathy, Aditya Khosla, Michael Bernstein,Alexander C. Berg, and Li Fei-Fei. 2015. ImageNetLarge Scale Visual Recognition Challenge. IJCV .

Erik F Sang and Jorn Veenstra. 1999. Representing textchunks. In EACL.

Lucia Specia, Stella Frank, Khalil Sima’an, andDesmond Elliott. 2016. A shared task on multi-modal machine translation and crosslingual imagedescription. In WMT .

Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al.2015. End-to-end memory networks. In NIPS.

C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. E. Reed,D. Anguelov, D. Erhan, V. Vanhoucke, and A. Ra-binovich. 2014. Going deeper with convolutions.CVPR .

Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe,Jonathon Shlens, and Zbigniew Wojna. 2015. Re-thinking the inception architecture for computer vi-sion. CoRR .

Oriol Vinyals, Alexander Toshev, Samy Bengio, andDumitru Erhan. 2015. Show and tell: A neural im-age caption generator. In CVPR.

Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho,Aaron Courville, Ruslan Salakhutdinov, Richard SZemel, and Yoshua Bengio. 2015. Show, attend andtell: Neural image caption generation with visual at-tention. ICML .

Li Yao, Atousa Torabi, Kyunghyun Cho, Nicolas Bal-las, Christopher Pal, Hugo Larochelle, and AaronCourville. 2015. Describing videos by exploitingtemporal structure. In ICCV .

860