Single-/Multi-Source Cross-Lingual NER via Teacher-Student Learningon Unlabeled Data in Target Language

Qianhui Wu1, Zijia Lin2, Borje F. Karlsson2, Jian-Guang Lou2, and Biqing Huang1

1Beijing National Research Center for Information Science and Technology (BNRist)Department of Automation, Tsinghua University, Beijing 100084, Chinawuqianhui@tsinghua.org.cn, hbq@tsinghua.edu.cn

2Microsoft Research, Beijing 100080, China{zijlin,borje.karlsson,jlou}@microsoft.com

Abstract

To better tackle the named entity recognition(NER) problem on languages with little/no la-beled data, cross-lingual NER must effectivelyleverage knowledge learned from source lan-guages with rich labeled data. Previous workson cross-lingual NER are mostly based on la-bel projection with pairwise texts or directmodel transfer. However, such methods ei-ther are not applicable if the labeled data inthe source languages is unavailable, or do notleverage information contained in unlabeleddata in the target language. In this paper, wepropose a teacher-student learning method toaddress such limitations, where NER modelsin the source languages are used as teachers totrain a student model on unlabeled data in thetarget language. The proposed method worksfor both single-source and multi-source cross-lingual NER. For the latter, we further proposea similarity measuring method to better weightthe supervision from different teacher models.Extensive experiments for 3 target languageson benchmark datasets well demonstrate thatour method outperforms existing state-of-the-art methods for both single-source and multi-source cross-lingual NER.

1 Introduction

Named entity recognition (NER) is the task ofidentifying text spans that belong to pre-definedcategories, like locations, person names, etc. It’sa fundamental component in many downstreamtasks, and has been greatly advanced by deep neuralnetworks (Lample et al., 2016; Chiu and Nichols,2016; Peters et al., 2017). However, these ap-proaches generally require massive manually la-beled data, which prohibits their adaptation to low-resource languages due to high annotation costs.

One solution to tackle that is to transfer knowl-edge from a source language with rich labeled datato a target language with little or even no labeled

ℳ𝑠𝑟𝑐

ℳ𝑡𝑔𝑡

Directly apply(i.e., ℳ𝑡𝑔𝑡 = ℳ𝑠𝑟𝑐)

{𝑋, 𝑌}𝑠𝑟𝑐 {𝑋′}𝑡𝑔𝑡

ℳ𝑡𝑔𝑡

PairwiseRelation

ℳ𝑠𝑟𝑐 {𝑋′}𝑡𝑔𝑡

ℳ𝑡𝑔𝑡

(b)(a) (c)

{𝑋′, 𝑌′}𝑡𝑔𝑡 {𝑋′, 𝑃′}𝑡𝑔𝑡

Training Training

Figure 1: Comparison between previous cross-lingualNER methods (a/b) and the proposed method (c).(a): direct model transfer; (b): label projection withpairwise texts; (c): proposed teacher-student learn-ing method. Msrc/tgt: learned NER model forsource/target language; {X,Y }src: labeled data insource language; {X ′}tgt: unlabeled data in target lan-guage; {X ′, Y ′}tgt/{X ′, P ′}tgt: pseudo-labeled datain target language with hard labels / soft labels.

data, which is referred to as cross-lingual NER (Wuand Dredze, 2019; Wu et al., 2020). In this paper,following Wu and Dredze (2019) and Wu et al.(2020), we focus on the extreme scenario of cross-lingual NER where no labeled data is available inthe target language, which is challenging in itselfand has attracted considerable attention from theresearch community in recent years.

Previous works on cross-lingual NER are mostlybased on label projection with pairwise texts ordirect model transfer. Label-projection based meth-ods focus on using labeled data in a source lan-guage to generate pseudo-labelled data in the targetlanguage for training an NER model. For example,Ni et al. (2017) creates automatically labeled NERdata for the target language via label projectionon comparable corpora and develops a heuristicscheme to select good-quality projection-labeleddata. Mayhew et al. (2017) and Xie et al. (2018)

arX

iv:2

004.

1244

0v2

[cs

.CL

] 1

5 Ju

l 202

0

translate the source language labeled data at thephrase/word level to generate pairwise labeled datafor the target language. Differently, model-transferbased methods (Wu and Dredze, 2019; Wu et al.,2020) focus on training a shared NER model on thelabeled data in the source language with language-independent features, such as cross-lingual wordrepresentations (Devlin et al., 2019), and then di-rectly testing the model on the target language.

However, there are limitations in both label-projection based methods and model-transfer basedmethods. The former relies on labeled data in thesource language for label projection, and thus isnot applicable in cases where the required labeleddata is inaccessible (e.g., due to privacy/sensitivityissues). Meanwhile, the later does not leverage un-labeled data in the target language, which can bemuch cheaper to obtain and probably contains veryuseful language information.

In this paper, we propose a teacher-student learn-ing method for cross-lingual NER to address thementioned limitations. Specifically, we leveragemultilingual BERT (Devlin et al., 2019) as the basemodel to produce language-independent features.A previously trained NER model for the sourcelanguage is then used as a teacher model to pre-dict the probability distribution of entity labels (i.e.,soft labels) for each token in the non-pairwise unla-beled data in the target language. Finally, we traina student NER model for the target language usingthe pseudo-labeled data with such soft labels. Theproposed method does not rely on labelled datain the source language, and it also leverages theavailable information from unlabeled data in thetarget language, thus avoiding the mentioned lim-itations of previous works. Note that we use theteacher model to predict soft labels rather than hardlabels (i.e., one-hot labelling vector), as soft labelscan provide much more information (Hinton et al.,2015) for the student model. Figure 1 shows thedifferences between the proposed teacher-studentlearning method and the typical label-projection ormodel-transfer based methods.

We further extend our teacher-student learningmethod to multi-source cross-lingual NER, con-sidering that there are usually multiple source lan-guages available in practice and we would prefertransferring knowledge from all source languagesrather than a single one. In this case, our methodstill enjoys the same advantages in terms of dataavailability and inference efficiency, compared with

existing works (Tackstrom, 2012; Chen et al., 2019;Enghoff et al., 2018; Rahimi et al., 2019). More-over, we propose a method to measure the similar-ity between each source language and the target lan-guage, and use this similarity to better weight thesupervision from the corresponding teacher model.

We evaluate our proposed method for 3 tar-get languages on benchmark datasets, using dif-ferent source language settings. Experimental re-sults show that our method outperforms existingstate-of-the-art methods for both single-source andmulti-source cross-lingual NER. We also conductcase studies and statistical analyses to discuss whyteacher-student learning reaches better results.

The main contributions of this work are:

• We propose a teacher-student learning methodfor single-source cross-lingual NER, whichaddresses limitations of previous works w.r.tdata availability and usage of unlabeled data.

• We extend the proposed method to multi-source cross-lingual NER, using a measureof the similarities between source/target lan-guages to better weight teacher models.

• We conduct extensive experiments validatingthe effectiveness and reasonableness of theproposed methods, and further analyse whythey attain superior performance.

2 Related Work

Single-Source Cross-Lingual NER: Such ap-proaches consider one single source language forknowledge transfer. Previous works can be dividedinto two categories: label-projection and model-transfer based methods.

Label-projection based methods aim to buildpseudo-labeled data for the target language to trainan NER model. Some early works proposed to usebilingual parallel corpora and project model expec-tations (Wang and Manning, 2014) or labels (Niet al., 2017) from the source language to the targetlanguage with external word alignment informa-tion. But obtaining parallel corpora is expensive oreven infeasible. To tackle that, recent methods pro-posed to firstly translate source-language labeleddata at the phrase level (Mayhew et al., 2017) orword level (Xie et al., 2018), and then directly copylabels across languages. But translation introducesextra noise due to sense ambiguity and word or-der differences between languages, thus hurting thetrained model.

Model-transfer based methods generally relyon language-independent features (e.g., cross-lingual word embeddings (Ni et al., 2017; Huanget al., 2019; Wu and Dredze, 2019; Moon et al.,2019), word clusters (Tackstrom et al., 2012),gazetteers (Zirikly and Hagiwara, 2015), and wik-ifier features (Tsai et al., 2016)), so that a modeltrained with such features can be directly appliedto the target language. For further improvement,Wu et al. (2020) proposed constructing a pseudo-training set for each test case and fine-tuning themodel before inference. However, these methodsdo not leverage any unlabeled data in the targetlanguage, though such data can be easy to obtainand benefit the language/domain adaptation.

Multi-Source Cross-Lingual NER: Multi-source cross-lingual NER considers multiplesource languages for knowledge transfer.

Tackstrom (2012) and Moon et al. (2019) con-catenated the labeled data of all source languages totrain a unified model, and performed cross-lingualNER in a direct model transfer manner. Chenet al. (2019) leveraged adversarial networks tolearn language-independent features, and learns amixture-of-experts model (Shazeer et al., 2017) toweight source models at the token level. However,both methods straightly rely on the availability oflabeled data in the source languages.

Differently, Enghoff et al. (2018) implementedmulti-source label projection and studied howsource data quality influence performance. Rahimiet al. (2019) applied truth inference to modelthe transfer annotation bias from multiple source-language models. However, both methods makepredictions via an ensemble of source-languagemodels, which is cumbersome and computation-ally expensive, especially when a source-languagemodel has massive parameter space.

Teacher-Student Learning: Early applicationsof teacher-student learning targeted model com-pression (Bucilu et al., 2006), where a smallstudent model is trained to mimic a pre-trained,larger teacher model or ensemble of models. Itwas soon applied to various tasks like imageclassification (Hinton et al., 2015; You et al., 2017),dialogue generation (Peng et al., 2019), and neuralmachine translation (Tan et al., 2019), whichdemonstrated the usefulness of the knowledgetransfer approach.

Encoder Layer

Linear Classification Layer

Student

Loss Function

GradientBack-Propagation

Encoder Layer

Linear Classification Layer

Teacher

Inference Training

UnlabeledTarget-Language Data

Figure 2: Framework of the proposed teacher-studentlearning method for single-source cross-lingual NER.

In this paper, we investigate teacher-studentlearning for the task of cross-lingual NER, in bothsingle-source and multi-source scenarios. Differentfrom previous works, our proposed method doesnot rely on the availability of labelled data insource languages or any pairwise texts, while it canalso leverage extra information in unlabeled datain the target language to enhance the cross-lingualtransfer. Moreover, compared with using anensemble of source-language models, our methoduses a single student model for inference, whichcan enjoy higher efficiency.

3 Methodology

Named entity recognition can be formulated as asequence labeling problem, i.e., given a sentencex = {xi}Li=1 with L tokens, an NER model issupposed to infer the entity label yi for each to-ken xi and output a label sequence y = {yi}Li=1.Under the paradigm of cross-lingual NER, we as-sume there are K source-language models previ-ously trained with language-independent features.Our proposed teacher-student learning method thenuses those K source-language models as teachersto train an effective student NER model for thetarget language on its unlabeled data Dtgt.

3.1 Single-Source Cross-Lingual NER

Here we firstly consider the case of only one sourcelanguage (K = 1) for cross-lingual NER. Theoverall framework of the proposed teacher-studentlearning method for single-source cross-lingualNER is illustrated in Figure 2.

3.1.1 NER Model StructureAs shown in Figure 2, for simplicity, we employthe same neural network structure for both teacher(source-language) and student (target-language)NER models. Note that the student model is flexi-ble and its structure can be determined accordingto the trade-off between performance and train-ing/inference efficiency.

Here the adopted NER model consists of an en-coder layer and a linear classification layer. Specifi-cally, given an input sequence x = {xi}Li=1 with Ltokens, the encoder layer fθ maps it into a sequenceof hidden vectors h = {hi}Li=1:

h = fθ(x) (1)

Here fθ(·) can be any encoder model that producescross-lingual token representations, and hi is thehidden vector corresponding to the i-th token xi.

With each hi derived, the linear classificationlayer computes the probability distribution of en-tity labels for the corresponding token xi, using asoftmax function:

p(xi,Θ) = softmax(Whi + b) (2)

where p(xi,Θ) ∈ R|C| with C being the entitylabel set, and Θ = {fθ,W, b} denotes the to-be-learned model parameters.

3.1.2 Teacher-Student LearningTraining: We train the student model to mimicthe output probability distribution of entity labelsby the teacher model, on the unlabeled data in thetarget language Dtgt. Knowledge from the teachermodel is expected to transfer to the student model,while the student model can also leverage help-ful language-specific information available in theunlabeled target-language data.

Given an unlabeled sentence x′ ∈ Dtgt in thetarget language, the teacher-student learning lossw.r.t x′ is formulated as the mean squared error(MSE) between the output probability distributionsof entity labels by the student model and those bythe teacher model, averaged over tokens. Note thathere we follow Yang et al. (2019) and use the MSEloss, because it is symmetric and mimics all prob-abilities equally. Suppose that for the i-token inx′, i.e., x′i, the probability distribution of entitylabels output by the student model is denoted asp(x′i,ΘS), and that output by the teacher model asp(x′i,ΘT ). Here ΘS and ΘT , respectively, denote

GradientBack-Propagation

Inference Training

UnlabeledTarget-Language Data

Student Θ𝑆

Loss Function

Teacher

Θ𝑇(𝐾)

. . .

⨀𝛼𝐾

Teacher

Θ𝑇(1)

⨀𝛼1

⨁

Figure 3: Framework of the proposed teacher-studentlearning method for multi-source cross-lingual NER.

the parameters of the student and the teacher mod-els. The teacher-student learning loss w.r.t x′ isthen defined as:

L(x′,ΘS) =1

L

L∑i=1

MSE(p(x′i,ΘS), p(x′i,ΘT )

)(3)

And the whole training loss is the summation oflosses w.r.t all sentences in Dtgt, as defined below.

L(ΘS) =∑

x′∈Dtgt

L(x′,ΘS) (4)

Minimizing L(ΘS) will derive the student model.

Inference: For inference in the target language,we only utilize the learned student model to predictthe probability distribution of entity labels for eachtoken xi in a test sentence x. Then we take theentity label c ∈ C with the highest probability asthe predicted label yi for xi:

yi = arg maxcp(xi,ΘS)c (5)

where p(xi,ΘS)c denotes the predicted probabilitycorresponding to the entity label c in p(xi,ΘS).

3.2 Multi-Source Cross-Lingual NERThe framework of the proposed teacher-studentlearning method for multi-source (K > 1) cross-lingual NER is illustrated in Figure 3.

3.2.1 Extension to Multiple Teacher ModelsAs illustrated in Figure 3, we extend the single-teacher framework in Figure 2 into a multi-teacherone, while keeping the student model unchanged.

Note that, for simplicity, all teacher models andthe student model use the same model structure as

3.1.1. Take the k-th teacher model for example, anddenote its parameters as Θ

(k)T . Given a sentence

x′ = {x′i}Li=1 with L tokens from the unlabeleddata Dtgt in the target language, the output proba-bility distribution of entity labels w.r.t the i-th tokenxi can be derived as Eq. 1 and 2, which is denotedas p(x′i,Θ

(k)T ). To combine all teacher models, we

add up their output probability distributions with agroup of weights {αk}Kk=1 as follows.

p(x′i,ΘT ) =K∑k=1

αk · p(x′i,Θ(k)T ) (6)

where p(x′i,ΘT ) is the combined probability dis-tribution of entity labels, ΘT = {Θ(k)

T }Kk=1 is theset of parameters of all teacher models, and αk isthe weight corresponding to the k-th teacher model,with

∑Kk=1 αk = 1 and αk ≥ 0,∀k ∈ {1, . . . ,K}.

3.2.2 Weighting Teacher ModelsHere we elaborate on how to derive the weights{αk}Kk=1 in cases w/ or w/o unlabeled data in thesource languages. Source languages more similarto the target language should generally be assignedhigher weights to transfer more knowledge.

Without Any Source-Language Data: It isstraightforward to average over all teacher mod-els:

αk =1

K, ∀k ∈ {1, 2, . . . ,K} (7)

With Unlabeled Source-Language Data: Asno labeled data is available, existing supervised lan-guage/domain similarity learning methods for a tar-get task (i.e., NER) (McClosky et al., 2010) are notapplicable here. Inspired by Pinheiro (2018), wepropose to introduce a language identification auxil-iary task for calculating similarities between sourceand target languages, and then weight teacher mod-els based on this metric.

In the language identification task, for the k-th source language, each unlabeled sentence u(k)

in it is associated with the language index k tobuild its training dataset, denoted as D

(k)src =

{(u(k), k)}. We also assume that in the m-dimensional language-independent feature space,sentences from each source language should beclustered around the corresponding language em-bedding vector. We thus introduce a learnable lan-guage embedding vector µ(k) ∈ Rm for the k-thsource language, and then utilize a bilinear opera-tor to measure similarity between a given sentence

u and the k-th source language:

s(u, µ(k)) = gT (u)Mµ(k) (8)

where g(·) can be any language-independent modelthat outputs sentence embeddings, and M ∈Rm×m denotes the parameters of the bilinear oper-ator.

By building a language embedding matrix P ∈Rm×K with each µ(k) column by column, and ap-plying a softmax function over the bilinear oper-ator, we can derive language-specific probabilitydistributions w.r.t u as below.

q(u,M, P ) = softmax(gT (u)MP

)(9)

Then the parameters M and P are trained to iden-tify the language of each sentence in {D(k)

src}Kk=1,via minimizing the cross-entropy (CE) loss:

L(P,M) =− 1

Z

∑(u(k),k)∈Dsrc

CE(q(u(k),M, P ), k

)+ γ‖PP T − I‖2F

(10)where Dsrc is the union set of {D(k)

src}Kk=1, Z =|Dsrc|, ‖ · ‖2F denotes the squared Frobenius norm,and I is an identity matrix. The regularizer inL(P,M) is to encourage different dimensions ofthe language embedding vectors to focus on differ-ent aspects, with γ ≥ 0 being its weighting factor.

With learned M and P = [µ(1), µ(2), . . . , µ(K)],we compute the weights {αk}Ki=1 using the unla-beled data in the target language Dtgt:

αk =1

|Dtgt|∑

x′∈Dtgt

exp(s(x′, µ(k))/τ

)∑Ki=1 exp

(s(x′, µ(i))/τ

)(11)

where τ is a temperature factor to smooth theoutput probability distribution. In our experi-ments, we set it as the variance of all values in{s(x′, µ(k))},∀x′ ∈ Dtgt,∀k ∈ {1, ...,K}, sothat αk would not be too biased to either 0 or 1.

3.2.3 Teacher-Student LearningTraining: With the combined probability distri-bution of entity labels from multiple teacher mod-els, i.e., p(x′i,ΘT ) in Eq. 6, the training loss for thestudent model is identical to Eq. 3 and 4.

Inference: For inference on the target language,we only use the learned student model and makepredictions as in the single-source scenario (Eq. 5).

Language Type Train Dev TestEnglish-en Sentence 14,987 3,466 3,684

(CoNLL-2003) Entity 23,499 5,942 5,648German-de Sentence 12,705 3,068 3,160

(CoNLL-2003) Entity 11,851 4,833 3,673Spanish-es Sentence 8,323 1,915 1,517

(CoNLL-2002) Entity 18,798 4,351 3,558Dutch-nl Sentence 15,806 2,895 5,195

(CoNLL-2002) Entity 13,344 2,616 3,941

Table 1: Statistics of the benchmark datasets.

4 Experiments

We conduct extensive experiments for 3 target lan-guages (i.e., Spanish, Dutch, and German) on stan-dard benchmark datasets, to validate the effective-ness and reasonableness of our proposed methodfor single- and multi-source cross lingual NER.

4.1 Settings

Datasets We use two NER benchmark datasets:CoNLL-2002 (Spanish and Dutch) (TjongKim Sang, 2002); CoNLL-2003 (English andGerman) (Tjong Kim Sang and De Meulder, 2003).Both are annotated with 4 entity types: PER, LOC,ORG, and MISC. Each language-specific datasetis split into training, development, and test sets.Table 1 reports the dataset statistics. All sentencesare tokenized into sequences of subwords withWordPiece (Wu et al., 2016). Following Wuand Dredze (2019), we also use the BIO entitylabelling scheme.

In our experiments, for each source language,an NER model is trained previously with its cor-responding labeled training set. As for the targetlanguage, we discard the entity labels from its train-ing set, and use it as unlabeled target-language dataDtgt. Similarly, unlabeled source-language data forlearning language similarities (Eq. 10) is simulatedvia discarding the entity labels of each training set.

Network Configurations We leverage the casedmultilingual BERTBASE (Wu and Dredze, 2019) forboth f(·) in Eq. 1 and g(·) in Eq. 8, with 12 Trans-former blocks, 768 hidden units, 12 self-attentionhead, GELU activations (Hendrycks and Gimpel,2016), and learned positional embeddings. We usethe final hidden vector of the first [CLS] token asthe sentence embedding for g(·), and use the meanvalue of sentence embeddings w.r.t the k-th sourcelanguage to initialize µ(k) in Eq. 8.

es nl deTackstrom et al. (2012) 59.30 58.40 40.40

Tsai et al. (2016) 60.55 61.56 48.12Ni et al. (2017) 65.10 65.40 58.50

Mayhew et al. (2017) 65.95 66.50 59.11Xie et al. (2018) 72.37 71.25 57.76

Wu and Dredze (2019)† 74.50 79.50 71.10Moon et al. (2019)† 75.67 80.38 71.42

Wu et al. (2020) 76.75 80.44 73.16Ours 76.94 80.89 73.22

Table 2: Performance comparisons of single-sourcecross-lingual NER. † denotes the reported results w.r.t.freezing the bottom three layers of BERTBASE as in thispaper.

Network Training We implement our proposedmethod based on huggingface Transformers1. Fol-lowing Wolf et al. (2019), we use a batch size of 32,and 3 training epochs to ensure convergence of op-timization. Following Wu and Dredze (2019), wefreeze the parameters of the embedding layer andthe bottom three layers of BERTBASE. For the op-timizers, we use AdamW (Loshchilov and Hutter,2017) with learning rate of 5e− 5 for teacher mod-els (Wolf et al., 2019), and 1e− 4 for the studentmodel (Yang et al., 2019) to converge faster. Asfor language similarity measuring (i.e., Eq. 10), weset γ = 0.01 following Pinheiro (2018). Besides,we use a low-rank approximation for the bilinearoperator M , i.e., M = UTV where U, V ∈ Rd×mwith d� m, and we empirically set d = 64.

Performance Metric We use phrase level F1-score as the evaluation metric, following TjongKim Sang (2002). For each experiment, we con-duct 5 runs and report the average F1-score.

4.2 Performance ComparisonSingle-Source Cross-Lingual NER Table 2 re-ports the results of different single-source cross-lingual NER methods. All results are obtained withEnglish as the source language and others as targetlanguages.

It can be seen that our proposed method outper-forms the previous state-of-the-art methods. Par-ticularly, compared with the remarkable Wu andDredze (2019) and Moon et al. (2019), which usenearly the same NER model as our method butis based on direct model transfer, our method ob-tains significant and consistent improvements in

1https://github.com/huggingface/transformers

es nl deTackstrom (2012) 61.90 59.90 36.40

Rahimi et al. (2019) 71.80 67.60 59.10Chen et al. (2019) 73.50 72.40 56.00

Moon et al. (2019)† 76.53 83.35 72.44Ours-avg 77.75 80.70 74.97Ours-sim 78.00 81.33 75.33

Table 3: Performance comparisons of multi-sourcecross-lingual NER. Ours-avg: averaging teacher mod-els (Eq. 7) . Ours-sim: weighting teacher models withlearned language similarities (Eq. 11). † denotes the re-ported results w.r.t. freezing the bottom three layers ofBERTBASE.

F1-scores, ranging from 0.51 for Dutch to 1.80for German. That well demonstrates the benefitsof teacher-student learning over unlabeled target-language data, compared to direct model transfer.Moreover, compared with the latest meta-learningbased method (Wu et al., 2020), our method re-quires much lower computational costs for bothtraining and inference, meanwhile reaching supe-rior performance.

Multi-Source Cross-Lingual NER Here we se-lect source languages in a leave-one-out manner,i.e., all languages except the target one are regardedas source languages. For fair comparisons, we takeSpanish, Dutch, and German as target languages,respectively.

Table 3 reports the results of different meth-ods for multi-source cross-lingual NER. Both ourteacher-student learning methods, i.e., Ours-avg(averaging teacher models, Eq. 7) and Ours-sim(weighting teacher models with learned languagesimilarities, Eq. 11), outperform previous state-of-the-art methods on Spanish and German by a largemargin, which well demonstrates their effective-ness. We attribute the large performance gain to theteacher-student learning process to further leveragehelpful information from unlabeled data in the tar-get language. Though Moon et al. (2019) achievessuperior performance on Dutch, it is not applicablein cases where the labeled source-language datais inaccessible, and thus it still suffers from theaforementioned limitation w.r.t. data availability.

Moreover, compared with Ours-avg, Ours-simbrings consistent performance improvements. Thatmeans, if unlabeled data in source languages isavailable, using our proposed language similaritymeasuring method for weighting different teacher

es nl deSingle-source:

Ours 76.94 80.89 73.22HL 76.60 (-0.34) 80.43 (-0.46) 72.98 (-0.24)MT 75.60 (-1.34) 79.99 (-0.90) 71.76 (-1.46)

Multi-source:Ours-avg 77.75 80.70 74.97HL-avg 77.65 (-0.10) 80.39 (-0.31) 74.31 (-0.66)MT-avg 77.25 (-0.50) 80.53 (-0.17) 74.18 (-0.79)

Ours-sim 78.00 81.33 75.33HL-sim 77.81 (-0.19) 80.27 (-1.06) 74.63 (-0.70)MT-sim 77.12 (-0.88) 80.24 (-1.09) 74.33 (-1.00)

Table 4: Ablation study of the proposed teacher-studentlearning method for cross-lingual NER. HL: HardLabel; MT: Direct Model Transfer; *-avg: averag-ing source-language models; *-sim: weighting source-language models with learned language similarities.

models can be superior to simply averaging them.

4.3 Ablation Study

Analyses on Teacher-Student Learning To val-idate the reasonableness of our proposed teacher-student learning method for cross-lingual NER, weintroduce the following baselines. 1) Hard Label(HL), which rounds the probability distribution ofentity labels (i.e., soft labels output by teacher mod-els) into a one-hot labelling vector (i.e., hard labels)to guide the learning of the student model. Notethat in multi-source cases, we use the combinedprobability distribution of multiple teacher models(Eq. 6) to derive the hard labels. To be consistentwith Eq. 3, we still adopt the MSE loss here. Infact, both MSE loss and cross-entropy loss leadto the same observation described in this subsec-tion. 2) Direct Model Transfer (MT), where NOunlabeled target-language data is available to per-form teacher-student learning, and thus it degener-ates into: a) directly applying the source-languagemodel in single-source cases, or b) directly apply-ing a weighted ensemble of source-language mod-els in multi-source cases, with weights derived viaEq. 6 and Eq. 11.

Table 4 reports the ablation study results. It canbe seen that using hard labels (i.e., HL-*) wouldresult in consistent performance drops in all cross-lingual NER settings, which validates using softlabels in our proposed teacher-student learningmethod can convey more information for knowl-edge transfer than hard labels. Moreover, we canalso observe that, using direct model transfer (i.e.,

#1Spanish

Source-Language Model: ...Etchart [I-PER, 1.00] Sydney [B-LOC, 0.98] ( Australia [B-LOC, 1.00] ) , 23 may ( EFE [O, 0.53] ) .Ours: Por Mario [B-PER] Etchart [I-PER] Sydney [B-LOC] ( Australia [B-LOC] ) , 23 may ( EFE [B-ORG] ) .Examples in Dtgt: Asi lo anunció a EFE [B-ORG, 1.00] Hans Gaasbek, el abogado de Murillo, argumentando que ...

#2Dutch

Source-Language Model: Vanderpoorten [O, 0.87] : ' Dit is een eerste stap in de herwaardering van het beroepsonderwijs "Ours: Vanderpoorten [B-PER] : ' Dit is een eerste stap in de herwaardering van het beroepsonderwijs "Examples in Dtgt: Vanderpoorten [B-PER, 0.99] stond op het punt die reputatie te bezwadderen.

#3German

Source-Language Model: ... dabei berücksichtigt werden müsse , forderte Hof [B-ORG, 0.85] eine “ Transparenz ” … Ours: Weil die Altersstruktur dabei berücksichtigt werden müsse , forderte Hof [B-PER] eine “ Transparenz ” … Examples in Dtgt: … meint Hof [B-PER, 0.99] , den der " erstaunliche Pragmatismus der Jugendlichen " beeindruckt .

Figure 4: Case study on why teacher-student learning works. The GREEN ( RED ) highlight indicates a correct(incorrect) label. The real-valued numbers indicate the predicted probability corresponding to the entity label.

es nl deOurs 78.00 81.33 75.33cosine 77.86 (-0.14) 79.94 (-1.39) 75.24 (-0.09)`2 77.72 (-0.28) 79.74 (-1.59) 75.09 (-0.24)

Table 5: Comparison between the proposed languagesimilarity measuring method and the commonly usedcosine/`2 metrics for multi-source cross-lingual NER.

MT-*) would lead to even more significant perfor-mance drops in all cross-lingual NER settings (upto 1.46 F1-score). Both demonstrate that leveragingunlabeled data in the target language can be help-ful, and that the proposed teacher-student learningmethod is capable of leveraging such informationeffectively for cross-lingual NER.

Analyses on Language Similarity MeasuringWe further compare the proposed language similar-ity measuring method with other commonly usedunsupervised metrics, i.e., cosine similarity and`2 distance. Specifically, s(x′, µ(k)) in Eq. 11 isreplaced by cosine similarity or negative `2 dis-tance between x′ and the mean value of sentenceembeddings w.r.t the k-th source language.

As shown in Table 5, replacing the proposedlanguage similarity measuring method with eithercosine / `2 metrics leads to consistent performancedrops across all target languages. This furtherdemonstrates the benefits of our language identifi-cation based similarity measuring method.

4.4 Why Teacher-Student Learning Works?

By analyzing which failed cases of directly apply-ing the source-language model are corrected by theproposed teacher-student learning method, we tryto bring up insights on why teacher-student learn-ing works, in the case of single-source cross-lingualNER.

0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9probability of the prediction

0.0

0.2

0.4

0.6

0.8

1.0

perc

enta

ge

esnlde

Figure 5: Percentage of corrected mispredictions, indifferent probability intervals.

Firstly, teacher-student learning can probablyhelp to learn label preferences for some specificwords in the target language. Specifically, if aword appears in the unlabeled target-language dataand the teacher model consistently predicts it tobe associated with an identical label with highprobabilities, the student model would learn thepreferred label w.r.t that word, and predict it incases where the sentence context may not provideenough information. Such label preference canhelp the predictions for tokens that are less am-biguous and generally associated with an identicalentity label. As illustrated in Figure 4, in exam-ple #1, the source-language (teacher) model, failsto identify “EFE” as an ORG in the test sentences,while the student model (i.e., Ours) can correctly la-bel it, because it has seen “EFE” labeled as ORG bythe teacher model with high probabilities in the un-labeled target-language data Dtgt. Similar resultscan also be observed in example #2 and #3.

Moreover, teacher-student learning may help tofind a better classifying hyperplane for the stu-dent NER model with unlabelled target-languagedata. Actually, we notice that the source-languagemodel generally makes correct label predictionswith higher probabilities, and makes mispredic-tions with relatively lower probabilities. By calcu-

lating the proportion of its mispredictions that arecorrected by our teacher-student learning methodin different probability intervals, we find that ourmethod tends to correct the low-confidence mispre-dictions, as illustrated in Figure 5. We conjecturethat, with the help of unlabeled target-languagedata, our method can probably find a better classi-fying hyperplane for the student model, so that thelow-confidence mispredictions, which are closer tothe classifying hyperplane of the source-languagemodel, can be clarified.

5 Conclusion

In this paper, we propose a teacher-student learn-ing method for single-/multi-source cross-lingualNER, via using source-language models as teach-ers to train a student model on unlabeled data inthe target language. The proposed method does notrely on labelled data in the source languages and iscapable of leveraging extra information in the un-labelled target-language data, which addresses thelimitations of previous label-projection based andmodel-transfer based methods. We also proposea language similarity measuring method based onlanguage identification, to better weight differentteacher models. Extensive experiments on bench-mark datasets show that our method outperformsthe existing state-of-the-art approaches.

ReferencesCristian Bucilu, Rich Caruana, and Alexandru

Niculescu-Mizil. 2006. Model compression. In Pro-ceedings of the 12th ACM SIGKDD InternationalConference on Knowledge Discovery and Data Min-ing, pages 535–541. ACM.

Xilun Chen, Ahmed Hassan Awadallah, Hany Has-san, Wei Wang, and Claire Cardie. 2019. Multi-source cross-lingual model transfer: Learning whatto share. In Proceedings of the 57th Annual Meet-ing of the Association for Computational Linguistics,pages 3098–3112.

Jason P.C. Chiu and Eric Nichols. 2016. Named entityrecognition with bidirectional LSTM-CNNs. Trans-actions of the Association for Computational Lin-guistics, 4:357–370.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: Pre-training ofdeep bidirectional transformers for language under-standing. In Proceedings of the 2019 Conference ofthe North American Chapter of the Association forComputational Linguistics: Human Language Tech-nologies, pages 4171–4186.

Jan Vium Enghoff, Søren Harrison, and Zeljko Agic.2018. Low-resource named entity recognition viamulti-source projection: Not quite there yet? InProceedings of the 2018 EMNLP Workshop W-NUT:The 4th Workshop on Noisy User-generated Text,pages 195–201.

Dan Hendrycks and Kevin Gimpel. 2016. Bridgingnonlinearities and stochastic regularizers with gaus-sian error linear units. CoRR, abs/1606.08415.

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015.Distilling the knowledge in a neural network. arXivpreprint arXiv:1503.02531.

Lifu Huang, Heng Ji, and Jonathan May. 2019. Cross-lingual multi-level adversarial transfer to enhancelow-resource name tagging. In Proceedings of the2019 Conference of the North American Chapterof the Association for Computational Linguistics:Human Language Technologies, pages 3823–3833,Minneapolis, Minnesota.

Guillaume Lample, Miguel Ballesteros, Sandeep Sub-ramanian, Kazuya Kawakami, and Chris Dyer. 2016.Neural architectures for named entity recognition.In Proceedings of the 2016 Conference of the NorthAmerican Chapter of the Association for Computa-tional Linguistics: Human Language Technologies,pages 260–270.

Ilya Loshchilov and Frank Hutter. 2017. Fixingweight decay regularization in adam. arXiv preprintarXiv:1711.05101.

Stephen Mayhew, Chen-Tse Tsai, and Dan Roth. 2017.Cheap translation for cross-lingual named entityrecognition. In Proceedings of the 2017 Conferenceon Empirical Methods in Natural Language Process-ing, pages 2536–2545.

David McClosky, Eugene Charniak, and Mark Johnson.2010. Automatic domain adaptation for parsing. InHuman Language Technologies: The 2010 AnnualConference of the North American Chapter of the As-sociation for Computational Linguistics, pages 28–36.

Taesun Moon, Parul Awasthy, Jian Ni, and RaduFlorian. 2019. Towards lingua franca namedentity recognition with bert. arXiv preprintarXiv:1912.01389.

Jian Ni, Georgiana Dinu, and Radu Florian. 2017.Weakly supervised cross-lingual named entity recog-nition via effective annotation and representationprojection. In Proceedings of the 55th Annual Meet-ing of the Association for Computational Linguistics,pages 1470–1480.

Shuke Peng, Xinjing Huang, Zehao Lin, Feng Ji,Haiqing Chen, and Yin Zhang. 2019. Teacher-student framework enhanced multi-domain dialoguegeneration. arXiv preprint arXiv:1908.07137.

Matthew Peters, Waleed Ammar, Chandra Bhagavat-ula, and Russell Power. 2017. Semi-supervised se-quence tagging with bidirectional language models.In Proceedings of the 55th Annual Meeting of theAssociation for Computational Linguistics, pages1756–1765.

Pedro O Pinheiro. 2018. Unsupervised domain adapta-tion with similarity learning. In Proceedings of theIEEE Conference on Computer Vision and PatternRecognition, pages 8004–8013.

Afshin Rahimi, Yuan Li, and Trevor Cohn. 2019. Mas-sively multilingual transfer for NER. In Proceed-ings of the 57th Annual Meeting of the Associationfor Computational Linguistics, pages 151–164.

Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz,Andy Davis, Quoc Le, Geoffrey Hinton, and JeffDean. 2017. Outrageously large neural networks:The sparsely-gated mixture-of-experts layer. arXivpreprint arXiv:1701.06538.

Oscar Tackstrom. 2012. Nudging the envelope of di-rect transfer methods for multilingual named entityrecognition. In Proceedings of the NAACL-HLTWorkshop on the Induction of Linguistic Structure,pages 55–63.

Oscar Tackstrom, Ryan McDonald, and Jakob Uszko-reit. 2012. Cross-lingual word clusters for directtransfer of linguistic structure. In Proceedings of the2012 Conference of the North American Chapter ofthe Association for Computational Linguistics: Hu-man Language Technologies, pages 477–487.

Xu Tan, Yi Ren, Di He, Tao Qin, and Tie-Yan Liu.2019. Multilingual neural machine translation withknowledge distillation. In International Conferenceon Learning Representations.

Erik F. Tjong Kim Sang. 2002. Introduction to theCoNLL-2002 shared task: Language-independentnamed entity recognition. In COLING-02: The6th Conference on Natural Language Learning 2002(CoNLL-2002).

Erik F. Tjong Kim Sang and Fien De Meulder.2003. Introduction to the CoNLL-2003 shared task:Language-independent named entity recognition. InProceedings of the Seventh Conference on Natu-ral Language Learning at HLT-NAACL 2003, pages142–147.

Chen-Tse Tsai, Stephen Mayhew, and Dan Roth. 2016.Cross-lingual named entity recognition via wikifica-tion. In Proceedings of The 20th SIGNLL Confer-ence on Computational Natural Language Learning,pages 219–228.

Mengqiu Wang and Christopher D. Manning. 2014.Cross-lingual projected expectation regularizationfor weakly supervised learning. Transactions of theAssociation for Computational Linguistics, 2:55–66.

Thomas Wolf, Lysandre Debut, Victor Sanh, JulienChaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Fun-towicz, et al. 2019. Transformers: State-of-the-art natural language processing. arXiv preprintarXiv:1910.03771.

Qianhui Wu, Zijia Lin, Guoxin Wang, Hui Chen,Borje F Karlsson, Biqing Huang, and Chin-Yew Lin.2020. Enhanced meta-learning for cross-lingualnamed entity recognition with minimal resources. InProceedings of the AAAI Conference on Artificial In-telligence.

Shijie Wu and Mark Dredze. 2019. Beto, bentz, be-cas: The surprising cross-lingual effectiveness ofBERT. In Proceedings of the 2019 Conference onEmpirical Methods in Natural Language Processingand the 9th International Joint Conference on Natu-ral Language Processing (EMNLP-IJCNLP), pages833–844.

Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc VLe, Mohammad Norouzi, Wolfgang Macherey,Maxim Krikun, Yuan Cao, Qin Gao, KlausMacherey, et al. 2016. Google’s neural machinetranslation system: Bridging the gap between hu-man and machine translation. arXiv preprintarXiv:1609.08144.

Jiateng Xie, Zhilin Yang, Graham Neubig, Noah A.Smith, and Jaime Carbonell. 2018. Neural cross-lingual named entity recognition with minimal re-sources. In Proceedings of the 2018 Conference onEmpirical Methods in Natural Language Processing,pages 369–379.

Ze Yang, Linjun Shou, Ming Gong, Wutao Lin,and Daxin Jiang. 2019. Model compression withtwo-stage multi-teacher knowledge distillation forweb question answering system. arXiv preprintarXiv:1910.08381.

Shan You, Chang Xu, Chao Xu, and Dacheng Tao.2017. Learning from multiple teacher networks.In Proceedings of the 23rd ACM SIGKDD Inter-national Conference on Knowledge Discovery andData Mining, pages 1285–1294. ACM.

Ayah Zirikly and Masato Hagiwara. 2015. Cross-lingual transfer of named entity recognizers withoutparallel corpora. In Proceedings of the 53rd AnnualMeeting of the Association for Computational Lin-guistics and the 7th International Joint Conferenceon Natural Language Processing (Volume 2: ShortPapers), pages 390–396.