800 IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING, VOL. 13, NO. 4, AUGUST 2019

SpeakerBeam: Speaker Aware Neural Network forTarget Speaker Extraction in Speech Mixtures

Katerina Žmolíková , Student Member, IEEE, Marc Delcroix , Senior Member, IEEE,Keisuke Kinoshita, Member, IEEE, Tsubasa Ochiai, Member, IEEE, Tomohiro Nakatani , Senior Member, IEEE,

Lukáš Burget, Member, IEEE, and Jan Cernocký , Senior Member, IEEE

Abstract—The processing of speech corrupted by interferingoverlapping speakers is one of the challenging problems withregards to today’s automatic speech recognition systems. Recently,approaches based on deep learning have made great progresstoward solving this problem. Most of these approaches tacklethe problem as speech separation, i.e., they blindly recover allthe speakers from the mixture. In some scenarios, such as smartpersonal devices, we may however be interested in recovering onetarget speaker from a mixture. In this paper, we introduce Speaker-Beam, a method for extracting a target speaker from the mixturebased on an adaptation utterance spoken by the target speaker.Formulating the problem as speaker extraction avoids certainissues such as label permutation and the need to determine thenumber of speakers in the mixture. With SpeakerBeam, we jointlylearn to extract a representation from the adaptation utterancecharacterizing the target speaker and to use this representationto extract the speaker. We explore several ways to do this, mostlyinspired by speaker adaptation in acoustic models for automaticspeech recognition. We evaluate the performance on the widelyused WSJ0-2mix and WSJ0-3mix datasets, and these datasets mod-ified with more noise or more realistic overlapping patterns. Wefurther analyze the learned behavior by exploring the speaker rep-resentations and assessing the effect of the length of the adaptationdata. The results show the benefit of including speaker informationin the processing and the effectiveness of the proposed method.

Index Terms—Speaker extraction, speaker-aware neural net-work, multi-speaker speech recognition.

I. INTRODUCTION

AUTOMATIC speech recognition systems are now becom-ing widely deployed in real applications, which increases

the need for robustness in adverse conditions. One particularlychallenging problem, commonly occurring in spontaneous

Manuscript received November 17, 2018; revised March 12, 2019 and May3, 2019; accepted May 29, 2019. Date of publication June 13, 2019; date ofcurrent version July 25, 2019. This paper was supported in part by the Tech-nology Agency of the Czech Republic project No. TJ01000208 “NOSICI,” inpart by the Czech National Science Foundation (GACR) project “NEUREM3”No. 19-26934X, and in part by the National Programme of Sustainability (NPUII) project “IT4Innovations excellence in science—LQ1602.” The guest editorcoordinating the review of this paper and approving it for publication was Dr.Michael Seltzer. (Corresponding author: Katerina Žmolíková.)

K. Žmolíková, L. Burget, and J. Cernocký are with Speech@FIT, BrnoUniversity of Technology, Brno 60190, Czech Republic (e-mail: izmolikova@fit.vutbr.cz; burget@fit.vutbr.cz; cernocky@fit.vutbr.cz).

M. Delcroix, K. Kinoshita, T. Ochiai, and T. Nakatani are with NTTCommunications Science Laboratories, NTT Corporation, Kyoto 619-0237,Japan (e-mail: marc.delcroix@ieee.org; kinoshita.k@lab.ntt.co.jp; ochiai.tsubasa@lab.ntt.co.jp; tnak@ieee.org).

Digital Object Identifier 10.1109/JSTSP.2019.2922820

conversations and human-machine communication, is speechcorrupted by interfering speakers. This type of interference hasshown to be very difficult to reduce and greatly deteriorates thequality of speech transcriptions. Most of the research dealingwith overlapping speech has focused on speech separation[1], [2], where all the source signals are recovered from theobserved mixture signal. This problem has been studied inthe past using methods, such as Computational auditory sceneanalysis [3], [4], Non-negative matrix factorization [5], [6]and Factorial Hidden Markov models [7], [8], and was greatlyadvanced recently thanks to deep learning based approaches[2], [9]–[11]. However, in some practical situations, such assmart personal devices, we may be interested in recovering asingle target speaker while reducing noise and the effect ofinterfering speakers [12]–[15]. We call this problem targetspeaker extraction. In contrast to speech separation, extractingthe target speaker avoids problems such as label permutation,dependence on the number of speakers and the speaker-tracingproblem (see Section II-A for further discussion).

Most previous studies aiming to extract the target speaker[16]–[18] realized their aim by training a neural network on thetarget speaker data only, thus creating a model specifically de-signed to extract this particular speaker. The models are trainedeither in a speaker-pair-dependent mode, where both the targetspeaker and the interferer are observed in the training data, orin a target-dependent mode where the model can generalize tounseen interfering speakers. Both of these modes rely on theassumption of having substantial amount of data from the targetspeaker and do not allow the extraction of a speaker that wasunseen during the training.

In this work, we follow the idea of target speaker extractionusing a neural network, but rather than using a specialized modelfor a particular target speaker, we train a speaker independentmodel and inform it about the target speaker using additionalspeaker information. The network can use this information tofocus on the target speaker, considering all the others as interfer-ence. We call this approach SpeakerBeam. The neural networkin SpeakerBeam can be trained on a variety of speakers andemployed to extract speakers unseen during the training. Theadditional speaker information determining the target speakeris obtained from an adaptation utterance spoken by the targetspeaker. In practice, this adaptation utterance could be obtained,for example, from part of a conversation without any overlap orpre-recorded by the target user on his/her personal device.

1932-4553 © 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

ŽMOLÍKOVÁ et al.: SPEAKERBEAM: SPEAKER AWARE NEURAL NETWORK FOR TARGET SPEAKER EXTRACTION IN SPEECH MIXTURES 801

We explore different approaches for utilizing the informationfrom adaptation utterances to cause the neural network to ex-tract the target speaker. Most of these approaches are inspiredby speaker adaptation of acoustic models. There are two mainproblems to be solved: a) how to use the speaker informationto modify the behavior of the neural network, and b) how toextract the speaker information from the adaptation utterance.For the first problem, we look into three different methods: in-put bias adaptation [19]–[21], factorized layer [22] and scaledactivations [23]. To extract the speaker information from theadaptation utterance, we can either use speaker representationsthat have been widely used for speaker identification tasks, suchas i-vectors [24] or jointly learn the speaker representation usingsequence summarization [21] or its modification with a simpleattention mechanism.

In this paper, we first explain how this work relates to re-cent speech separation and extraction methods and our previouswork (Section II). Then, we describe the proposed SpeakerBeammethod and its variants (Section III). Section IV describes the in-tegration of the method with multichannel processing and an au-tomatic speech recognizer. Sections V and VI outline the datasetsused and the experimental setup. Finally, the results are reportedin Section VII and further analysis is provided in Section VIII.

II. RELATIONSHIP TO PREVIOUS WORK

A. Related Speech Separation Work

Most recent work on the neural network processing of over-lapped speech tackles the problem from a speech separation per-spective, i.e. recovering all the sources from the given mixture.Compared with the separation of speech and non-speech signals(e.g. speech-noise or speech-music mixtures), where the indi-vidual sources have inherently different characteristics, speech-speech separation gives rise to several problems that requiremore specialized approaches. To introduce these problems, letus consider a simple approach, where the neural network pro-cesses a mixture signal and produces all the source signals asindividual outputs. This approach suffers from the followingproblems:

1) dependence on the number of speakers — the architec-ture of the neural network inherently limits the number ofspeakers in the mixture, that can be processed.

2) label-permutation problem — the correspondence be-tween outputs of the network and the speakers is arbitrary,therefore there are multiple possible correct outputs of thenetwork where the speaker order varies. This makes it dif-ficult to define the targets for the network and to computethe error function during training.

3) speaker-tracing problem — when processing a mixturewith a network frame-by-frame or block-by-block, the or-der of the speakers on the output may change arbitrarilyand proper alignment across the frames or blocks needs tobe ensured.

The two main approaches that address neural network basedspeech separation are Deep clustering (DC) and PermutationInvariant Training (PIT). In DC [10], [25] and its variants [26],

a neural network is used to compute embeddings for all time-frequency bins. These embeddings can be then clustered intogroup time-frequency bins corresponding to the same speaker.This solves the label-permutation problem as the estimated em-beddings are ignorant as regards the order of the sources. Thearchitecture of such a network is also independent of the numberof speakers, although this number must be determined during theclustering step.

In PIT [11], [27], the neural network outputs estimations of allsource signals. The main idea is to solve the label permutationproblem by finding the permutation of the estimated sources onthe output of the network that best matches the desired targets.Kolbæk et al. [27] have also shown that the same network canbe used to process mixtures with different numbers of speakersas long as we can define a maximum, which can be a reasonableassumption in many scenarios. The objective of PIT is moreclosely related to the actual separation task than in DC and canbe more easily combined with the joint training of e.g. an ASRsystem.

For the speaker tracing problem, both the DC and PIT meth-ods rely on the ability of a recurrent architecture to keep itsoutputs consistent over time. In DC, the network should keepthe embeddings for the same speaker in the same part of theembedding space, and in PIT, it should keep assigning the samespeaker to the same output of the network. This has proven towork well in cases where the mixture is short and fully over-lapped, but can cause problems for longer recordings or morecomplicated overlapping patterns, which naturally occur in realconversations.

The proposed SpeakerBeam method does not suffer fromproblems 1) and 2) as the neural network predicts the speech ofthe target speaker only. Additionally, it also solves the speakertracing problem as the explicit speaker information enables theneural network to follow the same speaker over different framesor processing segments.

B. Relationship With Our Previous Work

We gradually built and refined the SpeakerBeam approachover several studies [28]–[31]. In this section, we clarify therelationship between this work and our previous research.

In [28], we first introduced the speaker-aware extractionscheme as part of a multi-channel system and experimented withdifferent speaker-dependent neural network architectures. Thework in [28] focused mainly on a closed-speaker-set case andevaluated a factorized layer scheme (see Section III) as the mostsuitable method. We later extended this method with sequencesummarization in [29] to improve the performance in an open-speaker set scenario. Therein, we also evaluated SpeakerBeamas the front-end of an automatic speech recognition system. In[30], the automatic speech recognition performance was furtherimproved by exploring the joint training of the SpeakerBeamfront-end with an ASR system. While these studies [28]–[30]focused on a multichannel case, in [31], we investigated theASR performance in a single-channel setting.

This paper builds upon the previous ones, summarizes thefindings, and brings new modifications, evaluation and anal-ysis. In particular, we provide a thorough evaluation of the

802 IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING, VOL. 13, NO. 4, AUGUST 2019

Fig. 1. Overall scheme of single-channel extraction for an example with one interfering speaker and noise.

single-channel scenario and different variants of SpeakerBeamon standard WSJ0-2mix and WSJ0-3mix datasets. We also cre-ate new WSJ0-2mix-long and WSJ0-2mix-noisy datasets to ex-plore the effect of more natural overlapping patterns and a higheramount of noise on the results. Furthermore, we experiment witha combination of SpeakerBeam and DC, leading to improvedperformance. We finally provide an analysis of the learnedembeddings and behavior with different lengths of adaptationutterance.

C. Related Speaker Extraction Work

After our proposal of SpeakerBeam in [28], several other stud-ies followed the idea of extracting a target speaker using an adap-tation utterance [15], [32], [33]. The authors of [15] built upondeep attractor networks [26] and suggested using the adaptationutterance to map the time-frequency points of the mixture into acanonical embedding space, where the embeddings correspond-ing to the target speaker are pulled together. The results showeffectiveness even for very short adaptation utterances, however,the approach remains to be tested on a publicly available datasetor under more challenging conditions.

The work reported in [32] realized target speech extractionby combining speech separation and speaker identification. Theauthors proposed making use of embeddings in deep attractornetworks to identify the target speaker in the extracted signals.This approach cannot exploit auxiliary information about thetarget speaker to improve the separation process. Moreover, itrequires an additional module for speaker selection that mayintroduce speaker identification errors.

The method introduced in [33] proposes concatenating a d-vector [34] extracted from the adaptation utterance with one ofthe layers of the neural network to achieve the target speakerextraction. A similar way of using the speaker representationdid not work well in our experiments (see ’input-bias’ method inSection VII-A). This may possibly be due to the difference in theexperimental settings, i.e. in [33], the target speaker was notablydominant over the interference (10.1 dB signal-to-distortion ra-tio), while in our experiments, the target and interference areequally strong on average (0.2 dB SDR).

III. PROPOSED SPEAKERBEAM METHOD

In this section, we formally define the problem of speakerextraction, introduce the notation we use and describe the pro-posed SpeakerBeam method. Figure 1 shows the overall schemeof the mixing model and the extraction.

A. Problem Definition

The problem of speaker extraction is to isolate the speech of atarget speaker from an observed mixture of multiple overlappingspeakers and optionally an additional noise. We assume a mixingmodel:

y(m)[n] = s(m)0 [n] +

I−1∑

i=1

s(m)i [n] + v(m)[n], (1)

where n is the discrete time index, I is the number of speakers inthe mixture, s(m)

i [n] for i = 0, . . . , I − 1 is the speech signal ofthe ith speaker as captured by microphone m with i = 0 beingthe target speaker, v(m)[n] is the additional noise and y(m)[n] isthe observed mixture.

In this work, we perform the extraction in the short-timeFourier transform (STFT) domain, where we can model the mix-ing process as

Y (m)[t, f ] = S(m)0 [t, f ] +

I−1∑

i=1

S(m)i [t, f ] + V (m)[t, f ]. (2)

Here, [t, f ] are the indexes corresponding to the time frame andthe frequency bin and Y , Si, V are the STFT-domain counter-parts of y, si, v, respectively. We will use the notation Y, Si,V for the T × F matrices comprising all time-frequency pointsY [t, f ], Si[t, f ], V [t, f ], respectively, with T being the numberof time frames and F the number of frequency bins in the STFTrepresentation of given signal. In the remainder of this section,we will focus on a single channel case (in this case, index (m) canbe omitted). A multi-channel extension of SpeakerBeam will beaddressed in Section IV.

Our method extracts the target speaker from the mixture, usingadditional information about the target speaker in the form of anadaptation utterance. This utterance will be denoted a[n] in thetime domain, A[t, f ] in the STFT domain and A for a Ta × F

ŽMOLÍKOVÁ et al.: SPEAKERBEAM: SPEAKER AWARE NEURAL NETWORK FOR TARGET SPEAKER EXTRACTION IN SPEECH MIXTURES 803

Fig. 2. Three different methods of informing the neural network about the target speaker. The red box represents the speaker information λ derived from theadaptation utterance. �; denotes vector-scalar multiplication, �; denotes element-wise vector-vector multiplication.

matrix comprising all time-frequency points, where Ta is thenumber of frames in the adaptation utterance. The adaptationutterance a[n] contains speech from the same speaker as s0[n],however it is always a different utterance from s0[n].

The extraction is performed by means of a neural networkthat takes the mixture as an input, the adaptation utterance asauxiliary information and provides a mask that can be used toobtain an estimate of the target speech:

M = g(|Y|, |A|), (3)

∧S0 = M�Y, (4)

where g is the transformation carried out by the mask estima-tion neural network, | · | denotes the magnitude of a given STFTsignal, M is the estimated mask, � denotes element-wise mul-tiplication and

∧S0 is the estimated target-speaker STFT signal.

In general, it would be possible to process directly the complexspectrum of the signals, but in this work, we limit ourselves tousing the magnitudes only as in most of the related studies [10],[11], [26].

B. Informing the Network

Modifying the behavior of a neural network using additionalspeaker information is a task that has been heavily explored forspeaker adaptation in acoustic models. The methods applied inour work are thus inspired by previous findings in this field.We explore three different ways of informing the network —input bias adaptation, a factorized layer and scaled activations, asdepicted in Figure 2. Please note, that the figure depicts a roughschematic view of the network, and a more precise descriptionof the layers and the architecture will be given in the Systemconfiguration sub-section in Section VI. All three methods makeuse of the speaker information λ (red box in Figure 2). In III-C,we will specify how λ is obtained from the adaptation utterance.

1) Input Bias Adaptation: The most straightforward tech-nique, commonly used in acoustic modeling, is to append thespeaker information to the features on the input of the neuralnetwork [19]–[21]. This effectively performs the adaptation ofthe biases in the first layer of the network [22]. We can expressthe neural network processing as

X1 = σ0(L0([X0,λ(bias)];ψ0)), (5)

Xk+1 = σk(Lk(Xk;ψk)) for k ≥ 1, (6)

where Xk is the input to the kth layer, Lk(Xk, ψk) is the trans-formation computed by the kth layer parameterized by ψk, andσk is an activation function. For example, with fully connectedlayers, ψ = {W,b} and L(X, ψ) = Wx+ b, where W is aweight matrix and b is a bias vector.

2) Factorized Layer: Previous literature has shown, that amore powerful adaptation than simply adapting the input biascan be achieved by modifying all the parameters in one of thelayers of the network. In a method introduced in [35], one ofthe layers of the network is factorized into multiple sublayers,which are then combined using weights derived from the speakerinformation. Following the previous notation and denoting theindex of the factorized layer q and the number of sub-layers asJ , the network processing is defined as

Xk+1 =

{σk(Lk(Xk;ψk)) for k �= q,

σk(∑J−1

j=0 λ(fact)j Lk(Xk;ψ

(j)k )) for k = q.

(7)The network thus learns common basis for all the speakers,

which then can be combined with different weights λ(fact) tomake the network extract different speakers. The size of vectorλ(fact) is determined by the number of factorized sub-layers J ,which is chosen as a hyper-parameter.

3) Scaled Activations: An alternative speaker adaptationmethod is introduced in [23], [36], where the output of eachunit in one of the layers of the network is scaled by weights

804 IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING, VOL. 13, NO. 4, AUGUST 2019

derived from the speaker information. This method is similarto the factorized layer approach, however it is computationallysimpler. In this case, the processing performed by the neuralnetwork is:

Xk+1 =

{σk(Lk(Xk;ψk)) for k �= q,

σk(λ(act) � Lk(Xk;ψk)) for k = q.

(8)

Here the size of vector λ(act) is determined by the size of theadaptive layer, rather than the number of factorized sub-layersas in the previous approach. Note that although the method in-troduced here follows the same idea as in [23], [36], it differsslightly in how the scaling weights are obtained and where ex-actly they are applied.

C. Obtaining the Speaker Information

In this section, we describe methods for extracting speakerinformation λ from an adaptation utterance, which is then usedto inform the network, as described in the previous section. Weexplore three different methods — i-vector based extraction, asequence summarizing network and its extension using simpleattention.

1) I-Vectors: A common way to represent speaker-related in-formation in speech data is the i-vector, which has been usedextensively for e.g. speaker recognition [24], speaker adaptation[19], [37] or speaker diarization [38]. I-vectors are fixed-lengthlow-dimensional representations of speech segments of variablelength. For more information on i-vector extraction, we refer thereader to [24], [39]. In our work, we extract the i-vector fromthe adaptation utterance and post-process it with an auxiliarynetwork to obtain the vector λ used in one of the three schemespresented in the previous section.

2) Sequence Summarizing Network: Although i-vectorextraction is designed to preserve speaker variability, it usesa separate step, which is not optimized for the target speakerextraction task we are addressing. Therefore, some informationimportant for speaker extraction may be lost. The secondmethod we propose applies the adaptation utterance directly tothe input of the auxiliary network. To convert from frame-wisefeatures to an utterance-wise vector, we employ average pool-ing after the last layer in the auxiliary network. This way, theextraction of speaker information from the adaptation utterancemay be learned jointly with the speaker extraction:

λt = z(|A|), (9)

λ =1

Ta

∑λt, (10)

where z is the transformation performed by the auxiliary neuralnetwork, λt is the frame-wise vector extracted by the auxiliarynetwork for frame t, which is then averaged over the T framesin the adaptation utterance to obtain the final λ.

3) Sequence Summarizing Network With Attention: The av-erage pooling at the end of the auxiliary network weighs allframes equally. This may be detrimental when some of theframes are silence or for example, corrupted by noise. To makethe scheme more flexible, we extend it with a simple attentionmechanism. Here, the output of the auxiliary network is extended

with one value, at. This predicted value for each frame is thenused, after a softmax operation, to weigh the contribution of theindividual frames to the averaging operation:

(λt, at) = z(|A|), (11)

a = softmax(a), (12)

λ =∑

atλt, (13)

where a = [a1, . . . , aTa] denotes the attention energies (before

the softmax), and a = [a1, . . . , aTa] is the final attention vector,

after the softmax normalization.

D. Training Objective

The neural network in SpeakerBeam estimates a T-F maskcorresponding to the target speech. Different choices for the ob-jective function for training the mask estimation networks havebeen previously explored in the literature. Here, we followthe findings in [40], which show that a good choice for the objec-tive function is the mean square error between the magnitude ofthe STFT of the desired speech and the magnitude of the STFT ofthe observation, masked by the estimated mask. In addition, wealso weigh the different time-frequency points using the phasedifferences between the clean and observed signals as suggestedin [27]. This leads to an objective function with the form:

Jspkbeam = ||M� |Y| − |S0| �max(0, cos(θy − θs0))||2,(14)

where θy and θs0 are the T × F matrices of the phases ofobserved speech and target speaker speech, respectively.

We also explore the multi-task training of SpeakerBeam to-gether with the Deep clustering method, in a similar fashion tothat employed for Chimera networks [41], where the DC objec-tive serves as a regularizer in a singing voice separation task. Inthis case, the neural network has two output layers, one predict-ing a mask for SpeakerBeam and the other predicting embed-dings for Deep clustering

(M,E) = g(|Y|, |A|), (15)

whereE is the matrix of the embeddings. The objective functionof the training is then computed as the average of the Speaker-Beam and Deep clustering objective functions

Jspkbeam+dc = αJspkbeam + βJdc, (16)

where α, β are interpolation weights. In this paper, we set α andβ so that both objectives are in approximately the same range(α = 0.5, β = 0.5e−5). For details on the computation of theobjective function for deep clustering Jdc from the estimatedembeddings E, please refer to [10], [25].

IV. INTEGRATION WITH BEAMFORMING AND ASR

In this section, we describe how to integrate the SpeakerBeammethod with beamforming and an ASR-level objective criterion.

ŽMOLÍKOVÁ et al.: SPEAKERBEAM: SPEAKER AWARE NEURAL NETWORK FOR TARGET SPEAKER EXTRACTION IN SPEECH MIXTURES 805

A. Multi-Channel Extraction

Following the single-channel procedure, the neural networkestimates a mask corresponding to the target speech in the mix-ture. The mask is estimated for each channel separately and theoverall mask is obtained as a median across all the channels.The resulting mask is then used to accumulate statistics aboutthe target signal and compute statistically optimal beamformingfilters. Finally, to obtain the estimate of the target speech, thefilters are applied to the multi-channel signal. The procedure forestimating the statistics of the target signal can be described as

M(m) = g(|Y(m)|, |A|), (17)

M = median(M(m)), (18)

ΦSS [f ] =

∑tM [t, f ]y[t, f ]yH [t, f ]∑

tM [t, f ], (19)

ΦNN [f ] =

∑t(1−M [t, f ])y[t, f ]yH [t, f ]∑

t(1−M [t, f ]), (20)

where M(m) is the estimated mask for channel m andΦSS ,ΦNN are the spatial co-variance matrices (SCM) corre-sponding to the target speech and interference, respectively.y[t, f ] = [Y (1)[t, f ], . . . , Y (M)[t, f ]] is a vector comprising theobserved signal at time-frequency point [t, f ] for all micro-phones. Different beamforming filters, such as the GeneralizedEigenvalue beamformer (GEV) [42] or Minimum variance dis-tortionless response (MVDR) beamformer [43], can then becomputed using the estimated ΦSS ,ΦNN . In this work, we usethe GEV beamformer defined as

hGEV[f ] = argmaxh

hH [f ]ΦSS [f ]h[f ]

hH [f ]ΦNN [f ]h[f ], (21)

where h is a beamforming filter. The computed beamformingfilters then can be used to estimate the target signal as

∧S[t, f ] = hH [f ]y[t, f ]. (22)

This procedure for neural network mask-based beamform-ing was proposed for speech denoising [44], [45] and has beenshown to be very effective. Estimating the mask by using the neu-ral network from each channel separately ensures independencefrom microphone array configuration, and averaging across timeand channels when computing the statistics provides robustnessagainst small errors made by the network. Additionally, speechproduced by the linear filtering process is better suited for pro-cessing by automatic speech recognition systems than signalsproduced by masking as in Eq. (4).

B. Joint Training With ASR

For a case where SpeakerBeam is used in a chain with beam-forming and the ASR acoustic model, we also explore the optionof training it jointly with the acoustic model, using the cross-entropy between the estimated tied-state distribution and the truetied-state labels, Jasr. To train the SpeakerBeam network, thisobjective is then back-propagated through the acoustic model,

feature extraction and the beamforming process:

∂Jasr∂ψ

=∂Jasr

∂∧sfbank

∂∧sfbank

∂∧S

∂∧S

∂M

∂M

∂ψ, (23)

where∧sfbank are the features extracted from the estimated sig-

nal,∧S is the STFT of the estimated signal, M are the estimated

masks andψ is the vector of the parameters of the SpeakerBeamneural network. Most of the gradients can be computed usingbackpropagation through standard neural network blocks. Forgradient ∂

∧S/∂M, we need to backpropagate through a GEV

beamformer, in particular through complex eigenvalue decom-position. This step was thoroughly covered in [46].

V. DATASETS

For our evaluation, we chose the dataset introduced in [10],which has been used in many previous studies [10], [25]–[27].It consists of simulated mixtures based on utterances taken fromthe Wall Street Journal (WSJ0) corpus [47]. For different experi-ments, we report results for four different versions of the dataset,namely WSJ0-2mix [10] for single-channel 2-speaker exper-iments, WSJ0-3mix [10] for single-channel 3-speaker experi-ments, WSJ0-2mix-MC [48] for multi-channel experiments andour own modification of WSJ0-2mix, WSJ0-2mix-long, whichconsists of single-channel 2-speaker mixtures with a longer du-ration and more complicated overlapping pattern and WSJ0-2mix-noisy, where we mixed additional noise into the mix-tures. In the following, we describe these sets in detail. Withall datasets, the adaptation utterances are randomly chosen. Inevaluation set, for each mixture and each speaker in the mix-ture, we randomly choose one utterance from the same speaker,different than the utterance in the mixture, to be the adaptationutterance. For training set, for each mixture and each speaker,we randomly choose 100 adaptation utterances which we iteratethrough over the training epochs (the same adaptation utterancemay be repeated). The choice for both evaluation and training isfixed for all experiments.

A. WSJ0-2mix and WSJ0-3mix

The WSJ0-2mix [10] contains mixtures of two speakers atsignal-to-noise ratios between 0 dB and 5 dB. It consists of atraining set, a cross validation set and an evaluation set of 30,10 and 5 hours, respectively. For training and cross-validationsets, the mixed utterances were randomly selected from thesi_tr_s, while for evaluation set, the utterances were takenfrom si_dt_05 and si_et_05 parts of WSJ0. In total, thetraining set contains 20000 mixtures from 101 speakers, thecross-validation set contains 5000 mixtures from the same 101speakers and the evaluation set contains 3000 utterances from 18speakers (unseen in the training). The WSJ0-3mix [25] containsthree-speaker mixtures analogous to WSJ0-2mix in terms of theamounts of data, number of speakers and WSJ0 sets from whichthe utterances are selected. All data are used at an 8 kHz sam-pling rate for consistency with previous studies. In experimentsevaluating only signal-based measures, we use “min” versions

806 IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING, VOL. 13, NO. 4, AUGUST 2019

Fig. 3. Type of mixtures in datasets WSJ0-2mix and WSJ0-2mix-long. Thefirst row corresponds to speech from speaker A, the second row to speech fromspeaker B.

of the datasets, where the mixture is cut to the length of the short-est utterance (for consistency with previous work). However, tobe able to evaluate ASR accuracy in Sections VIII.D, VIII.E,VIII.F, we use the “max” version, where the shorter utterance inthe mixture is padded with zeros.

B. WSJ0-2mix-long

In addition to the original WSJ0-2mix, we created a datasetthat aims to model more realistic overlapping conditions, similarto those occurring in natural conversations. The mixing processfollowed the procedure used to create WSJ0-2mix, but for eachof the speakers in the mixture, we selected 3 random utterancesand placed them in sequence with random pauses in between(sampled uniformly in the 0-10 seconds range). This resultedin a dataset of mixtures with an average length of 45 secondsand an average overlap of 20%. Figure 3 shows a schematiccomparison of the types of mixtures in WSJ0-2mix and WSJ0-2mix-long.

C. WSJ0-2mix-MC

The WSJ0-2mix-MC [48] dataset is a spatialized version ofWSJ0-2mix. It is created by convolving the data with roomimpulse responses generated with the image method [49], [50]to simulate an 8-channel microphone array. The room char-acteristics, speaker locations and microphone array geometryare randomly generated — microphone array sizes range from15 to 25 cm, T60 is drawn from 0.2-0.6 seconds. The averagedistance of a speaker from the array is 1.3 m with a 0.4 mstandard deviation.

D. WSJ0-2mix-noisy

The WSJ0-2mix-noisy dataset is equivalent to WSJ0-2mix,but with additional noises added to the mixtures. The noiseswere randomly selected from the CHiME-1 [51] and CHiME-3[52] corpora. The CHiME-1 noises were recorded in a livingroom, and thus contain noises from typical domestic environ-ments and often children’s speech. The CHiME-3 noises arefrom four environments — buses, streets, cafes and pedestrianareas. We split the noises into training and test subsets and mixedthem into the mixtures at SNRs of 20 dB to 0 dB (with respectto the mixture signal).

VI. SYSTEM CONFIGURATION

A. Speaker Extraction Neural Network Settings

In the experiments, we used two different neural networkconfigurations. The first and smaller configuration, is used to

Fig. 4. Final configuration of the neural network for SpeakerBeam with thescaled activations method and sequence summarization with attention. For moredetails, see Equations (8), (11)–(13).

compare the different techniques of informing the neural net-work about the speaker in the SpeakerBeam scheme. For all thefollowing experiments, we used the larger configuration. Thesmall configuration consisted of one BLSTM and three fullyconnected layers. All the layers used ReLU activations and batchnormalization, except for the output layer with logistic sigmoidactivation. The numbers of neurons in the layers were 300-1024-1024-257. The larger configuration consisted of 3 BLSTM lay-ers, each followed by a linear projection layer and one linearoutput layer. The BLSTM layers had 512 units per directionand their output of dimensionality 1024 (512 forward + 512backward) was then transformed by the projection layer backto dimension 512. Each projection layer was followed by tanhnonlinearity. The larger configuration is depicted in Figure 4.For both the factorized layer and scaled activations methods,the second layer was used as speaker adaptive layer. With thefactorized layer, it was split into 30 sub-layers. For the input-biasmethod, the dimension of the appended speaker vector (extractedby the auxiliary network) was 100. The networks were trainedwith an Adam optimizer with a learning rate 1e− 4. With thelarger configuration, we did not use dropout, or batch normal-ization (in contrast with the smaller configuration where batchnormalization was used). The network parameters were initial-ized using the Glorot initialization [53]. The neural networksused for comparison with DC and PIT had the same architectureapart from the last layer, which was I × 257 for PIT (predictingmasks for all the speakers) andD × 257 for DC, whereD = 30is the embedding size.

ŽMOLÍKOVÁ et al.: SPEAKERBEAM: SPEAKER AWARE NEURAL NETWORK FOR TARGET SPEAKER EXTRACTION IN SPEECH MIXTURES 807

B. Speaker Information Extraction Settings

For i-vectors, we used a Kaldi i-vector extractor [54], trainedon clean data. The Universal Background model we usedconsisted of 2048 Gaussians, and the i-vectors were 100-dimensional. The i-vectors were computed per utterance.

As the auxiliary network, we used a network with 2 fullyconnected layers with 200 units per layer and ReLU activations.The output layer had linear activation. Its size was determinedby the method used (see Figure 2). The auxiliary network wastrained jointly with the main network.

C. Beamforming Settings

The beamforming was undertaken in the STFT domain with20 ms windows and a 10 ms shift. We used a GEV beamformeras specified in IV-A. We regularized the noise spatial co-variancematrix by adding 1e−3 to its diagonal to stabilize its inversion.The output signal was post-processed with a single-channel post-filter [42].

D. ASR Settings

The input acoustic features were 40-dimensional log Mel fil-terbanks with a context window extension of 11 frames. Thefeatures were mean-normalized per utterance. For the acousticmodel, we used a simple DNN with 5 fully connected hiddenlayers of 2048 units each and ReLU activation functions. Fortraining, we used HMM tied-state alignments obtained fromsingle-channel clean data using a GMM-HMM system.

VII. EXPERIMENTS

This section provides an experimental evaluation of our ap-proach. We compare the different methods used to inform theneural network about the target speaker and compare the perfor-mance with DC and PIT. We also explore the effectiveness withmixtures containing three speakers. Then, we explore the perfor-mance with noisy and multichannel data. All the experiments areevaluated using the signal-to-distortion ratio (SDR) (as definedby [55] and computed using [56]) or the frequency-weightedsignal-to-noise ratio (fw-SNR) computed using tools providedwith the REVERB challenge [57]. For automatic speech recog-nition experiments, we provide word error rates (WER).

A. Methods for Informing the Network

We compared different methods for informing the networkabout the target speaker and for extracting speaker informationfrom the adaptation utterance as described in Sections III-B,III-C. These experiments were performed on the WSJ0-2mixdataset and used the smaller architecture of the network, as somemethods do not scale well to a larger architecture. Table I showsthe results of the experiments.

We can observe that the input bias adaptation (input-bias)method performs rather poorly. In this case, the neural networkdoes not learn to make proper use of the additional input fea-tures and keeps extracting all speakers present in the mixture.

TABLE ICOMPARISON OF DIFFERENT METHODS FOR INFORMING THE NETWORK

ABOUT THE SPEAKER AND EXTRACTING THE SPEAKER INFORMATION.RESULTS SHOW SDR AND FW-SNR IMPROVEMENTS FOR THE 2MIX

DATASET. IBM STANDS FOR IDEAL BINARY (ORACLE) MASK

Although adapting the bias is a very successful approach to ASRacoustic models adaptation, for our task, it is arguably insuffi-ciently powerful. We confirmed that the poor results are not aconsequence of the smaller architecture of the network by re-peating the input-bias + seqsum + att experiment with the largerarchitecture. This lead to−1.7 dB SDR degradation and−0.9 dBfw-SNR degradation.

The factorized layer (fact-layer) and scaled activations(scaled-act) both yield notably better extraction. The factorizedlayer approach tends to be slightly better, however, this is atthe cost of increased computation and memory demands dueto the many sub-layers. Therefore, for experiments describedin the following sections, we used the scaled activations method,which constitutes a compromise between performance and com-putational cost.

Comparing the different methods of extracting the speaker in-formation, we find that all three methods (ivec, seqsum, seqsum-att) lead to similar results. Training the speaker representationjointly with the network performs slightly better. Although theattention does not significantly improve the performance, weobserved that the learned attention weights properly detect thenon-silent parts of the adaptation utterance, which could be help-ful when the adaptation utterances contain larger amounts of si-lence or noise. We therefore retained the attention mechanismfor the following experiments.

B. Comparison With DC and PIT

To better evaluate the ability of SpeakerBeam to extract atarget speaker, we compare its performance with Deep clusteringand Permutation invariant training. For these experiments, weuse the larger architecture, which is similar to settings used inpreviously published work on DC [10], [25] and PIT [27]. Notethat with PIT and DC, the outputs are assigned to individualspeakers in an oracle way, i.e. we choose an assignment thatminimizes the error. With SpeakerBeam, we extract each of thespeakers by providing the network with the speaker information,and the assignment is thus decided within the method. For afairer comparison, we could consider coupling DC and PIT with

808 IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING, VOL. 13, NO. 4, AUGUST 2019

TABLE IICOMPARISON OF THE SDR IMPROVEMENTS [DB] WITH WSJ0-2MIX AND

WSJ0-2MIX-LONG DATASETS FOR SPEAKERBEAM, DC AND PIT.FOR DC AND PIT, WE USE ORACLE PERMUTATIONS OF THE

SEPARATED SOURCES FOR EVALUATION

a speaker identification module, possibly introducing additionalerrors. However, we adhere to using the oracle assignment toinspect the upper bound of such an extraction.

The first set of experiments compares performance for the2mix dataset, which is commonly used to evaluate speech sep-aration methods. Table II shows that the results for this datasetare comparable, with Deep clustering performing slightly worsethan the other two methods. Previously published work on DC[58] achieved an SDR improvement of 9.4 dB with a similarnetwork architecture. The main differences between [58] andour setup are the optimization schedule and dropout regulariza-tion. Tuning these training settings could thus lead to improvedaccuracy.

The last experiment in the first part of Table II shows thatwe can combine Deep clustering and SpeakerBeam. For thisexperiment, the SpeakerBeam architecture was extended by anadditional output layer with a Deep clustering objective. Thisadditional loss can serve to better train the network, while dur-ing evaluation, this output is discarded. The results show thatthe combination indeed helps with training, and the accuracysurpasses both SpeakerBeam and DC when used individually.

The second part of Table II shows the performance with theWSJ0-2mix-long dataset with longer, less overlapped mixtures.For these mixtures, we used networks trained for the WSJ0-2mixand refined them using random 10-second excerpts from theWSJ0-2mix-long training data. The network could thus learn toprocess segments with no or a partial overlap. The results showthat for these data, SpeakerBeam performs better. The degra-dation of DC and PIT compared with SpeakerBeam originatesfrom the errors in tracing the speaker correctly over time; insome mixtures, the speakers on the output are switched in themiddle of the utterance as shown in the example in Figure 5.The outputs of DC and PIT would require further processing fortracing the speakers over the utterance, whereas SpeakerBeamdoes this jointly with the extraction. We can speculate that suchbehavior would appear more frequently with even longer mix-tures or more speakers. Combining the DC and SpeakerBeamobjectives during training again leads to a performance gain.

C. Three Speaker Experiments

Table III shows the results of the extraction when appliedto mixtures with 3 speakers. Since the neural network in

Fig. 5. Example of a mixture from the WSJ0-2mix-long dataset as processedby SpeakerBeam and Deep Clustering methods.

TABLE IIIRESULTS OF PERFORMING EXTRACTION ON MIXTURES OF TWO AND THREE

SPEAKERS, USING WSJ0-2MIX AND WSJ0-3MIX DATASETS. THE

RESULTS ARE IN TERMS OF SDR IMPROVEMENTS [DB]

SpeakerBeam is independent of the number of speakers in themixture, we can train the same network for both 2-speaker and3-speaker data. Table III compares the performance for both2-speaker and 3-speaker mixtures with different training sets.The results show that a network trained only on 2-speaker mix-tures does not generalize very well to 3-speaker mixtures. If wetrain only on 3-speaker mixtures, the network can extract speak-ers from both 2- and 3-speaker mixtures with a reasonable levelof performance. For the 2-speaker mixtures, there is still a gap inaccuracy compared with matched training. The use of all the datafor training leads to good performance with both 2 and 3 speakermixtures. We performed the same set of experiments with DCand PIT. For DC, we used the oracle number of speakers duringthe clustering step. For PIT, we used a network with 3 outputs.For 2-speaker mixtures, during the training, we considered one

ŽMOLÍKOVÁ et al.: SPEAKERBEAM: SPEAKER AWARE NEURAL NETWORK FOR TARGET SPEAKER EXTRACTION IN SPEECH MIXTURES 809

TABLE IVRESULTS OF AUTOMATIC SPEECH RECOGNITION WITH WSJ0-2MIX IN TERMS

OF WORD ERROR RATE USING SINGLE-CHANNEL RECORDINGS

TABLE VRESULTS OF AUTOMATIC SPEECH RECOGNITION ON WSJ0-2MIX IN

TERMS OF WORD ERROR RATE USING MULTI-CHANNEL

RECORDINGS AND BEAMFORMING

of the outputs to be silent channel, and during testing, we keptonly two outputs with the most energy. This follows the proce-dure described in [27]. The results of both PIT and DC showsimilar trend as with SpeakerBeam, with even slightly worsegeneralization from network trained on 2 speakers to 3-speakermixtures, especially with PIT.

D. Automatic Speech Recognition Experiments

Table IV shows the results we obtained for the automaticspeech recognition of the extracted speech in the WSJ0-2mixdataset. Note that in these experiments, to allow for ASR evalu-ation, we used the max version of the dataset, where the lengthof the mixture corresponds to the length of the longer of the twoutterances. By contrast, in the previous experiments the mix-tures were cut to the length of the shorter utterance. The singlespeaker and mixtures results show the lower and upper boundsof the error. In all the experiments, the speech recognition sys-tem is trained on matched training data (single-speaker, mixtureor processed with SpeakerBeam, PIT or DC). We can see thatSpeakerBeam significantly reduces the error compared with theoriginal mixtures and can thus work as a front-end for an ASRsystem. Additionally, we also processed single speaker data withSpeakerBeam, to see how much the processing degrades thespeech when there is no overlap. The result shows degradationfrom 12.2% to 15.3% WER. In this case, SpeakerBeam wasnot trained on single-speaker data, such training could possiblyreduce the performance gap.

E. Multi-Channel Experiments

The experimental results in Table V show the ASR perfor-mance with the multi-channel dataset WSJ0-2mix-MC. Notethat these results cannot be directly compared with Table IV asWSJ0-2mix-MC contains much more reverberation. Speaker-Beam is used here in combination with a GEV beamformer as

TABLE VIRESULTS OF EXTRACTION FOR DATASET WITH ADDITIONAL NOISE

IN TERMS OF SDR IMPROVEMENTS

TABLE VIIRESULTS OF EXTRACTION FOR DATASET WITH ADDITIONAL NOISE

IN TERMS OF SPEECH RECOGNITION ERRORS

described in Section IV-A. The use of the beamformer, whichemploys the SpeakerBeam output, improves the accuracy of theASR system to 22.5% WER.

In addition, training the front-end jointly with the ASR usingthe cross-entropy objective further improves the results. In thiscase, the SpeakerBeam network is initialized with the networktrained with the mask objective (Eq. (14)) and the acoustic modelwith the network trained on the data enhanced by SpeakerBeam.Both networks are then jointly fine-tuned with the final ASRobjective. The masks extracted with the front-end tend to besparser when trained for the ASR objective which may be moreconvenient for further processing with beamforming.

F. Noisy Data

Tables VI,VII show the results of experiments on WSJ0-2mix-noisy. For all the experiments (PIT, DC, SpkBeam), the networksare trained on a training set, where each mixture contains ad-ditional noise with a randomly selected SNR as described inSection V-D. For testing, we created several copies of the test-setwith various levels of noise ranging from 20 to 0 dB. We can seethat even with quite high levels of unstationary noises, Speaker-Beam still succeeds in extracting the target speaker and im-proves both the signal-level measure and the ASR performance.For more results on noisy and reverberant mixtures, reader canalso refer to our study in [59] or our demo video [60]. Notethat although the presented experiments use noises recordedin real environments, the mixtures consist of fully overlappedspeech, thus may not well reflect the nature of real conversations.The application of speech extraction methods in real conditionsis an important issue which we plan to investigate in futurework.

810 IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING, VOL. 13, NO. 4, AUGUST 2019

Fig. 6. t-SNE of derived speaker representations. The clusters correspond tospeakers in the test data.

Fig. 7. Matrix of pairwise distances of derived speaker representations.

VIII. ANALYSIS OF LEARNED BEHAVIOR

A. Learned Speaker Embeddings

The auxiliary network in the SpeakerBeam architectureshould convey information about the speaker from the adap-tation utterance to the main network performing the speakerextraction. However, the auxiliary network is never trained witha direct speaker-related objective, only with the final objectiveof the speaker extraction. In this section, we explore how thelearned vectors for the output of the auxiliary network capturethe speaker information. Figure 6 shows the embeddings ob-tained from the adaptation utterances in test data, projected intotwo dimensions by means of t-SNE [61]. We can see that the vec-tors form 18 clusters corresponding to the 18 speakers in the testdata. Note that there is no overlap between the speakers in thetraining and evaluation sets. The auxiliary network thus seemsto generalize well to unseen speakers. The same conclusionscan also be drawn from Figure 7, which shows the pair-wiseEuclidean distances of the embeddings. Apart from the distinct

speaker clusters, we can also see two main categories of speak-ers corresponding to males and females; the gender representsimportant variability in the embedding space.

B. Analysis of Performance Per Speaker

The speaker characteristics are arguably a big factor in the per-formance of speaker extraction. In this Section, we inspect moreclosely the accuracy of the method for different speakers. First,we examine whether the accuracy varies greatly for differenttarget speakers in the dataset. We used the WSJ0-2mix datasetand the larger neural network architecture for this analysis (cor-responds to the 9.7 dB improvement in Table II). Figure 8 showsthat the mean SDR improvement does not vary very significantlyfor different target speakers, with a minimum of 8.0 dB meanSDR improvement for speaker ’423’ and a maximum of 11.2 dBmean SDR improvement for speaker ’442’. A greater variationcan be observed in the results, if we consider the impact of thecombination of two speakers in the mixture. In Figure 9, weshow the mean SDR improvements for different combinationsof target and interfering speakers for SpeakerBeam, PIT and DC.Again, we can see two main groups of speakers correspondingto gender. Mixtures of same-gender speakers tend to be muchmore difficult to separate. Overall, the mean SDR improvementon same-gender mixtures is 7.2 dB, while for different-gendermixtures, it is 11.9 dB (For PIT, the SDR improvements are6.3 dB and 11.8 dB and for DC, 5.9 dB and 10.9 dB for same-gender and different-gender mixtures respectively). We can seea few speaker pairs where the method is unable to differentiatebetween the speakers sufficiently well and the improvements areclose to zero. By comparison with Figure 7, these correspondto cases where the extracted speaker embeddings are very sim-ilar. We believe that the ability to differentiate between thesespeakers would improve by training SpeakerBeam with a largerspeaker variability in the training set (the WSJ0-2mix datasetwe used comprises 101 training speakers).

C. Impact the Adaptation Utterance Length

In all of our experiments, the average length of an adaptationutterance was about 6 seconds. However, for some applications,it might be more convenient to use shorter utterances. In Fig-ure 10, we thus further analyze the impact the length of theadaptation utterance has on the accuracy of the separation. Forthis analysis, we used the WSJ0-2mix dataset and assigned eachtest utterance an adaptation utterance of longer than 8.5 seconds.All the adaptation utterances were then cut to different lengthsof 0.5 to 8 seconds and used as an input to the auxiliary network.During the cutting, we also removed the initial 0.5 seconds of theutterances to avoid an initial silence. The plot shows the averageSDR improvements achieved using these shortened adaptationdata. For an adaptation utterance of longer than 2.5 seconds, theperformance saturates. Already at 1 second, the accuracy of theextraction is fairly close to that of the longer utterances. Withless speech, the performance deteriorates, however even with0.5 seconds of adaptation data, SpeakerBeam manages to im-prove the SDR compared with the mixtures. Note that thesetendencies may be highly dependent on the training data.

ŽMOLÍKOVÁ et al.: SPEAKERBEAM: SPEAKER AWARE NEURAL NETWORK FOR TARGET SPEAKER EXTRACTION IN SPEECH MIXTURES 811

Fig. 8. SDR improvements for different speakers in the WSJ0-2mix testing set. The numbers at the top of the figure are the mean SDR improvements for eachspeaker. The violin plot shows the distribution shape, maximum, minimum and mean SDR improvement over the utterances from the target speaker.

Fig. 9. Mean SDR improvements for different target-interfering speaker combinations in the WSJ0-2mix testing set. Speakers are sorted by gender. Speakers051 to 447 are male, speakers 050 to 445 are female.

Fig. 10. Impact of the adaptation utterance length on SDR improvement.

IX. CONCLUSION

In this paper, we introduced the SpeakerBeam method for ex-tracting a target speaker from a mixture of multiple overlappingspeakers based on informing the neural network about the tar-get speaker using additional speaker information. We compareddifferent methods for informing the neural network. The resultsshow that the scaled-activations and factorized-layer methodsare more suitable than simply appending the speaker information

to the input. We compared the method to Deep Clustering andPermutation invariant training, where we observed comparableperformance for short, fully overlapped mixtures and the advan-tage of SpeakerBeam for longer mixtures with more complicatedoverlapping patterns. This is due to the ability of SpeakerBeamto better track the speaker over time. Furthermore, the methodcan be also combined with Deep Clustering for further gains. Inaddition to using our method with single-channel 2-speaker mix-tures, we also showed its ability to handle 3-speaker mixturesand the possibility of extending the method to multi-channel pro-cessing and joint training with an automatic speech recognitionsystem.

In future work, we plan to explore the effect of using largerdatasets, especially with higher numbers of speakers, to furtherimprove learned speaker representations and extraction accu-racy. Another possible direction involves combining Speaker-Beam with existing speaker diarization approaches and testingits performance on speaker diarization tasks.

REFERENCES

[1] S. Makino, T.-W. Lee, and H. Sawada, Blind Speech Separation, vol. 615.Berlin, Germany: Springer, 2007.

[2] Y.-M. Qian, C. Weng, X.-K. Chang, S. Wang, and D. Yu, “Past review,current progress, and challenges ahead on the cocktail party problem,”Frontiers Inf. Technol. Electron. Eng., vol. 19, no. 1, pp. 40–63, Jan. 2018.

[3] G. J. Brown and M. Cooke, “Computational auditory scene analysis,”Comput. Speech Lang., vol. 8, no. 4, pp. 297–336, 1994.

812 IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING, VOL. 13, NO. 4, AUGUST 2019

[4] D. P. W. Ellis, “Prediction-driven computational auditory scene analysis,”Ph.D. dissertation, Dept. Elect. Eng. Comput., Massachusetts Inst. Tech-nol., Cambridge, MA, USA, 1996.

[5] D. D. Lee and H. S. Seung, “Algorithms for non-negative matrix factor-ization,” in Adv. Neural Inf. Process. Syst., 2001, pp. 556–562.

[6] T. Virtanen, “Monaural sound source separation by nonnegative matrixfactorization with temporal continuity and sparseness criteria,” IEEE/ACMTrans. Audio, Speech, Lang. Process., vol. 15, no. 3, pp. 1066–1074, Mar.2007.

[7] J. R. Hershey, S. J. Rennie, P. A. Olsen, and T. T. Kristjansson, “Super-human multi-talker speech recognition: A graphical modeling approach,”Comput. Speech Lang., vol. 24, no. 1, pp. 45–66, 2010.

[8] S. J. Rennie, J. R. Hershey, and P. A. Olsen, “Single-channel multitalkerspeech recognition,” IEEE Signal Process. Mag., vol. 27, no. 6, pp. 66–80,Nov. 2010.

[9] D. Wang and J. Chen, “Supervised speech separation based on deep learn-ing: An overview,” IEEE/ACM Trans. Audio, Speech, Lang. Process.,vol. 26, no. 10, pp. 1702–1726, Oct. 2018.

[10] J. R. Hershey, Z. Chen, J. L. Roux, and S. Watanabe, “Deep clustering: Dis-criminative embeddings for segmentation and separation,” in Proc. IEEEInt. Conf. Acoust., Speech, Signal Process., 2016, pp. 31–35.

[11] D. Yu, M. Kolbæk, Z.-H. Tan, and J. Jensen, “Permutation invariant train-ing of deep models for speaker-independent multi-talker speech separa-tion,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2017,pp. 241–245.

[12] R. Maas, S. H. K. Parthasarathi, B. King, R. Huang, and B.Hoffmeister, “Anchored speech detection,” in Proc. Interspeech, 2016,pp. 2963–2967.

[13] B. King et al., “Robust speech recognition via anchor word representa-tions,” in Proc. Interspeech, 2017, pp. 2471–2475.

[14] S. H. Mallidi, R. Maas, K. Goehner, A. Rastrow, A. Matsoukas, and B.Hoffmeister, “Device-directed utterance detection,” in Proc. Interspeech,2018, pp. 1225–1228.

[15] J. Wang et al., “Deep extractor network for target speaker recovery fromsingle channel speech mixtures,” in Proc. Interspeech, 2018, pp. 307–311.

[16] J. Du, Y. Tu, Y. Xu, L. Dai, and C.-H. Lee, “Speech separation of a targetspeaker based on deep neural networks,” in Proc. IEEE 12th Int. Conf.Signal Process., 2014, pp. 473–477.

[17] X.-L. Zhang and D. Wang, “A deep ensemble learning method for monau-ral speech separation,” IEEE/ACM Trans. Audio, Speech, Lang. Process.,vol. 24, no. 5, pp. 967–977, May 2016.

[18] J. Du, Y. Tu, L.-R. Dai, and C.-H. Lee, “A regression approach tosingle-channel speech separation via high-resolution deep neural net-works,” IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 24, no. 8,pp. 1424–1437, Aug. 2016.

[19] G. Saon, H. Soltau, D. Nahamoo, and M. Picheny, “Speaker adaptation ofneural network acoustic models using i-vectors,” in Proc. IEEE WorkshopAutom. Speech Recognit. Understanding, 2013, pp. 55–59.

[20] A. Senior and I. Lopez-Moreno, “Improving DNN speaker independencewith i-vector inputs,” in Proc. IEEE Int. Conf. Acoust., Speech, SignalProcess., 2014, pp. 225–229.

[21] K. Vesely, S. Watanabe, K. Zmolikova, M. Karafiat, L. Burget, and J.H. Cernocky, “Sequence summarizing neural network for speaker adap-tation,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2016,pp. 5315–5319.

[22] M. Delcroix, K. Kinoshita, A. Ogawa, C. Huemmer, and T. Nakatani,“Context adaptive neural network based acoustic models for rapid adap-tation,” IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 26, no. 5,pp. 895–908, May 2018.

[23] P. Swietojanski and S. Renals, “Learning hidden unit contributions forunsupervised speaker adaptation of neural network acoustic models,” inProc. IEEE Spoken Lang. Technol. Workshop, 2014, pp. 171–176.

[24] N. Dehak, P. J. Kenny, R. Dehak, P. Dumouchel, and P. Ouellet, “Front-endfactor analysis for speaker verification,” IEEE/ACM Trans. Audio, Speech,Lang. Process., vol. 19, no. 4, pp. 788–798, May 2011.

[25] Y. Isik, J. L. Roux, Z. Chen, S. Watanabe, and J. R. Hershey, “Single-channel multi-speaker separation using deep clustering,” in Proc. Inter-speech, 2016, pp. 545–549.

[26] Z. Chen, Y. Luo, and N. Mesgarani, “Deep attractor network for single-microphone speaker separation,” in Proc. IEEE Int. Conf. Acoust., Speech,Signal Process, 2017, pp. 246–250.

[27] M. Kolbæk et al., “Multitalker speech separation with utterance-level per-mutation invariant training of deep recurrent neural networks,” IEEE/ACMTrans. Audio, Speech, Lang. Process., vol. 25, no. 10, pp. 1901–1913, Oct.2017.

[28] K. Zmolikova, M. Delcroix, K. Kinoshita, T. Higuchi, A. Ogawa, andT. Nakatani, “Speaker-aware neural network based beamformer forspeaker extraction in speech mixtures,” in Proc. Interspeech, Aug. 2017,pp. 2655–2659.

[29] K. Žmolíková, M. Delcroix, K. Kinoshita, T. Higuchi, A. Ogawa, andT. Nakatani, “Learning speaker representation for neural network basedmultichannel speaker extraction,” in Proc. IEEE Autom. Speech Recognit.Understanding Workshop, 2017, pp. 8–15.

[30] K. Zmolikova, M. Delcroix, K. Kinoshita, T. Higuchi, T. Nakatani, and J.Cernocký, “Optimization of speaker-aware multichannel speech extractionwith ASR criterion,” in Proc. IEEE Int. Conf. Acoust., Speech, SignalProcess., 2018, pp. 6702–6706.

[31] M. Delcroix, K. Zmolikova, K. Kinoshita, A. Ogawa, and T. Nakatani,“Single channel target speaker extraction and recognition with speakerbeam,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2018,pp. 5554–5558.

[32] L. Drude, T. von Neumann, and R. Haeb-Umbach, “Deep attractor net-works for speaker re-identification and blind source separation,” in Proc.IEEE Int. Conf. Acoust., Speech, Signal Process., 2018, pp. 11–15.

[33] Q. Wang et al., “Voicefilter: Targeted voice separation by speaker-conditioned spectrogram masking,” arXiv preprint arXiv:1810.04826,Oct. 2018.

[34] L. Wan, Q. Wang, A. Papir, and I. L. Moreno, “Generalized end-to-endloss for speaker verification,” in Proc. IEEE Int. Conf. Acoust., Speech,Signal Process., 2018, pp. 4879–4883.

[35] M. Delcroix, K. Kinoshita, T. Hori, and T. Nakatani, “Context adaptivedeep neural networks for fast acoustic model adaptation,” in Proc. IEEEInt. Conf. Acoust., Speech, Signal Process., 2015, pp. 4535–4539.

[36] L. Samarakoon and K. C. Sim, “Subspace LHUC for fast adaptationof deep neural network acoustic models,” in Proc. Interspeech, 2016,pp. 1593–1597.

[37] M. Karafiát, L. Burget, P. Matejka, O. Glembek, and J. Cernocký, “ivector-based discriminative adaptation for automatic speech recognition,” inProc. IEEE Workshop Autom. Speech Recognit. Understanding, 2011,pp. 152–157.

[38] P. Kenny, D. Reynolds, and F. Castaldo, “Diarization of telephone conver-sations using factor analysis,” IEEE J. Sel. Topics Signal Process., vol. 4,no. 6, pp. 1059–1070, Dec. 2010.

[39] O. Glembek, L. Burget, P. Matejka, M. Karafiát, and P. Kenny, “Simpli-fication and optimization of i-vector extraction,” in Proc. IEEE Int. Conf.Acoust., Speech, Signal Process., 2011, pp. 4516–4519.

[40] Y. Wang, A. Narayanan, and D. Wang, “On training targets for super-vised speech separation,” IEEE/ACM Trans. Audio, Speech, Lang. Pro-cess., vol. 22, no. 12, pp. 1849–1858, Dec. 2014.

[41] Y. Luo, Z. Chen, J. R. Hershey, J. L. Roux, and N. Mesgarani, “Deepclustering and conventional networks for music separation: Stronger to-gether,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2017,pp. 61–65.

[42] E. Warsitz and R. Haeb-Umbach, “Blind acoustic beamforming based ongeneralized eigenvalue decomposition,” IEEE/ACM Trans. Audio, Speech,Lang. Process., vol. 15, no. 5, pp. 1529–1539, Jul. 2007.

[43] K. Kumatani, J. McDonough, and B. Raj, “Microphone array processingfor distant speech recognition: From close-talking microphones to far-field sensors,” IEEE Signal Process. Mag., vol. 29, no. 6, pp. 127–140,Nov. 2012.

[44] J. Heymann, L. Drude, and R. Haeb-Umbach, “Neural network basedspectral mask estimation for acoustic beamforming,” in Proc. IEEE Int.Conf. Acoust., Speech, Signal Process., 2016, pp. 196–200.

[45] H. Erdogan, J. R. Hershey, S. Watanabe, M. Mandel, and J. L. Roux,“Improved MVDR beamforming using single-channel mask predictionnetworks,” in Proc. Interspeech, 2016, pp. 1981–1985.

[46] C. Böddeker, P. Hanebrink, L. Drude, J. Heymann, and R. Haeb-Umbach,“Optimizing neural-network supported acoustic beamforming by algo-rithmic differentiation,” in Proc. IEEE Int. Conf. Acoust., Speech, SignalProcess., Mar. 2017, pp. 171–175.

[47] J. Garofolo, “CSR-I (WSJ0) complete LDC93S6A,” 1993. [Online]. Avail-able: https://catalog.ldc.upenn.edu/ldc93s6a

[48] Z.-Q. Wang, J. L. Roux, and J. R. Hershey, “Multi-channel deep clustering:Discriminative spectral and spatial embeddings for speaker-independentspeech separation,” Proc. IEEE Int. Conf. Acoust., Speech, Signal Process.,2018, pp. 1–5.

[49] J. B. Allen and D. A. Berkley, “Image method for efficiently simulatingsmall-room acoustics,” J. Acoust. Soc. Am., vol. 65, no. 4, pp. 943–950,1979.

ŽMOLÍKOVÁ et al.: SPEAKERBEAM: SPEAKER AWARE NEURAL NETWORK FOR TARGET SPEAKER EXTRACTION IN SPEECH MIXTURES 813

[50] E. A. P. Habets, “Room impulse response generator,” Technische Uni-versiteit Eindhoven, Eindhoven, The Netherlands, Tech. Rep. 2.2.4,2010. [Online]. Available: http://home.tiscali.nl/ehabets/rir_generator/rir_generator.pdf

[51] J. Barker, E. Vincent, N. Ma, H. Christensen, and P. Green, “The PASCALCHiME speech separation and recognition challenge,” Comput. SpeechLang., vol. 27, no. 3, pp. 621–633, May 2013.

[52] J. Barker, R. Marxer, E. Vincent, and S. Watanabe, “The third CHiMEspeech separation and recognition challenge: Dataset, task and baselines,”in Proc. IEEE Workshop Autom. Speech Recognit. Understanding, 2015,pp. 504–511.

[53] X. Glorot and Y. Bengio, “Understanding the difficulty of training deepfeedforward neural networks,” in Proc. 13th Int. Conf. Artif. Intell. Statist.,2010, pp. 249–256.

[54] D. Povey et al., “The Kaldi speech recognition toolkit,” in Proc. IEEEWorkshop Autom. Speech Recognit. Understanding, 2011, Paper EPFL-CONF-192584.

[55] E. Vincent, R. Gribonval, and C. Févotte, “Performance measurement inblind audio source separation,” IEEE/ACM Trans. Audio, Speech, Lang.Process., vol. 14, no. 4, pp. 1462–1469, Jul. 2006.

[56] C. Raffel et al., “mir_eval: A transparent implementation of commonMIR metrics,” in Proc. 15th Int. Soc. Music Inf. Retrieval Conf., 2014,pp. 367–372.

[57] K. Kinoshita et al., “The REVERB challenge: A common evaluationframework for dereverberation and recognition of reverberant speech,” inProc. IEEE Workshop Appl. Signal Process. Audio Acoust., 2013, pp. 1–4.

[58] Y. Isik, J. L. Roux, Z. Chen, S. Watanabe, and J. R. Hershey, “Single-channel multi-speaker separation using deep clustering,” in Proc. Inter-speech, 2016, pp. 545–549.

[59] M. Delcroix, K. Zmolikova, T. Ochiai, K. Kinoshita, S. Araki, and T.Nakatani, “Compact network for speakerbeam target speaker extrac-tion,” in Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., 2019,pp. 6965–6969.

[60] “SpeakerBeam.” [Online]. Available: https://youtu.be/7FSHgKip6vI(English), https://youtu.be/BM0DXWgGY5A (Japanese).

[61] L. V. D. Maaten and G. Hinton, “Visualizing data using t-SNE,” J. Mach.Learn. Res., vol. 9, pp. 2579–2605, 2008.

Katerina Žmolíková received the B.Sc. degree ininformation technology in 2014 and the Ing. degreein mathematical methods in information technologyin 2016 from the Faculty of Information Technology,Brno University of Technology (BUT), Brno, CzechRepublic, where she is currently working toward thePh.D. degree. Since 2013, she has been part of theSpeech@FIT research group at BUT. She is the Ph.D.Talent scholarship holder. Her research interests in-clude robust speech recognition, speech separation,and deep learning.

Marc Delcroix (M’05–SM’16) received the M.Eng.degree from the Free University of Brussels,Brussels, Belgium, and Ecole Centrale Paris, Paris,France, in 2003, and the Ph.D. degree from the Grad-uate School of Information Science and Technol-ogy, Hokkaido University, Sapporo, Japan, in 2007.He is currently a Distinguished Researcher with theMedia Information Laboratory, Signal ProcessingGroup, NTT Communication Science Laboratories,NTT Corporation, Kyoto, Japan. From 2007 to 2008and 2010 to 2012, he was a Research Associate with

the NTT Communication Science Laboratories, where he became a permanentResearch Scientist in 2012. He was also a Visiting Lecturer with the Facultyof Science and Engineering, Waseda University, Tokyo, Japan, from 2015 to2018. His research interests include robust speech recognition, acoustic modeladaptation, and speech enhancement. He took an active part in the developmentof NTT robust speech recognition systems for the REVERB and the CHiME1 and 3 challenges. He was the recipient of several research awards includingthe 2006 Sato Paper Award from ASJ, the 2015 IEEE-ASRU Best Paper AwardHonorable Mention, and the 2016 ASJ Awaya Young Researcher Award. Hewas part of the organizing committee of the REVERB Challenge/Workshop2014 and ASRU 2017 and is a Member of the IEEE SPS Speech and LanguageProcessing Technical Committee. He is a member of ASJ.

Keisuke Kinoshita (M’05) received the M.Eng. andPh.D. degrees from Sophia University, Tokyo, Japan,in 2003 and 2010, respectively. He is currently a Dis-tinguished Researcher with the NTT Communica-tion Science Laboratories, NTT Corporation, Kyoto,Japan. After joining the NTT Communication Sci-ence Laboratories in 2003, he has been engaged in theresearch on various types of speech, audio, and mu-sic signal processing, including speech enhancementsuch as 1ch/multichannel blind dereverberation, noisereduction, source separation, distributed microphone

array processing, and robust speech recognition. He has authored or coauthoredmore than 100 technical papers in refereed journals and conference proceedings.He also contributed to five book chapters. He was a Chief Coordinator of theREVERB Challenge 2014, an Associate Editor of IEICE Transactions on Fun-damentals of Electronics, Communications and Computer Sciences from 2013to 2015, and has been a Member of IEEE AASP TC since 2018. He was therecipient of the 2006 IEICE Paper Award, the 2009 ASJ Outstanding TechnicalDevelopment Prize, the 2011 ASJ Awaya Prize, the 2012 Japan Audio SocietyAward, the 2015 IEEE-ASRU Best Paper Award Honorable Mention, and 2017Maeshima Hisoka Award. He is a member of ASJ and IEICE.

Tsubasa Ochiai received the B.E., M.E., and Ph.D.degrees in information engineering from DoshishaUniversity, Kyoto, Japan, in 2013, 2015, and 2018,respectively. Since 2018, he has been a Researcherwith the Communication Science Laboratories, NTTCorporation, Kyoto, Japan. His research interests in-clude speech recognition, speech enhancement, anddeep learning. He is a member of the IEICE and ASJ.

Tomohiro Nakatani (M’03–SM’06) received B.E.,M.E., and Ph.D. degrees from Kyoto University,Kyoto, Japan, in 1989, 1991, and 2002, respectively.He is currently a Senior Distinguished Researcherwith NTT Communication Science Laboratories,NTT Corporation, Kyoto, Japan. Since joining NTTCorporation as a Researcher in 1991, he has beeninvestigating audio signal processing technologiesfor intelligent human-machine interfaces, includingdereverberation, denoising, source separation, and ro-bust ASR. He was the recipient of the 2005 IEICE

Best Paper Award, the 2009 ASJ Technical Development Award, the 2012 JapanAudio Society Award, the 2015 IEEE ASRU Best Paper Award Honorable Men-tion, the 2017 Maejima Hisoka Award, and the 2018 IWAENC Best Paper Award.He was a Visiting Scholar with the Georgia Institute of Technology for a yearfrom 2005 and he was a Visiting Assistant Professor with the Department ofMedia Science, Nagoya University, from 2008 to 2017. He served as an Asso-ciate Editor for the IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE

PROCESSING from 2008 to 2010. He was a member of the IEEE Signal Process-ing Society (SPS) Audio and Acoustics Technical Committee (AASP-TC) from2009 to 2014 and served as the Review Subcommittee Chair for the TC from2013 to 2014. He has been an associate member of the AASP-TC since 2015 anda member of the IEEE SPS Speech and Language Processing Technical Com-mittee since 2016. He was also a member of IEEE CAS Society Blind SignalProcessing Technical Committee from 2007 to 2009. He served as the Chair ofthe IEEE Kansai Section Technical Program Committee from 2011 to 2012, andhe has been serving as the Chair of the IEEE SPS Kansai Chapter since 2019. Hewas a Technical Program Co-Chair of the IEEE WASPAA-2007, a Co-Chair ofthe 2014 REVERB Challenge Workshop, and a General Co-Chair of the IEEEASRU-2017. He is a member of IEICE and ASJ.

814 IEEE JOURNAL OF SELECTED TOPICS IN SIGNAL PROCESSING, VOL. 13, NO. 4, AUGUST 2019

Lukáš Burget is currently an Assistant Professorwith the Faculty of Information Technology, BrnoUniversity of Technology, Brno, Czech Republic,and the Research Director of the BUT Speech@FITgroup. He was a Visiting Researcher with OGI Port-land, USA, and with SRI International, Menlo Park,CA, USA. His scientific interests are in the field ofspeech data mining, concentrating on acoustic mod-eling for speech, speaker, and language recognition,including their software implementations. Dr. Burgetwas on numerous EU- and US-funded projects, was

the PI of US-Air Force EOARD project and BUTs PI in IARPA BEST.

Jan (Honza) Cernocký (M’2001–SM’2008) is cur-rently an Associate Professor and the Head of theDepartment of Computer Graphics and Multime-dia, BUT Faculty of Information Technology (FIT).He also serves as a Managing Director of BUTSpeech@FIT research group. His research interestsinclude artificial intelligence, signal processing, andspeech data mining (speech, speaker, and languagerecognition). He is responsible for signal and speechprocessing courses at FIT BUT. In 2006, he co-founded Phonexia. He is the General Chair of Inter-

speech 2021 in Brno.