Multi-label classification with weighted classifier selection and stacked ensemble Yuelong Xia a , Ke Chen b , Yun Yang c,⇑ a School of Information Science & Engineering, Yunnan University, 650091, China b School of Computer Science, The University of Manchester, Manchester M13 9PL, UK c National Pilot School of Software, Yunnan University, 650091, China article info Article history: Received 22 September 2019 Received in revised form 3 June 2020 Accepted 6 June 2020 Available online xxxx Keywords: Multi-label classification Stacked ensemble Label correlation Base classifier selection Regularization via sparsity abstract Multi-label classification has attracted increasing attention in various applications, such as medical diagnosis and semantic annotation. With such trend, a large number of ensemble approaches have been proposed for multi-label classification tasks. Most of these approaches construct the ensemble members by using bagging schemes, but few stacked ensemble approaches are developed. Existing research on stacked ensemble approaches remains active, but several issues remain such as (1) little has been done to learn the weights of classifiers for classifier selection; (2) the relationship between pairwise label correlations and multi-label classification performance has not been investigated suffi- ciently. To address these issues, we propose a novel stacked ensemble approach that simul- taneously exploits label correlations and the process of learning weights of ensemble members. In our approach, first, a weighted stacked ensemble with sparsity regularization is developed to facilitate classifier selection and ensemble members construction for multi- label classification. Second, in order to improve the classification performance, the pairwise label correlations are further considered for determining weights of these ensemble mem- bers. Finally, we develop an optimization algorithm based on both of the accelerated prox- imal gradient and the block coordinate descent techniques to achieve the optimal ensemble solution efficiently. Extensive experiments on publicly available datasets and real Cardiovascular and Cerebrovascular Disease datasets demonstrate that our proposed algorithm outperforms related state-of-the-art methods from perspectives of benchmark- ing and real-world applications. Ó 2020 Elsevier Inc. All rights reserved. 1. Introduction Multi-label learning has been widely used in various applications, such as text categorization [1], semantic annotation [2] and medical diagnosis [3], where each example can be associated with multiple class labels simultaneously. It is different from single-label classification tasks that multi-label classification can be affected by intrinsic latent label correlations. For example, a patient with high blood pressure is more likely to develop a heart disease than one with normal blood pres- sure but is less likely to develop a neuromuscular disease [4]. A significant number of algorithms have been proposed to solve multi-label classification problems [5,6]. These algo- rithms can be divided into two categories [7], namely problem transformation methods and algorithm adaptation methods. https://doi.org/10.1016/j.ins.2020.06.017 0020-0255/Ó 2020 Elsevier Inc. All rights reserved. ⇑ Corresponding author. E-mail address: [email protected](Y. Yang). Information Sciences xxx (xxxx) xxx Contents lists available at ScienceDirect Information Sciences journal homepage: www.elsevier.com/locate/ins Please cite this article as: Y. Xia, K. Chen and Y. Yang, Multi-label classification with weighted classifier selection and stacked ensemble, Information Sciences, https://doi.org/10.1016/j.ins.2020.06.017
Transcript
Information Sciences xxx (xxxx) xxx
Contents lists available at ScienceDirect
Information Sciences
journal homepage: www.elsevier .com/locate / ins
Multi-label classification with weighted classifier selection andstacked ensemble
https://doi.org/10.1016/j.ins.2020.06.0170020-0255/� 2020 Elsevier Inc. All rights reserved.
Please cite this article as: Y. Xia, K. Chen and Y. Yang, Multi-label classification with weighted classifier selection and stacked enInformation Sciences, https://doi.org/10.1016/j.ins.2020.06.017
Yuelong Xia a, Ke Chen b, Yun Yang c,⇑a School of Information Science & Engineering, Yunnan University, 650091, Chinab School of Computer Science, The University of Manchester, Manchester M13 9PL, UKcNational Pilot School of Software, Yunnan University, 650091, China
a r t i c l e i n f o
Article history:Received 22 September 2019Received in revised form 3 June 2020Accepted 6 June 2020Available online xxxx
Keywords:Multi-label classificationStacked ensembleLabel correlationBase classifier selectionRegularization via sparsity
a b s t r a c t
Multi-label classification has attracted increasing attention in various applications, such asmedical diagnosis and semantic annotation. With such trend, a large number of ensembleapproaches have been proposed for multi-label classification tasks. Most of theseapproaches construct the ensemble members by using bagging schemes, but few stackedensemble approaches are developed. Existing research on stacked ensemble approachesremains active, but several issues remain such as (1) little has been done to learn theweights of classifiers for classifier selection; (2) the relationship between pairwise labelcorrelations and multi-label classification performance has not been investigated suffi-ciently. To address these issues, we propose a novel stacked ensemble approach that simul-taneously exploits label correlations and the process of learning weights of ensemblemembers. In our approach, first, a weighted stacked ensemble with sparsity regularizationis developed to facilitate classifier selection and ensemble members construction for multi-label classification. Second, in order to improve the classification performance, the pairwiselabel correlations are further considered for determining weights of these ensemble mem-bers. Finally, we develop an optimization algorithm based on both of the accelerated prox-imal gradient and the block coordinate descent techniques to achieve the optimalensemble solution efficiently. Extensive experiments on publicly available datasets andreal Cardiovascular and Cerebrovascular Disease datasets demonstrate that our proposedalgorithm outperforms related state-of-the-art methods from perspectives of benchmark-ing and real-world applications.
� 2020 Elsevier Inc. All rights reserved.
1. Introduction
Multi-label learning has been widely used in various applications, such as text categorization [1], semantic annotation [2]and medical diagnosis [3], where each example can be associated with multiple class labels simultaneously. It is differentfrom single-label classification tasks that multi-label classification can be affected by intrinsic latent label correlations.For example, a patient with high blood pressure is more likely to develop a heart disease than one with normal blood pres-sure but is less likely to develop a neuromuscular disease [4].
A significant number of algorithms have been proposed to solve multi-label classification problems [5,6]. These algo-rithms can be divided into two categories [7], namely problem transformation methods and algorithm adaptation methods.
2 Y. Xia et al. / Information Sciences xxx (xxxx) xxx
The first category transforms a multi-label classification problem into either several independent binary classification prob-lems or one multi-class classification problem. Typical algorithms in this category include Binary Relevance (BR) [8], Classi-fier Chains (CC) [9] and Label Powerset (LP) [10]. The second category extends a specific learning algorithm to process multi-label data, such as MLKNN [11], ML-DT [12], and Rank-SVM [13]. However, the aforementioned algorithms still face the chal-lenges of label correlations, class imbalance and high dimensionality, which also leads the unsatisfied classification perfor-mance in real-world applications.
Ensemble learning algorithms have drawn considerable attention by combining individual learners from heterogeneousor homogeneous models to obtain a joint learner that improves learning permanence by overcoming the model overfittingand initialization sensitivity [14,51]. In recent years, some ensemble approaches [5,15] have been proposed as benchmarksfor multi-label classification tasks, they generally employ bagging schemes to generate diverse classifiers as ensemble mem-bers, and combine them into a consolidated ensemble result by majority voting, where class prediction of a new instance isdetermined by averaging the confidence values of all the classifiers for each label, rather than capturing the optimal weightsto different labels, which neglects the effect of local pairwise label correlation. Although stacked ensemble approaches haveoutstanding performance for many learning tasks, few of them address the aforementioned issues. To the best of our knowl-edge, Multi-label Stacking (MLS) [17] can be viewed as a representative of the stacked ensemble technique, where it initiallyobtains independent binary classifiers for each labelling, and uses their predictions as input into a meta-level learning model.Then, stacked ensembles of multi-labelling are carried out by a consensus function (a meta-level classifier) to achieve thefinal ensemble solution. Although MLS considers the global correlations among labels at the meta-level, it still neglectsthe effect of local pairwise label correlation. Moreover, existing stacked ensemble approaches do not take into account clas-sifier weights in association with classifier selection.
To alleviate the aforementioned issues, we simultaneously take the advantages of weighted stacked ensemble and pair-wise label correlations, which leads to a novel approach named Multi-Label classification with Weighted classifier selectionand Stacked Ensemble (MLWSE). In the MLWSE, different weights are assigned to individual classifiers as base learners fordifferent class labels, and any two strongly correlated class labels share high similar weights than two uncorrelated orweakly correlated ones. Unlike the existing stacked ensemble approaches, the MLWSE does not only apply sparsity as reg-ularization for classifier selection and ensemble construction, but also learns label meta-level specific features to addressthose well-known issues in multi-label classification.
In summary, our contributions are highlighted as follows:
� We propose a novel weighted stacked ensemble scheme named MLWSE for multi-label classification, it employs the spar-sity regularization to facilitate classifier selection and ensemble construction, and compatible with any multi-label clas-sifier as its base classifiers.� We simultaneously exploit the classifier weights and pairwise label correlations to select label meta-level specific fea-tures in MLWSE, which can be considered a label meta-level specific features selection method.� Our approach significantly achieves the robustness and effectiveness in the real-world applications of medical aided diag-nosis, which have been demonstrated on the practical Cardiovascular and Cerebrovascular Disease dataset.
The remainder of the paper is organized as follows. Section 2 reviews related work about ensembles of multi-label clas-sification. Section 3 presents our MLWSE approach. Section 4 reports the experimental results and analysis. Section 5 dis-cusses the issues related to our approach. Finally, the conclusions are drawn in Section 6.
2. Related work
In this section, we review state-of-the-art ensemble methods for multi-label classification, and provide an overview ofweighted ensembles of multi-label classification.
2.1. Ensemble of multi-label classification
Ensembles of multi-label classification are developed on top of the problem transformation or algorithm adaptationmethods [5]. The method aims to overcome the drawbacks of single multi-label classifiers by constructing performanceand diverse base classifiers. In this section, we mainly focus on ensemble construction in multi-label classification in con-junction with classifier combination strategies.
2.1.1. Bagging combination schemeEnsemble of Binary Relevance classifiers (EBR) [15] and Ensemble of Label Powerset classifiers (ELP) [18] are two repre-
sentative algorithms under the bagging framework.In EBR, each BR classifier is obtained from a random sub-sample of the training dataset, and can effectively improve the
performance of BR owing to the diversity among these classifiers. However, EBR does not consider the correlation informa-tion among the different labels, which is critical for semantic assiciation between different labels. For mining label correla-tion, Read et al. proposed in Ref. [9] the Ensemble of Classifier Chains (ECC) by using multiple CC as base classifier, where
Please cite this article as: Y. Xia, K. Chen and Y. Yang, Multi-label classification with weighted classifier selection and stacked ensemble,Information Sciences, https://doi.org/10.1016/j.ins.2020.06.017
Y. Xia et al. / Information Sciences xxx (xxxx) xxx 3
each CC is learned using a random subset of the training instances. ECC takes into account correlations among labels by aug-menting the feature space of each classifier with the label predictions of previous classifiers. Although the diversity of ECC isgenerated by selecting different chains and random subsets, it risks selecting sub-optimal chain ordering, which couldadversely affect the prediction performance.
Similar to the EBR models, ELP is proposed by combining several LP classifiers, which uses bagging to generate diverseclassifiers and then combines the predictions of the base classifiers by majority voting. Since the LP method generates asingle-label dataset with a different class for each different combination of labels, this forces that ELPs label sets are usuallyassociated with only few examples, which may lead to an imbalanced dataset and complicate the learning process. Manyensemble methods based on LP have been proposed, for example, the Ensemble of Pruned Sets (EPS) method [18] was pro-posed to overcome class-imbalance problems by pruning infrequently occurring label sets. The RAndom k-labEL (RAkEL)method [19] selects n random k-labels and learns n LP classifiers to reduce the sensitivity of the model to class-imbalance, where the base classifiers include a much more balanced distribution of classes. The Chi-Dep Ensemble (CDE)method [20] addresses the class-imbalance problem by selecting the n distinct top-scored partitions, in which each partitionis computed based on the v2 score for all label pairs.
Another multi-label ensemble method, Random Forest of Predictive Clustering Trees (RF-PCT) [21], is also based on a bag-ging combination scheme. This scheme predicts a new instance by averaging the confidence values of all the classifiers foreach label, rather than capturing the optimal weights to different labels. At the same time, it only partially considers globallabel correlations, rather than exploiting local pairwise label correlations.
2.1.2. Stacked combination schemeStacked ensemble techniques have become a well-established means for improving prediction accuracy. Model stacking
[22] is an efficient ensemble method in which the predictions that are generated by using multiple base classifiers are usedas inputs in a meta-level classifier. Similar to classical stacking, Multi-label stacking (MLS) [17] involves applying BR twice,and takes the predictions of the first step as new meta-level featurs to make final predictions for the corresponding label,thus considering the correlations among labels in the meta-level. To avoid overfitting, MLS usually used cross validationto train the meta-level classifier, where the data is partitioned into F disjoint parts, generating each base classifier F times,each using F-1 partitions for training and the remaining for gathering the predictions. In this way, MLS can be diversified byusing a different feature space in each classifier. However, MLS only considers the global label correlations, rather thanexploiting local pairwise label correlations.
To improve performance, many MLS methods take different meta-level data types into account, which are either discretevalues (0/1) or continuous values (the confidence scores). However, MLS may introduce irrelevant information into themeta-level data. If a label is completely uncorrelated with a predicted value generated by the base classifiers, the meta-levelclassifier introduces uninteresting information and noise, which would cause the performance to deteriorate. In this regard,it is important to take into account the weights of the confidence scores of different base classifiers for different labels.
2.2. Weighted ensemble of multi-label classification
How to optimally combine the contributions of each classifier is still a question in multi-label classification. In Ref. [23], aweighted classifier ensemble is proposed, which is designed for MLKNN with a weight adjustment strategy that employs aconfidence coefficient obtained by utilizing the distance in MLKNN. In Ref. [24], Improved BR (IBR) employs the weightedmajority voting strategy to achieve the classification of multi-label data streams. However, it is difficult to extend thesemethods to any other base classifier because these models involve algorithm-specific properties.
The AdaBoost.MH method [25] was extensively studied and used in multi-label classification, which not only maintains aset of weights over the instances, but also over the labels. Its weight adjustment strategy is the following: if traininginstances and the corresponding labels are difficult to predict, then incrementally increase the weights in the following clas-sifiers, whereas if instances and labels are easy to classify then lower the weights. Since the AdaBoost.MH is based on BR andit is the same as applying AdaBoost to multiple binary classifiers, thus it does not consider correlations among labels.
Spyromitros-Xioufis et al. [27] proposed a multi-target regression method based on weighted stacking, which does notconsider classifier selection. Sen and Erdogan [28] used the weighted sum rule (WS) and class-dependent weighted sum rule(CWS) for multi-class classification. Gunes [29] proposed model selection methods based on the Lasso penalized for optimalmodel selection. However, these methods do not consider local pairwise label correlations. For geometric explanation, Taiand Lin [26] proposed a novel hypercube view that perceived the label space of multi-label classification problems geomet-rically. Wu and Crestani [49] proposed an ensemble with spatial construction in information retrieval. Bonab and Can [50]proposed a geometrically optimum and online-weighted ensemble approach for data stream classification. Büyükçakir et al.[47] also proposed an online stacked ensemble for multi-label stream classification, named GOOWE-ML, which shows anintuitive geometric explanation for spatial modeling and weighting scheme. Following the spirit of these works, a geometricexplanation for three-dimensional weighted selection ensemble is shown in Fig. 1.
As shown in Fig. 1, the weighted stacked ensemble minimizes the Euclidean distance between the weighted predictionscores vector S~w and the target vector ~y that represents the ground truth in the label space, which can be seen as the fol-lowing linear least-squares problem.
PleaseInform
cite this article as: Y. Xia, K. Chen and Y. Yang, Multi-label classification with weighted classifier selection and stacked ensemble,ation Sciences, https://doi.org/10.1016/j.ins.2020.06.017
Fig. 1. A hypercube with three-dimensional case for weighted selection ensemble. It aims to minimize the Euclidean distance between the prediction scoresvector S~w of weighted selection ensemble and the target vector ~y that represents the ground truth in the label space. Here, ~w is sparse to prevent thestacked ensemble from combining all the~s1 and~s2, which takes label correlations into account.
4 Y. Xia et al. / Information Sciences xxx (xxxx) xxx
PleaseInform
min~w
~y � S~w�� ��2
2 ð1Þ
where S is the prediction scores matrix, ~w is the weight vector to be determined, and~y is the vector representing the groundtruth for a given data point.
Notations For an n� dmatrix A ¼ Ai;j� �
, where i 2 1;2; . . . ;nf g; j 2 1;2; . . . ; df g. AT denotes its transpose, tr Að Þ ¼Pni¼1Ai;i
is the trace of A, and Ak kF ¼ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiPn
i¼1Pd
j¼1A2i;j
qis its Frobenius norm. For any vector a ¼ a1; a2; . . . an½ �T, its l2 � norm is defined as
ak k2 ¼ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiPn
i¼1a2i
qand its l1 � norm is ak k1 ¼
Pni¼1 aij j.
3. Proposed approach
In this section, we present our proposed approach, including the model, algorithm, and optimization method.
3.1. Preliminary
Suppose X ¼ Rd denotes the d-dimensional input space, and Y ¼ y1; y2; . . . ; ylf g denotes the label space with l possibleclass labels. D ¼ xi; yið Þj1 6 i 6 nf g is the training dataset with n instances. For each multi-label example xi; yið Þ;xi 2 X isa d-dimensional feature vector xi ¼ xi1; xi2; . . . ; xid½ � and yi ¼ yi1; yi2; . . . ; yid½ � is the ground truth label of xi. Each elementyij ¼ 1 if the label yj is associated with xi, otherwise yij ¼ 0. The task of multi-label learning is to learn a function
h : X ! 2Y from training set D. For any unseen example x 2 X , the multi-label classifier hpredicts h xð Þ#Y as the set of labelsappropriate for x. In this paper, we denote the input data as a matrix X ¼ x1;x2; . . . ;xn½ �T 2 Rn�d, and denote the output label
matrix as Y ¼ y1; y2; . . . ; yn½ �T 2 Rn�l.
3.2. Generating the confidence score matrix
In the classifier combination problem with confidence score outputs, the combining process accepts as its input the pre-diction scores belonging to the different labels obtained from the base classifiers. Let skj be the prediction score of label j
obtained from classifier k for any data instances. Let sk ¼ sk1; sk2; . . . ; s
kl
� �Tbe the prediction score of all labels obtained from
classifier k, then the input to the combiner is s ¼ s1js2j . . . jsm� �, where m is the number of classifiers. Let si contain the scores
for training data point i obtained from the base classifiers, then the final confidence score matrix is S ¼ skijh i
cite this article as: Y. Xia, K. Chen and Y. Yang, Multi-label classification with weighted classifier selection and stacked ensemble,ation Sciences, https://doi.org/10.1016/j.ins.2020.06.017
Y. Xia et al. / Information Sciences xxx (xxxx) xxx 5
3.3. Weighted classifier selection and stacked ensemble
In the stacked ensemble framework, the combiner is defined as a function g : Rm�l ! Rl, hence our aim is to learn the g
function using data si; yij� � l
j¼1
n on
i¼1. With the loss function given in Eq. (1), our objective function to be minimized is the
following:
PleaseInform
g w1;w2; . . . ;wm� � ¼Xn
i¼1
Xl
j¼1
Xmk¼1
skijwkj � yij
� � !2
ð2Þ
where wkj denotes the weight of classifier k for label j, and wk ¼ wk
1;wk2; . . . ;w
kl
� �is the weight vector of classifier k.
Let Wj ¼ w1j jw2
j j . . . jwmj
h iTrepresent the combined weight vector of all classifiers for the j� th label, and
Yj ¼ y1j; y2j; . . . ; ynj� �T represent the j� th column of Y; 1 6 j 6 l. Based on generating the confidence score matrix, the objec-
tive function given in Eq. (2) can be further derived as
minWj
12
SWj � Yj
�� ��22 ð3Þ
3.3.1. Sparsity regularization for classifier selectionAs mentioned above, generating the confidence score matrix might contain irrelevant information that is not helpful for
label prediction. To combine the selection of classifiers, we add a regularization term to ensure that the weights are sparse toprevent the stacked ensemble from combining all the base classifiers. One main advantage of using sparsity regularizationfor classifier selection is that classifiers are selected automatically, and the number of selected classifiers is not specifiedbeforehand. In this regard, l1-norm regularization (Lasso) [30] can be considered the most successful method for inducingsparsity. We introduce the l1-norm regularization to the model for each weight vector Wj. Combined with the least-squares loss given in Eq. (3), the objective function for classifier selection can be formulated as
minWj
12
SWj � Yj
�� ��22 þ a Wj
�� ��1 ð4Þ
where ais a regularized parameter shared by all labels for balancing the loss and regularization term, and which can beadjusted to determine the number of selected classifiers. The details are discussed in the experimental section. Further, con-sidering all the binary classifiers simultaneously, Eq. (4) can be rewritten as
minW
12
SW� Yk k2F þ a Wk k1 ð5Þ
If wkj ¼ 0, it indicates that the k-th classifier will be eliminated and will have no effect on the prediction of the j-th label,
thereby accelerating the testing instance. However, the problem with l1-norm regularization for MLWSE is that not allwkj are
zero, which means that all information from a selected classifier would not be used effectively. Motivated by group sparsityregularization [31], we propose to use Group Sparsity Lasso to solve this problem. The difference between Lasso and GroupSparsity Lasso is illustrated by the example in Fig. 2.
As shown in Fig. 2, Lasso forces many confidence scores to become useless, and the corresponding classifier weight is zero.Group Lasso only selects two groups G2 and G4, whereas the other two groups G1 and G3 are not selected. However, in manycases, not all information in the selected group would be useful. Group Sparsity Lasso takes advantage of both Lasso andGroup Lasso in that it first selects groups, before making another selection from the selected group; i.e., it simultaneouslyconsiders intra-classifier and inter-classifier sparsity. Thus we obtain the MLWSE of Group Sparsity Lasso in combinationwith the regularization term of Lasso and Group Lasso
minW
12
SW� Yk k2F þ ak Wk k1 þ 1� að ÞkXmk¼1
ck WGk
�� ��2 ð6Þ
where a 2 0;1½ � is a convex combination of the Lasso and Group Lasso penalties, and the sparsity is determined by the mag-nitude of the tuning parameter k. Further, ck is a weight for the k-th group WGk
, which can be formulated as a prior to gen-
erating the contribution of the k-th group in the classifier selection process. In our experiments, we set ck ¼ffiffilp
.
3.3.2. Modeling label correlationsExploiting label correlations generally plays a key role in multi-label classification. This is motivated by work on multi-
task learning [32], which shares correlated information between tasks or modalities by considering their correlation. Weassume that if label yj and label yk are strong correlated, the classifier discriminative to yj may also be discriminative toyk with a higher probability. In other words, if two labels yj and yk are strongly correlated, the weight vector pair
cite this article as: Y. Xia, K. Chen and Y. Yang, Multi-label classification with weighted classifier selection and stacked ensemble,ation Sciences, https://doi.org/10.1016/j.ins.2020.06.017
Fig. 2. Illustrative comparison of Lasso, Group Lasso, and Group Sparsity Lasso. The confidence score matrix can be divided into four groups by using thefour base classifiers, G1; G2; G3; G4. The solid circles denote selected prediction scores whereas the open circles denote unselected prediction scores.
6 Y. Xia et al. / Information Sciences xxx (xxxx) xxx
Wj;Wk� �
should have a high similarity; otherwise, they would have low similarity. We construct a graph < V ; E >in the labelspace, where V denotes the vertex/label set, and E is the set of edges containing the edges between each label pair. Given thelabel correlation matrix R on E, the target can be formulated as minimizing the following equation
PleaseInform
12
Xl
j¼1
Xl
k¼1Wj �Wk
�� ��2Rjk ¼ tr W D� Rð ÞWT� �
¼ tr WHWT� �
ð7Þ
where H ¼ D� R is the graph Laplacian matrix and D is a diagonal matrix with Dii ¼Pn
j¼1Rij. Rjk denotes the similaritybetween label yj and label yk. In this study, we take the cosine similarity to calculate the label correlation matrix.
Combining Eqs. (5) and (7), we obtain the final objective function based on Lasso, MLWSE-L1:
minW
12
SW� Yk k2F þ a Wk k1 þb2tr WHWT� �
ð8Þ
Combining Eqs. (6) and (7), we obtain the final objective function based on Group Sparsity Lasso, MLWSE-L21:
minW
12
SW� Yk k2F þ ak Wk k1 þ 1� að ÞkXmk¼1
ck WGk
�� ��2 þ
b2tr WHWT� �
ð9Þ
In MLWSE-L1 and MLWSE-L21, a and b are tradeoff parameters with non-negative values. In MLWSE-L1, parameter acontrols the sparsity of the model while parameter b balances the contribution of label correlations and weight learning.In MLWSE-L21, parameter a is the tradeoff parameter with the Lasso and Group Lasso, and parameter k controls the sparsityof the model whereas parameter b balances the contribution of label correlations and weight learning.
3.3.3. Multi-label predictionAfter training MLWSE-L1 and MLWSE-L21, we obtain the classifier weight matrix W�. Given test data represented by
matrix X�, we first generate the confidence score matrix S� by the different base classifiers, after which we determine thepredict labels by a thresholding function sign : X ! R
sign S �W�; sð Þ ¼ 1; if S �W� P s0; otherwise
ð10Þ
where sis a threshold, and in our experiments, s is set to 0.5.
3.4. Optimization method
Although the minimization of Eqs. (8) and (9) are two convex optimization problems, the objective functions are non-smooth due to the non-smoothness of the l1-norm regularization terms. In this section, we use the accelerated proximal gra-dient and block coordinate descent to optimize MLWSE-L1 and MLWSE-L21, respectively.
3.4.1. Optimization of MLWSE-L1A general accelerated proximal gradient method can be formulated as the following convex optimization problem [33]:
minW2H
F Wð Þ ¼ f Wð Þ þ g Wð Þf g ð11Þ
where H is a real Hilbert space, f Wð Þ is convex and smooth, g Wð Þ is convex and can be non-smooth. If f Wð Þ has a Lipschitzcontinuous gradient with Lipschitz constant L, i.e., rf W1ð Þ � rf W2ð Þk k 6 L W1 �W2k k, instead of directly minimizing F Wð Þ,the proximal gradient algorithms can minimize its composite quadratic approximation
cite this article as: Y. Xia, K. Chen and Y. Yang, Multi-label classification with weighted classifier selection and stacked ensemble,ation Sciences, https://doi.org/10.1016/j.ins.2020.06.017
Y. Xia et al. / Information Sciences xxx (xxxx) xxx 7
PleaseInform
QL W;W tð Þ� �
¼ f W tð Þ� �
þ rf W tð Þ� �
;W�W tð ÞD E
þ L2
W�W tð Þ��� ���2
Fþ g Wð Þ ð12Þ
According to Eqs. (8) and (11), f Wð Þ and g Wð Þ can be defined as follows
f Wð Þ ¼ 12
SW� Yk k2F þb2tr WHWT� �
ð13Þ
g Wð Þ ¼ a Wk k1 ð14Þ
According to Eqs. (13), we can calculate rf Wð Þ as rf Wð Þ ¼ ST SW� Yð Þ þ bWH ð15Þ
Inspired by the work of Ref. [34,35], for MLWSE-L1, given W1 and W2, we obtain the Lipschitz constant as
L ¼ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi2 STS��� ���2
2þ 2 bHk k22
rð16Þ
According to Eqs. (12), (14), and (16), the weight matrix W can be optimized by
W� ¼ argminW
QL W;W tð Þ� �
¼ argminW
12
W� Z tð Þ��� ���2
Fþ g Wð Þ ¼ argmin
W
12
W� Z tð Þ��� ���2
Fþ a
LWk k1 ð17Þ
where Z tð Þ ¼W tð Þ � 1Lrf W tð Þ
� �.
The accelerated proximal gradient has shown thatW tð Þ ¼Wt þ bt�1�1bt
Wt �Wt�1ð Þ for a sequence bt satisfying b2t � bt 6 b2
t�1can improve the convergence rate to O 1=t2
� �, where Wt is the result of W at the t � th iteration [33]. The proximal operator
associated with the g Wð Þ of Eq. (17) is the soft-thresholding operator; then, in each iterative step, W� can be obtained by thefollowing optimization problem:
W tþ1ð Þ ¼ proxe Z tð Þh i
¼ argminW
12
W� Z tð Þ��� ���2
Fþ e Wk k1 ð18Þ
where proxe �½ � is a soft-thresholding operation is defined as
proxe wij� � ¼ wij � e; if wij > e
wij þ e; if wij < e0; otherwise
8><>: ð19Þ
According to Eqs. (17) and (19), W can be obtained by the following soft-thresholding operation,
W tþ1ð Þ ¼ proxaLZ tð Þh i
ð20Þ
The details of the accelerate proximal gradient for MLWSE-L1 are summarized in Algorithm 1. In our experiments, weemploy APG [33] to learn the weight matrix W�.
3.4.2. Optimization of MLWSE-L21Following the spirit of the work reported by Refs. [36,37], we use block coordinate descent to optimize MLWSE-L21. It
essentially has two components: the outer loop over the different feature groups and the inner loop that solves each ofthe block subproblems. The confidence score matrix S given in Eq. (9) can be broken down into m groups, S1; S2; . . . ; Sm, with
each Sk 2 Rn�l. Let S�k denote the remaining groups when S is associated to group k, andW�k is the components ofW over theother groups.
When a group k is selected (cyclically or otherwise), the other groups of the currentW are fixed and the objective function
is minimized only over Wk. Then at each block we have to minimize
12
r�k � S kð ÞW kð Þ��� ���2
2þ 1� að Þkck W kð Þ
��� ���2þ ak W kð Þ
��� ���1þ b2tr W kð ÞHW kð ÞT� �
ð21Þ
where r�k is the partial residual of Y, subtracting all group fits other than group k
r�k ¼ Y �Xj–k
S jð ÞW jð Þ�����
����� ð22Þ
Let ‘ r�k;Wkð Þ
� �¼ 1
2 r�k � S kð ÞW kð Þ��� ���2
2denote the least-squares loss function, and letr‘ r�k;W
kð Þ� �
denote its gradient, then
our goal is to findW kð Þ� to minimize Eq. (21). Minimizing Eq. (21) is equivalent to minimizing the following equation, centered
at a point W kð Þ0 by
cite this article as: Y. Xia, K. Chen and Y. Yang, Multi-label classification with weighted classifier selection and stacked ensemble,ation Sciences, https://doi.org/10.1016/j.ins.2020.06.017
8 Y. Xia et al. / Information Sciences xxx (xxxx) xxx
PleaseInform
12t
W kð Þ � W kð Þ0 � tr‘ r�k;W
kð Þ0
� �� ���� ���22þ 1� að Þkck W kð Þ
��� ���2þ ak W kð Þ
��� ���1þ b2tr W kð ÞHW kð ÞT� �
ð23Þ
where t denotes our usual gradient step.
It was previously shown in Ref. [36] that W kð Þ� ¼ 0 if
f W kð Þ0 � tr‘ r�k;W
kð Þ0
� �; tak
� ���� ���26 t 1� að Þkck ð24Þ
and otherwise W kð Þ� satisfies
1� t 1� að Þkckf W kð Þ
0 � tr‘ r�k;Wkð Þ0
� �; tak
� ���� ���2
0B@
1CAþ
f W kð Þ0 � tr‘ r�k;W
kð Þ0
� �; tak
� �ð25Þ
where f �ð Þ denotes the soft-thresholding operator
f z; takð Þð Þi ¼ sign zið Þ zij j � takð Þþ ð26Þ
Catalina et al. [37] showed that the inner loop can be accelerated by using proximal gradient, thus we set t ¼ 1
L with theLipschitz constant L given in Eq. (16). The details of the block coordinate descent for MLWSE-L21 are summarized in Algo-rithm 2.
Algorithm 1. Optimization of MLWSE-L1
Input: Training data matrix X 2 Rn�d, label matrix Y 2 Rn�l, base classifier Cif gmi¼1 and parameter a, b, g.Output: The weight matrix W� 2 Rml�l.1: Generate the confidence score matrix S 2 Rn�ml by the base classifier Cif gmi¼1.2: Initialization:b0; b1 1; W0;W1 STSþ gI
� ��1STY;
3: Compute the graph Laplacian matrix H on Y;4: Compute L according to Eq. (16);5: while not converged
Input: Training data matrix X 2 Rn�d, label matrix Y 2 Rn�l, base classifier Cif gmi¼1 and parameter a, b, k, g.Output: The weight matrix W� 2 Rml�l.1: Generate the confidence score matrix S 2 Rn�ml by the base classifier Cif gmi¼1.2: Compute the graph Laplacian matrix H on Y;3: Compute L according to Eq. (16);4: Compute r�k according to Eq. (22);5: Cyclically iterate through the groups; at each group kð Þ execute step 6;
6: Initialization: t 1=L, W kð Þ STSþ gI� ��1
STY;
7: Check if W kð Þ ¼ 0 according to Eq. (24), otherwise, within the group apply step 8;8: while not converged
9: Update gradient r‘ r�k;Wkð Þ
� �;
10: Update W kþ1ð Þ according to Eq. (25);
11: return W� W kþ1ð Þ
cite this article as: Y. Xia, K. Chen and Y. Yang, Multi-label classification with weighted classifier selection and stacked ensemble,ation Sciences, https://doi.org/10.1016/j.ins.2020.06.017
Y. Xia et al. / Information Sciences xxx (xxxx) xxx 9
4. Experiments
We carry out our experiments on three sets of datasets as following tasks: 1) investigating selection ability of weightedclassifier in our approach; 2) comparing our approach to the state-of-the-art multi-label ensemble methods via differentmulti-label evaluation metrics; 3) analyzing our approach from perspectives of robustness, parameter sensitivity andconvergence.
The experiment section is presented in four parts. Section 4.1 describes the experimental datasets. Section 4.2 presentsexperimental settings. Section 4.3 discusses experimental results, and finally, the algorithm analyses are drawn inSection 4.4.
4.1. Experimental datasets
In this section, we present our experimental datasets, including 2-D synthetic datasets, multi-label benchmark datasets,and the real-world Cardiovascular and Cerebrovascular Disease dataset, which are used to evaluate the effectiveness of ourproposed approach from different perspectives.
4.1.1. 2-D synthetic datasetsTo evaluate the weighted classifier selection ability, we conducted experiments on four 2-D synthetic datasets with dif-
ferent distribution scenarios. Since a multi-label classification problem can be transformed into several independent single-label classification problems [7], we only need to consider the case of the single-label classification. All four simulations arepresented in Fig. 3, and involve a univariate X drawn from a uniform distribution in [�4, +4]. The outcome follows the func-tion described below [51]:
� Scenario1: Y ¼ �2� I X < �3ð Þ þ 2:55� I X > �2ð Þ � 2� I X > 0ð Þ þ 4� I X > 2ð Þ � 1� I X > 3ð Þ þ N 0;1ð Þ� Scenario2: Y ¼ 5þ 0:4X � 0:36X2 þ 0:005X3 þ N 0;1ð Þ� Scenario3: Y ¼ 2:85� sin p
2 � X� �þ N 0;1ð Þ
� Scenario4: Y ¼ 3:85� sin 3p� Xð Þ � I X > 0ð Þ þ N 0;1ð Þ
where I �ð Þ is the usual indicator function and N 0;1ð Þ is an independent standard normal distribution for each scenario.As shown in Fig. 3, it contains a scatterplot of the 300 samples for each simulation, and the true curve for each scenario is
represented by the red line. These scenarios were chosen because they represent a diverse set of true models. With these 2-Dsynthetic datasets, we can evaluate the weighted classifier selection ability of our approach. The detailed experimentalresults are shown in Section 4.3.1.
4.1.2. Benchmark datasetsTo verify the performance of the proposed method, we compare the proposed method on 13 widely used multi-label
benchmark datasets. The datasets are summarized in Table 1. LC ¼ 1N
PNi¼1
Yij j denotes the label cardinality, which is the aver-
age number of labels associated with each instance. All datasets can be downloaded from the websites of Mulan,1 KDIS,2 andMeka.3 With these benchmarking multi-label datasets, we initially compare our approach with seven state-of-the-art ensemblemulti-label classification methods. The detailed experimental results are shown in Section 4.3.2.
4.1.3. Real-world datasetsTo explore the potential application of our proposed method, we apply our approach to a real Cardiovascular and Cere-
brovascular Disease (CCD) dataset [38] to demonstrate its potential for practical applications in medical diagnosis, and wetake CCD dataset as another benchmarking dataset to run the experiments. The dataset is collected from cardiovascular andcerebrovascular patients in a hospital in Yunnan Province, China. It contains 3,823 samples, 59 features, and 9 labels, wherethe nine labels are cerebral ischemic stroke (CIS), cerebral hemorrhage (CH), subarachnoid hemorrhage (SAH), cerebral venousthrombosis (CVT), intracranial aneurysm (IA), cerebrovascular malformation (CVM), heart disease (HD), diabetes mellitus(DM), and hypertension (HT). The number of examples corresponding to each label is listed in Table 2. The detailed experi-mental results are shown in Section 4.3.3.
4.2. Experimental settings
In this section, we describe our experimental settings,including baseline methods, hyper-parameter setting and evalua-tion metrics.
Please cite this article as: Y. Xia, K. Chen and Y. Yang, Multi-label classification with weighted classifier selection and stacked ensemble,Information Sciences, https://doi.org/10.1016/j.ins.2020.06.017
Fig. 3. Scatterplots of the four scenarios. The red line represents the true relationship. The number of simulation samples is 300.
10 Y. Xia et al. / Information Sciences xxx (xxxx) xxx
4.2.1. Baseline methodsWe compare our proposed method MLWSE with the seven state-of-the-art ensemble multi-label classification methods.
� EBR [15]: an ensemble version of the BR model. The instances of each BR are generated randomly. It does not consider therelationship between labels.� ECC [9]: an ensemble version of CC, where the chain order of each CC is generated randomly. It considers the global labelcorrelations.� EPS [18]: an improved ensemble version of LP to focus on the most important relationships of labels by pruning infre-quently occurring label sets. It focuses on the most important relationships of labels by pruning infrequently occurringlabel sets.� RAkEL [19]: an improved ensemble version of LP based on a random small subset of k labels, where k is set to be 3. Itconsiders a small subset of label relationships.
Please cite this article as: Y. Xia, K. Chen and Y. Yang, Multi-label classification with weighted classifier selection and stacked ensemble,Information Sciences, https://doi.org/10.1016/j.ins.2020.06.017
Y. Xia et al. / Information Sciences xxx (xxxx) xxx 11
� CDE [20]: an ensemble of the Chi-Dep algorithm, which builds an LP classifier for each group of dependent labels. Thepartitions of the label sets are set to be 100. It considers the label relationships between groups.� AdaBoost.MH [25]: a weighted ensemble version based on BR that not only maintains a set of weights over the instancesas AdaBoost does, but also over the labels.� MLS [17]: a stacked ensemble version based on BR that takes into account the relationship among labels in the meta-level.
4.2.2. Hyper-parameter settingAll of the baseline methods were implemented using the Mulan [39] and Meka [40] frameworks, which provide an API to
use their functionalities in Java code.4 These methods used n ¼ 10 classifiers in the ensemble, a threshold value of s ¼ 0:5 andthe C4.5 decision tree as a single-label base classifier.
In our experiments, the confidence score matrix S is generated by using BR, CC, and LP, where SVM is used as the single-label base classifier, and the other parameters are set as default parameters in the scikit-multilearn library [41]. The number
of groups m is set to 3. For MLWSE-L1, parameters a; b are searched in 10�5;10�4; . . . ;103;104n o
, and g is searched in
0:1;1f g. For MLWSE-L21, parameter a is searched in 0:01;0:05;0:1;0:15;0:2f g; b is searched in
10�4;10�3; . . . ;101;102n o
; k is searched in 10�5;10�4; . . . ;101;102n o
, and g is searched in 0:1;1f g. The software implemen-
tation of the proposed algorithms has been available at https://github.com/AiXia520/MLWSE.
4.2.3. Evaluation metricsWe use six common evaluation metrics to evaluate the performance of multi-label classification. In general terms, they
can be categorized into two groups [5], i.e., example-based metrics (Hamming loss, Accuracy, Ranking loss and F1) and label-based metrics (Macro B(h) and Micro B(h)). For each evaluation metric, the testing dataset is defined as
Dt ¼ xi; yið Þj1 6 i 6 nf g, where yi 2 0;1f gl is the ground truth labels of the i� th test example, and yi ¼ h xið Þ is its predictedlabels.
� Hamming loss: it evaluates the fraction of misclassified example-label pairs. The smaller the Hamming loss, the moreaccurate the performance of the classifier is.
Hamming loss ¼ 1n
Xni¼1
1l
Xl
j¼1I yij – yij� �
where I �ð Þ is an indication function that returns 1 if I �ð Þ holds and 0, otherwise.� Accuracy: it evaluates the Jaccard similarity between the ground truth labels and predicted labels.
4 Cod
PleaseInform
Accuracy ¼ 1n
Xni¼1
yi \ yij jyi [ yij j
� Ranking loss: it evaluates the fraction of reversely ordered label pairs, i.e., when an irrelevant label is ranked higher than arelevant label.
Rankingloss ¼ 1n
Xni¼1
1yij j �yij j j y1; y2ð Þjf xi; y1ð Þ 6 f xi; y2ð Þ; y1; y2ð Þ 2 yi � �yif g
where f x; yð Þ can be regarded as the confidence score of y 2 Y being the proper label of x.� F1: it is the harmonic mean of recall and precision, where pi and qi are the recall and precision for the i� th example.
e of Mulan is available at:https://github.com/kdis-lab/ExecuteMulan.
cite this article as: Y. Xia, K. Chen and Y. Yang, Multi-label classification with weighted classifier selection and stacked ensemble,ation Sciences, https://doi.org/10.1016/j.ins.2020.06.017
12 Y. Xia et al. / Information Sciences xxx (xxxx) xxx
PleaseInform
F1 ¼ 1n
Xni¼1
2piqi
pi þ qi
� Label-based classification metrics can be obtained in either of the following modes [39]:
Macro B hð Þ ¼ 1l
Xl
j¼1B TPj; FPj; TNj; FNj� �
MicroB hð Þ ¼ BXl
j¼1TPj;
Xl
j¼1FPj;
Xl
j¼1TNj;
Xl
j¼1FNj
!
where TPj; FPj; TNj, and FNj represent the number of true positive, false positive, true negative, and false negative testexamples with respect to label yj. B TPj; FPj; TNj; FNj
� �indicate some specific binary classification metric (e.g., F1). Macro
B(h) and Micro B(h) assume equal weights for labels and examples, respectively.
4.3. Experimental results
Based on the above experimental datasets and settings, we present experimental results on three type of datasets andgive a detailed experimental analysis for each dataset.
4.3.1. 2-D synthetic datasets resultsWith 2-D synthetic datasets, we evaluate the weighted classifier selection ability of our approach by gradually adding
different technical components, including the weighted setting given in Eq. (3) as baseline, the Lasso selection given inEq. (5), and the Group Sparsity Lasso selection given in Eq. (6). We randomly divided each dataset into a training set(35%), a validation set (35%), and a testing set (30%). The experimental results for all four scenarios are presented in Table 3.
In the first scenario, all three base classifiers presented in our experiment perform well, but the SVM-based poly kernelperforms the best. The accuracy of the Lasso selection and Group Sparsity Lasso selection methods is 0.5222 and 0.5333,respectively, and the corresponding weight vectors are 0:3391;0:5894;0:0521h i and 0;0:9326;0:0445h i, which are higherthan the baseline method. The results show that Lasso selection and Group Sparsity Lasso selection learn optimal weightscompared to the baseline method, which assigns a higher weight to a more effective base classifier. In the second scenario,the SVM-based poly kernel is a more optimal base classifier. Similarly, Lasso selection and Group Sparsity Lasso selection areable to adapt to the underlying structure to achieve superior accuracy. The same trend is exhibited in scenarios 3 and 4, inwhich the weighted classifier selection methods do nearly as well as the individual best algorithm, and even outperform theindividual best algorithm. The individual best algorithm is not known, especially in a multi-label setting in which the per-formance of the base classifier might improve on some labels and decline on others. However, our algorithms can adaptivelylearn the optimal weights to select the base classifier and adapt to real changeable scenarios. The results implicitly indicatethat the proposed weighted classifier selection is effective for improving multi-label classification performance.
4.3.2. Benchmark datasets resultsTables 4 and 5 compare the analysis of the results of the proposed method MLWSE-L1 and MLWSE-L21 against state-of-
the-art algorithms on 13 datasets. For each dataset, the parameters a; b, and g of MLWSE-L1 are set to 10�4; 10�3, and 0.1,respectively, and the parameters a; k; b, and g of MLWSE-L21 are set to 0.05, 10�3; 10�2, and 0.1, respectively. We con-ducted a fivefold validation and recorded the mean and standard deviation for each evaluation metric. In the largest datasets,some algorithms cannot complete corresponding to the available resources, and these cases are marked as DNF in the resulttables. The best results among all the algorithms being compared are highlighted in boldface. According to these results, wehave the following points.
� Comparing with bagging combination scheme (i.e., EBR, ECC, EPS, RAkEL and CDE), in more cases, MLWSE outperformsbagging combination methods. The reason is that MLWSE can comprehensively capture the optimal weights to differentlabels and take local pairwise label correlations into account.� Comparing with stacked combination scheme (i.e., MLS), in most cases, MLWSE significantly improves the performance(i.e., Accuracy and F1). This is because our approaches can exploit the weights of the confidence scores of different baseclassifiers for different labels and consider local pairwise label correlations.� Comparing with weighted ensemble method (i.e., AdaBoost.MH), in most cases, MLWSE outperforms AdaBoost.MH. Thereason is that we simultaneously exploit the classifier weights and pairwise label to address those well-known issues inmulti-label classification.
4.3.3. Real-world application resultsTable 6 reports the experimental results that obtained from the different multi-label ensemble algorithms. The results
demonstrate that our approach achieves statistically superior performance compared to the other approaches. To verify thatif any two labels are strongly correlated, the pairs of classifier weight vectors should have high similarity. Fig. 4 shows theaffinity matrices of the label matrix and the learned weight matrix for CCD, where a stronger greyscale represents strongerlabel correlations. The results shown in Fig. 4 indicate that the affinity matrix of the label matrix and the affinity of thelearned weight matrix are surprisingly consistent, which means that, if two labels yj and yk are strongly correlated, the
cite this article as: Y. Xia, K. Chen and Y. Yang, Multi-label classification with weighted classifier selection and stacked ensemble,ation Sciences, https://doi.org/10.1016/j.ins.2020.06.017
Random Forest 0.0521 �0.1970 0.9197 0.2771Group sparsity lasso
selectionSVM (linearkernel)
0.5333 0 0.5111 0 0.8333 0 0.7667 0
SVM (polykernel)
0.9326 1.1716 0.0797 0.7317
Random Forest 0.0445 �0.1811 0.9226 0.2737
Y. Xia et al. / Information Sciences xxx (xxxx) xxx 13
weight vector pair Wj;Wk
� �should be highly similar. This experimental result verifies the correctness of our proposed
hypothesis.
4.4. Algorithm analysis
We analyze our proposed approach from different perspectives, including Friedman statistics analysis, parameter sensi-tivity analysis and Convergence analysis.
4.4.1. Friedman statistics analysisWe employed the Friedman test [42,43] to statistically analyze the performance of the different algorithms systemati-
cally. Table 7 provides the Friedman statistics FF and the corresponding critical value in terms of each evaluation metric.As shown in Table 7, at the significance level a ¼ 0:05, the null hypothesis of equal performance is rejected for each evalu-ation metric. Hence, we can use the post hoc test [43] for relative performance comparisons. We employed the Nemenyi test[43] to analyze the performance of our proposed method MLWSE-L1 and MLWSE-L21 against that of the other algorithms.The performance of two classifiers is significantly different if the corresponding average ranks differ by at least the critical
difference CD ¼ qa
ffiffiffiffiffiffiffiffiffiffik kþ1ð Þ6N
q. At the significance level a ¼ 0:05, the value of qa ¼ 3:102 [43], for the Nemenyi test with
k ¼ 9; N ¼ 14 (including 13 benchmark datasets and a real-world dataset), and thus CD ¼ 3:211. Fig. 5 shows the CD dia-grams for each evaluation metric, where the average rank of each algorithm is marked along the axis (lower ranks to theleft). In each subfigure, any algorithm connected with MLWSE is considered to have significantly different performancebetween them. The Friedman statistics results indicate that:
� EBR outperforms the other approaches on Hamming Loss. This is because EBR is a first-order approach (which does notconsider label correlations) that tries to optimize the Hamming Loss. But the proposed MLWSE-L21 outperforms MLWSE-L1 and the other approaches in terms of Hamming Loss.� ECC outperforms the other approaches on Micro-F1, because ECC is a high-order approach (which considers global labelcorrelations) that tries to model global label correlations. But the proposed MLWSE-L1 outperforms MLWSE-L21 and theother approaches in terms of Micro-F1.� MLWSE outperforms the other approaches in terms of the other four evaluation metrics. The superior performance ofMLWSE against these approaches indicates the effectiveness of learning label correlations and the classifier weight.
To summarize, MLWSE achieves competitive performance against other well-established multi-label ensembleapproaches.
4.4.2. Parameter sensitivity analysisWe analyzed the parameter sensitivity of MLWSE-L1 and MLWSE-L21 by conducting experiments on the Emotions and
GpositiveGO datasets. We performed a fivefold cross validation on each dataset and analyzed the relative parameter.
Please cite this article as: Y. Xia, K. Chen and Y. Yang, Multi-label classification with weighted classifier selection and stacked ensemble,Information Sciences, https://doi.org/10.1016/j.ins.2020.06.017
Fig. 4. Label and weight affinity matrices of the CCD dataset.
Table 7Summary of the Friedman statistics FF k ¼ 9;N ¼ 14ð Þ and theCritical Value in term of each evaluation Metric.
Metric FF Critical Value (a ¼ 0:05)
Accuracy 35.075 3.211Hamming loss 6.348Ranking loss 37.824F1 9.261Macro-F1 10.243Micro-F1 8.312
16 Y. Xia et al. / Information Sciences xxx (xxxx) xxx
In MLWSE-L1, the regularization parameters a and b were searched in 10�5;10�4; . . . ;103;104n o
, and gwas set to 0.1. For
each a; bð Þ-pair, we recorded the mean value of F1. Fig. 6(a) and (b) report the influence of parameters a and b on the Emo-tions and GpositiveGO datasets, respectively. It can be seen that in most cases:
� The performance of MLWSE-L1 is worse when the value of ais large, especially, a > 10 is often harmful;� The performance of MLWSE-L1 initially improves and then degrades with the increasing of b. Therefore, we fixed the reg-ularization parameters a and b to 10�4 and 10�3 for multi-label benchmark datasets, respectively.
In MLWSE-L21, the regularization parameters ais searched in 0:01;0:05;0:1;0:15;0:2f g, the parameter k is searched in
10�5;10�4; . . . ;101;102n o
; b is searched in 10�4;10�3; . . . ;101;102n o
, and g is set to 0.1. We first find the group of best con-
figurations for the parameters by using fivefold cross validation on the training data of the Emotions dataset, and then keepthe value of one parameter constant and vary the values of the other two parameters. The average results of MLWSE-L21with different values of a; b, and k are depicted in Fig. 7(a)–(l). The figure shows that in most cases:
Please cite this article as: Y. Xia, K. Chen and Y. Yang, Multi-label classification with weighted classifier selection and stacked ensemble,Information Sciences, https://doi.org/10.1016/j.ins.2020.06.017
Fig. 5. CD diagrams of the algorithms that were compared under each evaluation criterion.
Fig. 6. Parameter sensitivity analysis of MLWSE-L1.
Y. Xia et al. / Information Sciences xxx (xxxx) xxx 17
� With a fixed setting of a, the candidate sets for kand b can be employed in 10�4;10�3;10�2n o
to obtain satisfactory
performance.� With a fixed setting of k, the performance of MLWSE-L21 is stable with different values of each a; bð Þ-pair.� With a fixed setting of b, the candidate sets for a and k can be employed by searching k in 10�5;10�4;10�3;10�2
n oto
obtain satisfactory performance.
4.4.3. Convergence analysisWe analyzed convergence of MLWSE-L1 and MLWSE-L21 by conducting experiments on the Emotions, Scene, Yeast and
VirusGO datasets. In this work, our approach is solved by using accelerated proximal gradient and block coordinate descent,which can be seen as iterative shrinkage-thresholding algorithms. Accelerated proximal gradient was proven to converge infunction values as O 1=t2
� �with a backtracking step size rule [33]. Block coordinate descent is proven to converge in function
values as O log t=tð Þ2� �
[33–35]. Fig. 8 shows the value of the loss function of MLWSE-L1 (Eq. (8)) and MLWSE-L21 (Eq. (9))
with the number of iterative cycles, respectively. For MLWSE-L1, the loss value tends to stabilize after 300 iterative cycles.Fig. 8(a) shows that when the number of iterative cycles exceeds 200, the loss value (which is less than 0.008) has little effecton the performance; hence, the number of cycles in the experiment was specified to be 200. For MLWSE-L21, the loss valuetends to stabilize after 200 cycles. Here, the number of iterative cycles refers to the outer loop, whereas the number of cycles
Please cite this article as: Y. Xia, K. Chen and Y. Yang, Multi-label classification with weighted classifier selection and stacked ensemble,Information Sciences, https://doi.org/10.1016/j.ins.2020.06.017
Fig. 8. Loss of MLWSE-L1 and MLWSE-L21 with the number of iterative cycles.
Fig. 7. Parameter sensitivity analysis of MLWSE-L21. (a)–(d): the parameter analysis of MLWSE-L21 with fixed a; (e)–(h): the parameter analysis ofMLWSE-L21 with fixed k; (i)–(l): the parameter analysis of MLWSE-L21 with fixed b.
18 Y. Xia et al. / Information Sciences xxx (xxxx) xxx
of the inner loop in the experiment was set to 100. The experimental results show that the proposed methods are more effi-cient than most of the multi-label ensemble methods.
5. Discussion
We conducted a comprehensive investigation based on a series of simulations. As demonstrated by our experiments, ourapproach is able to achieve high-quality generalization performance by implementing a simple iterative shrinkage-thresholding algorithm. Thus, a promising yet easy-to-use technique for multi-label ensemble classification is introduced.We summarize the advantages of our approach as follows.
First, we proposed a novel weighted stacked ensemble approach for multi-label classification compatible with any exist-ing multi-label classification algorithm such as MLKNN [11] and ML-DT [12] as base learners, and used sparsity for regular-ization to facilitate classifier selection and ensemble construction with the ultimate goal of developing a simple and efficientmethod to select multi-label base classifiers. Our approach is geometrically explained in Fig. 1, and its ability of the weightedclassifier selection is systematically evaluated, and results are presented in Table 3. Further, our approach was tested on dif-ferent datasets from a variety of domains such as Text, Imaging, Biology, and Medicine. Tables 4 and 5 indicate that our
Please cite this article as: Y. Xia, K. Chen and Y. Yang, Multi-label classification with weighted classifier selection and stacked ensemble,Information Sciences, https://doi.org/10.1016/j.ins.2020.06.017
Y. Xia et al. / Information Sciences xxx (xxxx) xxx 19
approach outperforms state-of-the-art multi-label ensemble algorithms, while Fig. 8 demonstrates that our approach pro-vides an effective and easy to use technique for multi-label classification tasks due to its fast convergence of learning process.
Second, our approach can be seen as label meta-specific-feature selection methods because of their exploitation of labelcorrelations. Here, generating the confidence score matrix S can be seen as a meta feature. Due to the fact of that both clas-sifier weights and label correlations are exploited simultaneously, constructing weight matrixW can be regarded as selectinga label meta-specific-feature. With assumption of that the classifier weight vector pair would be highly similar if a strongcorrelation between any two labels existed, any two strongly correlated class labels would share more meta features witheach other than two uncorrelated or weakly correlated ones, which has been illustrated in Fig. 4. Similar approaches to learn-ing label specific features have been proposed, such as MLSF [44] and LLSF [45]. In our approaches, parameter acan be usedto adjust feature sparsity, and parameter b balances the contribution of label correlations and weight learning. The resultsshown in Figs. 6 and 7 demonstrate the effect of parameter adjustments, and show that the proposed MLWSE-L21 method ismore stable than the MLWSE-L1 method. Finally, Fig. 5 and Table 6 indicate that our approach is much more robust thanother according to different evaluation metrics.
A subsequent literature survey revealed that Zhou and Tao [46] proposed a multi-label subspace ensemble method basedon the group sparsity Lasso, without considering stacked ensemble scheme. In contrast, our approach simultaneously adoptssparsity-based regularization for classifier selection and ensemble construction, and cosine similarity to calculate the labelcorrelation matrix, which improves computational efficiency with the significant interaction between base classifiers.
Similar to other multi-label stacking algorithms, our approach also needs to generate meta-level features, with additionalcomputational cost, which could be problematic for extreme multi-label classification [48]. Hence, finding a compromisebetween computational efficiency and classification accuracy for extreme multi-label ensemble classification would be aninteresting and challenging research topic in our future work.
6. Conclusion
In this paper, we presented a novel weighted classifier selection and stacked ensemble for multi-label classification,MLWSE, it uses sparsity for regularization to facilitate classifier selection and ensemble construction, and simultaneouslyexploits the classifier weights and label correlation to improve the classification performance. On the other hand, our ensem-ble approach is not only providing a label meta-specific-feature selection method, but also compatible with any existingmulti-label classification algorithm as its base classifier. We compared our method MLWSE-L1 and MLWSE-L21 with severalwell-established multi-label ensemble classification algorithms on 13 multi-label benchmark datasets and real Cardiovascu-lar and Cerebrovascular Disease datasets. The results of the comparison confirmed the competitive performance of our pro-posed method, and verified the effectiveness of the weighted stacked ensemble, and suitable for the application in the real-world applications.
Declaration of Competing Interest
The authors declare that they have no known competing financial interests or personal relationships that could haveappeared to influence the work reported in this paper.
The authors would like to thank KDIS Research Group for providing the multi-label benchmark datasets and Mulan codeonline to enable us to complete the comparative studies. This work was supported in part by the Natural Science Foundationof China under Grant 61663046 and Grant 61876166, and in part by the Program for Excellent Young Talents of YunnanUniversity.
References
[1] A. McCallum, Multi-label text classification with a mixture model trained by EM, AAAI’99 Workshop on Text Learning, 1999.[2] C. Sanden, J.Z. Zhang, Enhancing multi-label music genre classification through ensemble techniques, in: SIGIR’11 – Proceedings of the 34th
International ACM SIGIR Conference on Research and Development in Information Retrieval, 2011, pp. 705–714, https://doi.org/10.1145/2009916.2010011.
[3] H. Weng, Z. Liu, A. Maxwell, X. Li, C. Zhang, E. Peng, G. Li, A. Ou, Multi-label symptom analysis and modeling of TCM diagnosis of hypertension, in:Proceedings – 2018 IEEE International Conference on Bioinformatics and Biomedicine, BIBM, 2018, 2019,, pp. 1922–1929, https://doi.org/10.1109/BIBM.2018.8621173.
[4] B. Jin, B. Muller, C. Zhai, X. Lu, Multi-label literature classification based on the Gene Ontology graph, BMC Bioinformatics (2008).
Please cite this article as: Y. Xia, K. Chen and Y. Yang, Multi-label classification with weighted classifier selection and stacked ensemble,Information Sciences, https://doi.org/10.1016/j.ins.2020.06.017
20 Y. Xia et al. / Information Sciences xxx (xxxx) xxx
[5] M.L. Zhang, Z.H. Zhou, A review on multi-label learning algorithms 26 (2014) 1819–1837.[6] P. Li, H. Li, M. Wu, Multi-label ensemble based on variable pairwise constraint projection, Information Sciences 222 (2013) 269–281.[7] G. Madjarov, D. Kocev, D. Gjorgjevikj, S. Dzeroski, An extensive experimental comparison of methods for multi-label learning 45 (2012) 3084–3104.[8] M.R. Boutell, J. Luo, X. Shen, C.M. Brown, Learning multi-label scene classification, Pattern Recognition 37 (2004) 1757–1771.[9] J. Read, B. Pfahringer, G. Holmes, E. Frank, Classifier chains for multi-label classification, Machine Learning 85 (2011) 333–359.[10] G. Tsoumakas, I. Katakis, I. Vlahavas, Random k-labelsets for multilabel classification, IEEE Transactions on Knowledge and Data Engineering 23 (2011)
1079–1089.[11] M.L. Zhang, Z.H. Zhou, ML-KNN: A lazy learning approach to multi-label learning, Pattern Recognition 40 (2007) 2038–2048.[12] A. Clare, R.D. King, Knowledge discovery in multi-label phenotype data, in: Lecture Notes in Computer Science (including subseries Lecture Notes in
Artificial Intelligence and Lecture Notes in Bioinformatics), 2001, pp. 42–53, doi: 10.1007/3-540-44794-6_4.[13] A. Elisseeff, J. Weston, A kernel method for multi-labelled classification, in: Advances in Neural Information Processing Systems, 2002, pp. 681–687,
doi: 10.7551/mitpress/1120.003.0092.[14] Y. Yang, J. Jiang, Adaptive bi-weighting toward automatic initialization and model selection for HMM-based hybrid meta-clustering ensembles, IEEE
Transactions on Cybernetics 49 (2019) 1657–1668.[15] J.M. Moyano, E.L. Gibaja, K.J. Cios, S. Ventura, Review of ensembles of multi-label classifiers: Models, experimental study and prospects, Information
Fusion 44 (2018) 33–45.[17] G. Tsoumakas, A. Dimou, E. Spyromitros, V. Mezaris, I. Kompatsiaris, I. Vlahavas, Correlation-based pruning of stacked binary relevance models for
multi-label learning, 2009, p. 101–116.[18] J. Read, B. Pfahringer, G. Holmes, Multi-label classification using ensembles of pruned sets, in: Proceedings – IEEE International Conference on Data
Mining ICDM, 2008, pp. 995–1000, https://doi.org/10.1109/ICDM.2008.74.[19] G. Tsoumakas, I. Katakis, I. Vlahavas, Mining Multi-label Data, in: Data Mining and Knowledge Discovery Handbook, 2009, pp. 667–685, doi: 10.1007/
978-0-387-09823-4_34.[20] L. Tenenboim-Chekina, L. Rokach, B. Shapira, Identification of label dependencies for multi-label classification, in: Proceedings of the 2nd International
Workshop on Learning form Multi-label Data, 2010, pp. 53–60.[21] D. Kocev, C. Vens, J. Struyf, S. Dzeroski, Ensembles of multi-objective decision trees, in: Lecture Notes in Computer Science (including subseries Lecture
Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2007, pp. 624–631, doi: 10.1007/978-3-540-74958-5_61.[22] D.H. Wolpert, Stacked generalization, Neural Networks 5 (1992) 241–259.[23] L. Wang, H. Shen, H. Tian, Weighted ensemble classification of multi-label data streams, in: Lecture Notes in Computer Science (including subseries
Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2017, pp. 551–562, doi: 10.1007/978-3-319-57529-2_43.[24] W. Qu, Y. Zhang, J. Zhu, Q. Qiu, Mining multi-label concept-drifting data streams using dynamic classifier ensemble, in: Lecture Notes in Computer
Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2009, pp. 308–321, doi: 10.1007/978-3-642-05224-8_24.
[25] R.E. Schapire, Y. Singer, BoosTexter: a boosting-based system for text categorization, Machine Learning 39 (2000) 135–168.[26] F. Tai, H.T. Lin, Multilabel classification with principal label space transformation, Neural Computation 24 (2012) 2508–2542.[27] E. Spyromitros-Xioufis, G. Tsoumakas, W. Groves, I. Vlahavas, Multi-target regression via input space expansion: treating targets as inputs, Machine
Learning (2016).[28] M.U. Sen, H. Erdogan, Max-margin stacking and sparse regularization for linear classifier combination and selection, 2011, arXiv:1106.1684.[29] F. Gunes, Penalized regression methods for linear models in sas/stat, 2015.[30] C. Cui, D. Wang, High dimensional data regression using Lasso model and neural networks with random weights, Information Sciences 372 (2016)
505–517.[31] M. Yuan, Y. Lin, Model selection and estimation in regression with grouped variables, Journal of the Royal Statistical Society. Series B: Statistical
Methodology 68 (2006) 49–67.[32] S. Pan, J. Wu, X. Zhu, G. Long, C. Zhang, Task sensitive feature exploration and learning for multitask graph classification, IEEE Transactions on
Cybernetics 47 (2017) 744–758.[33] N. Ito, A. Takeda, K.C. Toh, A unified formulation and fast accelerated proximal gradient method for classification, Journal of Machine Learning Research
18 (2017) 510–558.[34] V. Kumar, A.K. Pujari, V. Padmanabhan, V.R. Kagita, Group preserving label embedding for multi-label classification, Pattern Recognition 90 (2019) 23–
34.[35] J. Nutini, M. Schmidt, I.H. Laradji, M. Friedlander, H. Koepke, Coordinate descent converges faster with the Gauss-Southwell rule than random selection,
in: 32nd International Conference on Machine Learning, ICML 2015, 2015, pp. 1632–1641, arXiv:1506.00552.[36] N. Simon, J. Friedman, T. Hastie, R. Tibshirani, A sparse-group lasso, Journal of Computational and Graphical Statistics 22 (2013) 231–245.[37] A. Catalina, C.M. Alaíz, J.R. Dorronsoro, Accelerated block coordinate descent for sparse group lasso, in: Proceedings of the International Joint
Conference on Neural Networks, 2018, pp. 1–8, https://doi.org/10.1109/IJCNN.2018.8489078.[38] M. Deng, C. Wang, M. Tang, T. Zheng, Extracting cardiac dynamics within ECG signal for human identification and cardiovascular diseases classification,
Neural Networks 100 (2018) 70–83.[39] G. Tsoumakas, E. Spyromitros-Xioufis, J. Vilcek, I. Vlahavas, MULAN: A Java library for multi-label learning, Journal of Machine Learning Research 12
(2011) 2411–2414.[40] J. Read, P. Reutemann, B. Pfahringer, G. Holmes, MEKA: A multi-label/multi-target extension to WEKA, Journal of Machine Learning Research 17 (2016)
667–671.[41] P. Szymánski, T. Kajdanowicz, Scikit-multilearn: A scikit-based Python environment for performing multi-label classification, Journal of Machine
Learning Research 20 (2019) 209–230.[42] M. Friedman, A comparison of alternative tests of significance for the problem of m rankings, The Annals of Mathematical Statistics 11 (1940) 86–92.[43] J. Demšar, Statistical comparisons of classifiers over multiple data sets, Journal of Machine Learning Research 7 (2006) 1–30.[44] L. Sun, M. Kudo, K. Kimura, Multi-label classification with meta-label-specific features, in: Proceedings – International Conference on Pattern
Recognition, 2016, pp. 1612–1617, https://doi.org/10.1109/ICPR.2016.7899867.[45] J. Huang, G. Li, Q. Huang, X. Wu, Learning label specific features for multi-label classification, in: Proceedings – IEEE International Conference on Data
Mining ICDM, 2016, pp. 181–190, https://doi.org/10.1109/ICDM.2015.67.[46] T. Zhou, D. Tao, Multi-label subspace ensemble, Journal of Machine Learning Research (2012) 1444–1452.[47] A. Büyükçakir, H. Bonab, F. Can, A novel online stacked ensemble for multi-label stream classification, in: Proceedings of the 27th ACM International
Conference on Information and Knowledge Management, 2018.[48] W. Zhang, J. Yan, X. Wang, H. Zha, Deep extreme multi-label learning, in: ICMR 2018 – Proceedings of the 2018 ACM International Conference on
Multimedia Retrieval, 2018, pp. 100–107, doi: 10.1145/3206025.3206030, arXiv:1704.03718.[49] S. Wu, F. Crestani, A geometric framework for data fusion in information retrieval, Information Systems (2015).[50] H.R. Bonab, F. Can, GOOWE: Geometrically Optimum and Online-Weighted Ensemble Classifier for Evolving Data Streams, ACM Transactions on
Knowledge Discovery from Data 12 (2018) 1–33, https://doi.org/10.1145/3139240.[51] E.C. Polley, M.J. van der Laan, Super Learner in Prediction, U.C. Berkeley Division of Biostatistics Working Paper (2010).
Please cite this article as: Y. Xia, K. Chen and Y. Yang, Multi-label classification with weighted classifier selection and stacked ensemble,Information Sciences, https://doi.org/10.1016/j.ins.2020.06.017
