Research ArticleA Hybrid Classification System for Heart Disease DiagnosisBased on the RFRS Method
Xiao Liu1 Xiaoli Wang1 Qiang Su1 Mo Zhang2 Yanhong Zhu3
Qiugen Wang4 and Qian Wang4
1School of Economics and Management Tongji University Shanghai China2School of Economics and Management Shanghai Maritime University Shanghai China3Department of Scientific Research Shanghai General Hospital School of Medicine Shanghai Jiaotong University Shanghai China4Trauma Center Shanghai General Hospital School of Medicine Shanghai Jiaotong University Shanghai China
Correspondence should be addressed to Xiaoli Wang xiaoli-wangtongjieducn
Received 19 March 2016 Revised 12 July 2016 Accepted 1 August 2016 Published 3 January 2017
Academic Editor Xiaopeng Zhao
Copyright copy 2017 Xiao Liu et alThis is an open access article distributed under the Creative Commons Attribution License whichpermits unrestricted use distribution and reproduction in any medium provided the original work is properly cited
Heart disease is one of the most common diseases in the world The objective of this study is to aid the diagnosis of heart diseaseusing a hybrid classification system based on the ReliefF and Rough Set (RFRS) method The proposed system contains twosubsystems the RFRS feature selection system and a classification systemwith an ensemble classifierThe first system includes threestages (i) data discretization (ii) feature extraction using theReliefF algorithm and (iii) feature reduction using the heuristic RoughSet reduction algorithm that we developed In the second system an ensemble classifier is proposed based on the C45 classifierTheStatlog (Heart) dataset obtained from the UCI database was used for experiments A maximum classification accuracy of 9259was achieved according to a jackknife cross-validation scheme The results demonstrate that the performance of the proposedsystem is superior to the performances of previously reported classification techniques
1 Introduction
Cardiovascular disease (CVD) is a primary cause of deathAn estimated 175 million people died from CVD in2012 representing 31 of all global deaths (httpwwwwhointmediacentrefactsheetsfs317en) In the United Statesheart disease kills one person every 34 seconds [1]
Numerous factors are involved in the diagnosis of heartdisease which complicates a physicianrsquos task To help physi-ciansmake quick decisions andminimize errors in diagnosisclassification systems enable physicians to rapidly examinemedical data in considerable detail [2] These systems areimplemented by developing a model that can classify existingrecords using sample data Various classification algorithmshave been developed and used as classifiers to assist doctorsin diagnosing heart disease patients
The performances obtained using the Statlog (Heart)dataset [3] from the UCI machine learning database arecompared in this context Lee [4] proposed a novel supervised
feature selection method based on the bounded sum ofweighted fuzzy membership functions (BSWFM) andEuclidean distances and obtained an accuracy of 874Tomar and Agarwal [5] used the F-score feature selectionmethod and the Least Square Twin Support Vector Machine(LSTSVM) to diagnose heart diseases obtaining an averageclassification accuracy of 8559 Buscema et al [6] used theTraining with Input Selection and Testing (TWIST)algorithm to classify patterns obtaining an accuracy of8414 The Extreme LearningMachine (ELM) has also beenused as a classifier obtaining a reported classificationaccuracy of 875 [7] The genetic algorithm with the NaıveBayes classifier has been shown to have a classificationaccuracy of 8587 [8] Srinivas et al [9] obtained an 8370classification accuracy using Naıve Bayes Polat and Gunes[10] used the RBF kernel F-score feature selection method todetect heart disease The LS-SVM classifier was usedobtaining a classification accuracy of 8370 In [11] the GA-AWAIS method was used for heart disease detection with
HindawiComputational and Mathematical Methods in MedicineVolume 2017 Article ID 8272091 11 pageshttpsdoiorg10115520178272091
2 Computational and Mathematical Methods in Medicine
a classification accuracy of 8743 The Algebraic SigmoidMethod has also been proposed to classify heart diseasewith a reported accuracy of 8524 [12] Wang et al [13]used linear kernel SVM classifiers for heart disease detectionand obtained an accuracy of 8337 In [14] three distancecriteria were applied in simple AIS and the accuracyobtained on the Statlog (Heart) dataset was 8395 In[15] a hybrid neural network method was proposed andthe reported accuracy was 868 Yan et al [16] achieved an8375 classification accuracy using ICA and SVMclassifiersSahan et al [17] proposed a new artificial immune systemnamed the Attribute Weighted Artificial Immune System(AWAIS) and obtained an accuracy of 8259 using thek-fold cross-validation method In [18] the k-NN k-NN with Manhattan feature space mapping (FSM) andseparability split value (SSV) algorithms were used for heartdisease detection and the highest classification accuracy(856) was obtained by k-NN
From these works it can be observed that feature selec-tion methods can effectively increase the performance ofsingle classifier algorithms in diagnosing heart disease [19]Noisy features and dependency relationships in the heartdisease dataset can influence the diagnosis process Typicallythere are numerous records of accompanied syndromes in theoriginal datasets aswell as a large number of redundant symp-toms Consequently it is necessary to reduce the dimensionsof the original feature set by a feature selection method thatcan remove the irrelevant and redundant features
ReliefF is one of the most popular and successful featureestimation algorithms It can accurately estimate the qualityof features with strong dependencies and is not affected bytheir relations [20] There are two advantages to using theReliefF algorithm (i) it follows the filter approach and doesnot employ domain specific knowledge to set feature weights[21 22] and (ii) it is a feature weighting (FW) engineeringtechnique ReliefF assigns a weight to each feature thatrepresents the usefulness of that feature for distinguishingpattern classes First theweight vector can be used to improvethe performance of the lazy algorithms [21] Furthermorethe weight vector can also be used as a method for rankingfeatures to guide the search for the best subset of features[22ndash26] The ReliefF algorithm has proved its usefulness inFS [20 23] feature ranking [27] and building tree-basedmodels [22] with an association rules-based classifier [28]in improving the efficiencies of the genetic algorithms [29]and with lazy classifiers [21]
ReliefF has excellent performance in both supervised andunsupervised learning However it does not help identifyredundant features [30ndash32] ReliefF algorithm estimates thequality of each feature according to its weight When most ofthe given features are relevant to the concept this algorithmwill select most of them even though only some fractionis necessary for concept description [32] Furthermore theReliefF algorithm does not attempt to determine the usefulsubsets of these weakly relevant features [33]
Redundant features increase dimensionality unnecessar-ily [34] and adversely affect learning performancewhen facedwith shortage of data It has also been empirically shownthat removing redundant features can result in significant
performance improvement [35] Rough Set (RS) theory is anewmathematical approach to data analysis and data miningthat has been applied successfully to many real-life problemsin medicine pharmacology engineering banking financialand market analysis and others [36] The RS reductionalgorithm can reduce all redundant features of datasets andseek the minimum subset of features to attain a satisfactoryclassification [37]
There are three advantages to combining ReliefF and RS(RFRS) approach as an integrated feature selection system forheart disease diagnosis
(i)The RFRSmethod can remove superfluous and redun-dant features more effectively The ReliefF algorithm canselect relevant features for disease diagnosis however redun-dant featuresmay still exist in the selected relevant features Insuch cases the RS reduction algorithm can remove remainingredundant features to offset this limitation of the ReliefFalgorithm
(ii)TheRFRSmethod helps to accelerate the RS reductionprocess and guide the search of the reducts Finding aminimal reduct of a given information system is an NP-hardproblem as was demonstrated in [38] The complexity ofcomputing all reducts in an information system is ratherhigh [39] On one hand as a data preprocessing tool thefeatures revealed by the ReliefF method can accelerate theoperation process by serving as the input for the RS reductionalgorithm On the other hand the weight vector obtainedby the ReliefF algorithm can act as a heuristic to guide thesearch for the reducts [25 26] thus helping to improve theperformance of the heuristic algorithm [21]
(iii) The RFRS method can reduce the number andimprove the quality of reducts Usually more than one reductexists in the dataset and larger numbers of features result inlarger numbers of reducts [40] The number of reducts willdecrease if superfluous features are removed using the ReliefFalgorithm When unnecessary features are removed moreimportant features can be extracted which will also improvethe quality of reducts
It is obvious that the choice of an efficient feature selectionmethod and an excellent classifier is extremely important forthe heart disease diagnosis problem [41] Most of the com-mon classifiers from the machine learning community havebeen used for heart disease diagnosis It is now recognizedthat no single model exists that is superior for all patternrecognition problems andno single technique is applicable toall problems [42] One solution to overcome the limitations ofa single classifier is to use an ensemble model An ensemblemodel is a multiclassifier combination model that results inmore precise decisions because the same problem is solved byseveral different trained classifiers which reduces the vari-ance of error estimation [43] In recent years ensemble learn-ing has been employed to increase classification accuraciesbeyond the level that can be achieved by individual classifiers[44 45] In this paper we used an ensemble classifier toevaluate the feature selection model
To improve the efficiency and effectiveness of the classi-fication performance for the diagnosis of heart disease wepropose a hybrid classification system based on the ReliefFand RS (RFRS) approach in handling relevant and redundant
Computational and Mathematical Methods in Medicine 3
features The system contains two subsystems the RFRSfeature selection subsystem and a classification subsystemIn the RFRS feature selection subsystem we use a two-stage hybrid modeling procedure by integrating ReliefF withthe RS (RFRS) method First the proposed method adoptsthe ReliefF algorithm to obtain feature weights and selectmore relevant and important features from heart diseasedatasets Then the feature estimation obtained from the firstphase is used as the input for the RS reduction algorithmand guide the initialization of the necessary parameters forthe genetic algorithm We use a GA-based search engine tofind satisfactory reducts In the classification subsystem theresulting reducts serve as the input for the chosen classifiersFinally the optimal reduct and performance can be obtained
To evaluate the performance of the proposed hybridmethod a confusion matrix sensitivity specificity accuracyand ROC were used The experimental results show that theproposed method achieves very promising results using thejack knife test
The main contributions of this paper are summarized asfollows
(i) We propose a feature selection system to integrate theReliefF approach with the RS method (RFRS) to detect heartdisease in an efficient and effective way The idea is to use thefeature estimation from the ReliefF phase as the input andheuristics for the RS reduction phase
(ii) In the classification system we propose an ensembleclassifier using C45 as the base classifier Ensemble learningcan achieve better performance at the cost of computationthan single classifiers The experimental results show that theensemble classifier in this paper is superior to three commonclassifiers
(iii) Compared with three classifiers and previous studiesthe proposed diagnostic system achieved excellent classifi-cation results On the Statlog (Heart) dataset from the UCImachine learning database [3] the resulting classificationaccuracy was 9259 which is higher than that achieved byother studies
The rest of the paper is organized as follows Section 2offers brief background information concerning the ReliefFalgorithm and RS theory The details of the diagnosis sys-tem implementation are presented in Section 3 Section 4describes the experimental results and discusses the pro-posed method Finally conclusions and recommendationsfor future work are summarized in Section 5
2 Theoretical Background
21 Basic Concepts of Rough Set Theory Rough Set (RS)theory which was proposed by Pawlak in the early 1980sis a new mathematical approach to addressing vaguenessand uncertainty [46] RS theory has been applied in manydomains including classification system analysis patternreorganization and data mining [47] RS-based classificationalgorithms are based on equivalence relations and have beenused as classifiers in medical diagnosis [37 46] In this paperwe primarily focus on the RS reduction algorithm whichcan reduce all redundant features of datasets and seek the
minimum subset of features necessary to attain a satisfactoryclassification [37] A few basic concepts of RS theory aredefined [46 47] as follows
Definition 1 U is a certain set that is referred to as theuniverse R is an equivalence relation in U The pair 119860 =(119880 119877) is referred to as an approximation space
Definition 2 119875 sub 119877 cap119875 (the intersection of all equivalencerelations in P) is an equivalence relation which is referredto as the R-indiscernibility relation and it is represented byInd(119877)
Definition 3 Let X be a certain subset of U The leastcomposed set in R that contains X is referred to as the bestupper approximation ofX in R and represented by119877minus(119883) thegreatest composed set in R contained inX is referred to as thebest lower approximation of X in R and it is represented by119877minus(119883)
119877minus (119883) = 119909 isin 119880 [119909]119877 sub 119883
whereU is the universe that consists of a finite set of n objects119860 = 119862 cup 119863 in which C is a set of condition attributes andD is a set of decision attributes V is the set of domains ofattributes and F is the information function for each 119886 isin 119860119909 isin 119880 119865(119909 119886) isin 119881119886
Definition 5 In an information system C and D are sets ofattributes in119880119883 isin 119880ind(119876) and pos119901(119876) which is referredto as a positive region is defined as
pos119901 (119876) = cup119875minus (119883) (3)
Definition 6 P and Q are sets of attributes in U 119875119876 sube 119860and the dependency 119903119901(119876) is defined as
119903119901 (119876) =card (pos119901 (119876))
card (119880) (4)
Card (X) denotes the cardinality of X 0 le 119903119901(119876) le 1
Definition 7 P andQ are sets of attributes inU 119875119876 sube 119860 andthe significance of 119886119894 is defined as
sig (119886119894) = 119903119901 (119876) minus 119903119901minus119886119894 (119876) (5)
22 ReliefF Algorithm Many feature selection algorithmshave been developed ReliefF is one of the most widely usedand effective algorithms [48] ReliefF is a simple yet efficientprocedure for estimating the quality of features in problemswith dependencies between features [20] The pseudocode ofReliefF algorithm is listed in Algorithm 1
4 Computational and Mathematical Methods in Medicine
ReliefF algorithmInput A decision table 119878 = (119880 119875 119876)Output the vector119882 of estimations of the qualities of features(1) set all weights119882[119860] fl 00(2) for 119894 fl 1 to119898 do begin(3) randomly select a sample 119877119894(4) find 119896 nearest hits119867119895(5) for each class 119862 = class(119877119894) do(6) from class 119862 find 119896 nearest misses119872119895(119862)(7) for 119860 fl 1 to a do(8) 119882[119860] = 119882[119860] minus sum119896119895=1 diff(119860 119877119894 119867119895)119898119896 + sum119862 =class(119877119894)[119875(119862)1 minus 119875(class(119877119894)) sum
31 Overview The proposed hybrid classification systemconsists of two main components (i) feature selection usingthe RFRS subsystem and (ii) data classification using theclassification system A flow chart of the proposed systemis shown in Figure 1 We describe the preprocessing andclassification systems in the following subsections
32 RFRS Feature Selection Subsystem We propose a two-phase feature selection method based on the ReliefF algo-rithm and the RS (RFRS) algorithm The idea is to use thefeature estimation from the ReliefF phase as the input andheuristics for the subsequent RS reduction phase In thefirst phase we adopt the ReliefF algorithm to obtain featureweights and select important features in the second phasethe feature estimation obtained from the first phase is usedto guide the initialization of the parameters required for thegenetic algorithm We use a GA-based search engine to findsatisfactory reducts
The RFRS feature selection subsystem consists of threemain modules (i) data discretization (ii) feature extractionusing the ReliefF algorithm and (iii) feature reduction usingthe heuristic RS reduction algorithm we propose
321 Data Discretization RS reduction requires categoricaldata Consequently data discretization is the first step Weused an approximate equal interval binning method to binthe data variables into a small number of categories
322 Feature Extraction by the ReliefF Algorithm Module 2is used for feature extraction by the ReliefF algorithm To dealwith incomplete data we change the diff function Missingfeature values are treated probabilistically [20] We calculatethe probability that two given instances have different valuesfor a given feature conditioned over the class value [20]When one instance has an unknown value then
diff (119860 1198681 1198682) = 1 minus 119875 (value (119860 1198682) | class (1198681)) (6)
When both instances have unknown values then
diff (119860 1198681 1198682)
= 1
minusvalues(119860)sum119881
(119875 (119881 | class (1198681)) times 119875 (119881 | class (1198682)))
(7)
Conditional probabilities are approximated by relativefrequencies in the training set The process of feature extrac-tion is shown as follows
The Process of Feature Extraction Using ReliefF Algorithm
Input A decision table 119878 = (119880 119875 119876) 119875 = 1198861 1198862 119886119898119876 = 1198891 1198892 119889119899 (119898 ge 1 119899 ge 1)
Output The selected feature subset 119870 = 1198861 1198862 119886119896(1 le119896 le 119898)
Step 1 Obtain the weight matrix of each feature using ReliefFalgorithm119882 = 1199081 1199082 119908119894 119908119898 (1 le 119894 le 119898)
Step 2 Set a threshold 120575
Step 3 If 119908119894 gt 120575 then feature 119886119894 is selected
323 Feature Reduction by the Heuristic RS Reduction Algo-rithm The evaluation result obtained by the ReliefF algo-rithm is the feature rank A higher ranking means that thefeature has stronger distinguishing qualities and a higherweight [30] Consequently in the process of reduct searchingthe features in the front rank should have a higher probabilityof being selected
We proposed the RS reduction algorithmby using the fea-ture estimation as heuristics and a GA-based search engine tosearch for the satisfactory reducts The pseudocode of thealgorithm is provided in Algorithm 2 The algorithm wasimplemented in MATLAB R2014a
33 Classification Subsystem In the classification subsystemthe dataset is split into training sets and corresponding test
Computational and Mathematical Methods in Medicine 5
RFRSfeatureselectionsubsystem
Heart disease dataset Data discretization
Classificationsubsystem
An ensemble classifierCross-validation
ReliefF algorithm
Feature extraction
Feature reduction
Heuristic RS reduction algorithm
Optimal performance pi Optimal reduct rij
S = (UA V f) A = C cup D C = C1 D = D1 m ge 1 n ge 1C2 Cm D2 Dn
K = C1 (1 le k le m)C2 Ck
Reducts R = R1 Ri = ri1 i ge 1 1 le j le kR2 Ri ri2 rij
Training set T = T1 Ti = ti1 D i ge 1 1 le j le kti2 tijT2 Ti
Test set V = V1 Vi = i1 D i ge 1 1 le j le kV2 Vi i2 ij
Performance set P = p1 pi = pi1 i ge 1 1 le j le kpi2 pijp2 pi
Trained set T998400 = T9984001 T998400i = t998400i1 D i ge 1 1 le j le kT9984002 T998400i t998400i2 t
998400ij
Figure 1 Structure of RFRS-based classification system
sets The decision tree is a nonparametric learning algorithmthat does not need to search for optimal parameters in thetraining stage and thus is used as a weak learner for ensemblelearning [49] In this paper the ensemble classifier uses theC45 decision tree as the base classifier We use the boostingtechnique to construct ensemble classifiers Jackknife cross-validation is used to increase the amount of data for testingthe results The optimal reduct is the reduct that obtains thebest classification accuracy
4 Experimental Results
41 Dataset TheStatlog (Heart) dataset used in ourworkwasobtained from the UCI machine learning database [3] Thisdataset contains 270 observations and 2 classes the presence
and absence of heart disease The samples include 13 condi-tion features presented in Table 1 We denote the 13 featuresas C1 to C13
42 Performance Evaluation Methods
421 Confusion Matrix Sensitivity Specificity and AccuracyA confusion matrix [50] contains information about actualand predicted classifications performed by a classificationsystem The performance of such systems is commonlyevaluated using the data in the matrix Table 2 shows theconfusion matrix for a two-class classifier
In the confusion matrix TP is the number of true posi-tives representing the cases with heart disease that are cor-rectly classified into the heart disease class FN is the numberof false negatives representing cases with heart disease that
6 Computational and Mathematical Methods in Medicine
Heuristic RS reduction algorithmInput a decision table 119878 = (119880 119862119863) 119862 = 1198881 1198882 119888119898119863 = 1198891 1198892 119889119899Output RedStep 1 Return Core(1) Corelarr (2) For 119894 = 1 to119898(3) Select 119888119894 from 119862(4) Calculate Ind(119862) Ind(119862 minus 119888119894) and Ind(119863)(5) Calculate pos119862(119863) pos119862minus119888119894 (119863) and 119903119862(119863)(6) Calculate Sig(119886119894) Sig(119886119894) = 119903119862(119863) minus 119903119862minus119888119894 (119863)(7) If sig(119888119894) = 0(8) core = core cup 119888119894(9) End if(10) End forStep 2 Return Red(1) Red = Core(2) 1198621015840 = 119862 minus Red(3) While Sig(Red119863) = Sig(119862119863) do
Compute the weight of each feature 119888 in 1198621015840 using the ReliefF algorithmSelect a feature 119888 according to its weight let Red=Red cup 119888Initialize all the necessary parameters for the GA-based search engine according to theresults of the last step and search for satisfactory reducts
End while
Algorithm 2 Pseudocode of heuristic RS reduction algorithm
are classified into the healthy class TN is the number of truenegatives representing healthy cases that are correctly classi-fied into the healthy class Finally FP is the number of falsepositives representing the healthy cases that are incorrectlyclassified into the heart disease class [50]
The performance of the proposed system was evaluatedbased on sensitivity specificity and accuracy tests which usethe true positive (TP) true negative (TN) false negative (FN)and false positive (FP) terms [33]These criteria are calculatedas follows [41]
Sensitivity (Sn) = TPTP + FN
times 100
Specificity (Sp) = TNFP + TN
times 100
Accuracy (Acc) = TP + TNTP + TN + FP + FN
times 100
(8)
422 Cross-Validation Three cross-validation methodsnamely subsampling tests independent dataset tests andjackknife tests are often employed to evaluate the predictivecapability of a predictor [51] Among the three methodsthe jackknife test is deemed the least arbitrary and the mostobjective and rigorous [52 53] because it always yields aunique outcome as demonstrated by a penetrating analysis ina recent comprehensive review [54 55] Therefore thejackknife test has been widely and increasingly adopted inmany areas [56 57]
Accordingly the jackknife test was employed to examinethe performance of the model proposed in this paper Forjackknife cross-validation each sequence in the training
dataset is in turn singled out as an independent test sampleand all the parameter rules are calculated based on theremaining samples without including the one being treatedas the test sample
423 Receiver Operating Characteristics (ROC) The receiveroperating characteristic (ROC) curve is used for analyzing theprediction performance of a predictor [58] It is usually plot-ted using the true positive rate versus the false positive rate asthe discrimination threshold of classification algorithm isvaried The area under the ROC curve (AUC) is widely usedand relatively accepted in classification studies because itprovides a good summary of a classifierrsquos performance [59]
43 Results and Discussion
431 Results and Analysis on the Statlog (Heart) DatasetFirst we used the equal interval binningmethod to discretizethe original data In the feature extraction module the num-ber of k-nearest neighbors in the ReliefF algorithm was set to10 and the threshold 120575 was set to 002 Table 3 summarizesthe results of the ReliefF algorithm Based on these resultsC5and C6 were removed In Module 3 we obtained 15 reductsusing the heuristic RS reduction algorithm implemented inMATLAB 2014a
Trials were conducted using 70ndash30 training-test par-titions using all the reduced feature sets Jackknife cross-validation was performed on the dataset The number ofdesired base classifiers k was set to 50 100 and 150 Thecalculations were run 10 times and the highest classification
Computational and Mathematical Methods in Medicine 7
Table 1 Feature information of Statlog (Heart) dataset
Feature Code Description Domain Data type Mean Standard deviationAge 1198621 mdash 29ndash77 Real 54 9Sex 1198622 Male female 0 1 Binary mdash mdash
Chest pain type 1198623Angina
asymptomaticabnormal
1 2 3 4 Nominal mdash mdash
Resting blood pressure 1198624 mdash 94ndash200 Real 131344 17862Serum cholesterol in mgdl 1198625 mdash 126ndash564 Real 249659 51686Fasting blood sugar gt 120mgdl 1198626 mdash 0 1 Binary mdash mdash
Resting electrocardiographic results 1198627Norm
abnormal hyper 0 1 2 Nominal mdash mdash
Maximum heart rate achieved 1198628 mdash 71ndash202 Real 149678 231666Exercise-induced angina 1198629 mdash 0 1 Binary mdash mdashOld peak = ST depression induced by exercise relative to rest 11986210 mdash 0ndash62 Real 105 1145Slope of the peak exercise ST segment 11986211 Up flat down 1 2 3 Ordered mdash mdashNumber of major vessels (0ndash3) colored by fluoroscopy 11986212 mdash 0 1 2 3 Real mdash mdash
Thal 11986213Normal fixed
defectreversible defect
3 6 7 Nominal mdash mdash
Table 2 The confusion matrix
Predicted patientswith heart disease
Predicted healthypersons
Actual patients withheart disease True positive (TP) False negative (FN)
Actual healthypersons False positive (FP) True negative (TN)
performances for each training-test partition are provided inTable 4
In Table 4 1198772 obtains the best test set classificationaccuracy (9259) using the ensemble classifiers when 119896 =100 The training process is shown in Figure 2 The trainingand test ROC curves are shown in Figure 3
432 Comparison with Other Classifiers In this sectionour ensemble classification method is compared with theindividual C45 decision tree and Naıve Bayes and BayesianNeural Networks (BNN)methodsTheC45 decision tree andNaıve Bayes are common classifiers Bayesian Neural Net-works (BNN) is a classifier that uses Bayesian regularizationto train feed-forward neural networks [60] and has betterperformance than pure neural networks The classificationaccuracy results of the four classifiers are listed in Table 5Theensemble classification method has better performance thanthe individual C45 classifier and the other two classifiers
433 Comparison of the Results with Other Studies Wecompared our results with the results of other studies Table 6shows the classification accuracies of our study and previousmethods
0 10 20 30 40 50 60 70 80 90 100008
01
012
014
016
018
02
022
024
Number of trees
Test
class
ifica
tion
erro
r
Figure 2 Training process of 1198777
The results show that our proposed method obtainssuperior and promising results in classifying heart diseasepatients We believe that the proposed RFRS-based classi-fication system can be exceedingly beneficial in assistingphysicians in making accurate decisions
8 Computational and Mathematical Methods in Medicine
In this paper a novel ReliefF and Rough Set- (RFRS-)based classification system is proposed for heart diseasediagnosisThemain novelty of this paper lies in the proposedapproach the combination of the ReliefF and RS methodsto classify heart disease problems in an efficient and fastmanner The RFRS classification system consists of twosubsystems the RFRS feature selection subsystem and theclassification subsystemThe Statlog (Heart) dataset from theUCI machine learning database [3] was selected to test thesystemThe experimental results show that the reduct1198772 (11986211198623 1198627 1198628 11986211 11986212 11986213) achieves the highest classificationaccuracy (9259) using an ensemble classifier with the C45decision tree as the weak learner The results also show thatthe RFRS method has superior performance compared to
three common classifiers in terms of ACC sensitivity andspecificity In addition the performance of the proposedsystem is superior to that of existingmethods in the literatureBased on empirical analysis the results indicate that theproposed classification system can be used as a promisingalternative tool in medical decision making for heart diseasediagnosis
However the proposed method also has some weak-nesses The number of the nearest neighbors (119896) and theweight threshold (120579) are not stable in the ReliefF algorithm[20] One solution to this problem is to compute estimatesfor all possible numbers and take the highest estimate of eachfeature as the final result [20] We need to perform moreexperiments to find the optimal parameter values for theReliefF algorithm in the future
Computational and Mathematical Methods in Medicine 9
Table 6 Comparison of our results with those of other studies
Author Method Classification accuracy ()Our study RFRS classification system 9259Lee [4] Graphical characteristics of BSWFM combined with Euclidean distance 874Tomar and Agarwal [5] Feature selection-based LSTSVM 8559Buscema et al [6] TWIST algorithm 8414Subbulakshmi et al [7] ELM 875Karegowda et al [8] GA + Naıve Bayes 8587Srinivas et al [9] Naıve Bayes 8370Polat and Gunes [10] RBF kernel 119865-score + LS-SVM 8370Ozsen and Gunes [11] GA-AWAIS 8743Helmy and Rasheed [12] Algebraic Sigmoid 8524Wang et al [13] Linear kernel SVM classifiers 8337Ozsen and Gunes [14] Hybrid similarity measure 8395Kahramanli and Allahverdi [15] Hybrid neural network method 868Yan et al [16] ICA + SVM 8375Sahan et al [17] AWAIS 8259Duch et al [18] KNN classifier 856BSWFM bounded sumof weighted fuzzymembership functions LSTSVM Least Square Twin Support VectorMachine TWIST Training with Input Selectionand Testing ELM Extreme LearningMachine GA genetic algorithm SVM support vectormachine ICA imperialist competitive algorithm AWAIS attributeweighted artificial immune system119870NN 119896-nearest neighbor
Test ROC
0 10
01
02
02
03
04
04
05
06
06
07
08
08
09
1
False positive rate
True
pos
itive
rate
0 10
01
02
02
03
04
04
05
06
06
07
08
08
09
1
False positive rate
True
pos
itive
rate
Training ROC
Figure 3 ROC curves for training and test sets
Competing Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
This study was supported by the National Natural ScienceFoundation of China (Grant no 71432007)
References
[1] K D Kochanek J Xu S L Murphy A M Minino and H-C Kung ldquoDeaths final data for 2009rdquo National Vital StatisticsReports vol 60 no 3 pp 1ndash116 2011
[2] H Temurtas N Yumusak and F Temurtas ldquoA comparativestudy on diabetes disease diagnosis using neural networksrdquoExpert Systems with Applications vol 36 no 4 pp 8610ndash86152009
10 Computational and Mathematical Methods in Medicine
[3] UCI Repository of Machine Learning Databases httparchiveicsuciedumldatasetsStatlog+28Heart29
[4] S-H Lee ldquoFeature selection based on the center of gravity ofBSWFMs using NEWFMrdquo Engineering Applications of ArtificialIntelligence vol 45 pp 482ndash487 2015
[5] D Tomar and S Agarwal ldquoFeature selection based least squaretwin support vector machine for diagnosis of heart diseaserdquoInternational Journal of Bio-Science and Bio-Technology vol 6no 2 pp 69ndash82 2014
[6] M Buscema M Breda and W Lodwick ldquoTraining with InputSelection and Testing (TWIST) algorithm a significant advancein pattern recognition performance of machine learningrdquoJournal of Intelligent Learning Systems and Applications vol 5no 1 pp 29ndash38 2013
[7] C V Subbulakshmi S N Deepa and N Malathi ldquoExtremelearning machine for two category data classificationrdquo inProceedings of the IEEE International Conference on AdvancedCommunication Control and Computing Technologies (ICAC-CCT rsquo12) pp 458ndash461 Ramanathapuram India August 2012
[8] A G Karegowda A SManjunath andM A Jayaram ldquoFeaturesubset selection problem using wrapper approach in supervisedlearningrdquo International Journal of Computer Applications vol 1no 7 pp 13ndash17 2010
[9] K Srinivas B K Rani and A Govrdhan ldquoApplications ofdata mining techniques in healthcare and prediction of heartattacksrdquo International Journal on Computer Science and Engi-neering vol 2 pp 250ndash255 2010
[10] K Polat and S Gunes ldquoA new feature selection methodon classification of medical datasets kernel F-score featureselectionrdquo Expert Systems with Applications vol 36 no 7 pp10367ndash10373 2009
[11] S Ozsen and S Gunes ldquoAttribute weighting via genetic algo-rithms for attributeweighted artificial immune system (AWAIS)and its application to heart disease and liver disorders prob-lemsrdquo Expert Systems with Applications vol 36 no 1 pp 386ndash392 2009
[12] T Helmy and Z Rasheed ldquoMulti-category bioinformaticsdataset classification using extreme learning machinerdquo in Pro-ceedings of the IEEE Congress on Evolutionary Computation(CEC rsquo09) pp 3234ndash3240 Trondheim Norway May 2009
[13] S-J Wang A Mathew Y Chen L-F Xi L Ma and JLee ldquoEmpirical analysis of support vector machine ensembleclassifiersrdquo Expert Systems with Applications vol 36 no 3 pp6466ndash6476 2009
[14] S Ozsen and S Gunes ldquoEffect of feature-type in selectingdistance measure for an artificial immune system as a patternrecognizerrdquoDigital Signal Processing vol 18 no 4 pp 635ndash6452008
[15] H Kahramanli and N Allahverdi ldquoDesign of a hybrid systemfor the diabetes and heart diseasesrdquo Expert Systems withApplications vol 35 no 1-2 pp 82ndash89 2008
[16] G Yan G Ma J Lv and B Song ldquoCombining independentcomponent analysis with support vector machinesrdquo in Pro-ceedings of the in 1st International Symposium on Systems andControl in Aerospace and Astronautics (ISSCAA rsquo06) pp 493ndash496 Harbin China January 2006
[17] S Sahan K Polat H Kodaz and S Gunes ldquoThe medicalapplications of attribute weighted artificial immune system(AWAIS) diagnosis of heart and diabetes diseasesrdquo ArtificialImmune Systems vol 3627 pp 456ndash468 2005
[18] W Duch R Adamczak and K Grabczewski ldquoA new method-ology of extraction optimization and application of crisp and
fuzzy logical rulesrdquo IEEE Transactions on Neural Networks vol12 no 2 pp 277ndash306 2001
[19] M P McRae B Bozkurt C M Ballantyne et al ldquoCardiacScoreCard a diagnostic multivariate index assay system forpredicting a spectrumof cardiovascular diseaserdquoExpert Systemswith Applications vol 54 pp 136ndash147 2016
[20] M Robnik-Sikonja and I Kononenko ldquoTheoretical and empir-ical analysis of ReliefF and RReliefFrdquoMachine Learning vol 53no 1-2 pp 23ndash69 2003
[21] D Wettschereck D W Aha and T Mohri ldquoA review andempirical evaluation of feature weighting methods for a classof lazy learning algorithmsrdquo Artificial Intelligence Review vol11 no 1ndash5 pp 273ndash314 1997
[22] I Kononenko E Simec and M R Sikonja ldquoOvercomingthe myopia of inductive learning algorithms with RELIEFFrdquoApplied Intelligence vol 7 no 1 pp 39ndash55 1997
[23] K Kira and L Rendell ldquoA practical approach to featureselectionrdquo in Proceedings of the International Conference onMachine Learning pp 249ndash256 Morgan Kaufmann AberdeenScotland 1992
[24] I Kononenko ldquoEstimating attributes analysis and extension ofReliefFrdquo in Machine Learning ECML-94 European ConferenceonMachine Learning Catania Italy April 6ndash8 1994 Proceedingsvol 784 of Lecture Notes in Computer Science pp 171ndash182Springer Berlin Germany 1994
[25] R Ruiz J C Riquelme and J S Aguilar-Ruiz ldquoHeuristicsearch over a ranking for feature selectionrdquo in Proceedings of8th InternationalWork-Conference onArtificial NeuralNetworks(IWANN rsquo05) Barcelona Spain June 2005 vol 3512 of LecturesNotes in Computer Science pp 742ndash749 Springer Berlin Ger-many 2005
[26] N Spolaor E A ChermanM CMonard andHD Lee ldquoFilterapproach feature selection methods to support multi-labellearning based on ReliefF and information gainrdquo in Advancesin Artificial IntelligencemdashSBIA 2012 21th Brazilian Symposiumon Artificial Intelligence Curitiba Brazil October 20ndash25 2012Proceedings vol 7589 of Lectures Notes in Computer Science pp72ndash81 Springer Berlin Germany 2012
[27] R Ruiz J C Riquelme and J S Aguilar-Ruiz ldquoFast featureranking algorithmrdquo in Proceedings of the Knowledge-BasedIntelligent Information and Engineering Systems (KES rsquo03) pp325ndash331 Springer Berlin Germany 2003
[28] V Jovanoski and N Lavrac ldquoFeature subset selection inassociation rules learning systemsrdquo in Proceedings of AnalysisWarehousing and Mining the Data pp 74ndash77 1999
[29] J J Liu and J T Y Kwok ldquoAn extended genetic rule inductionalgorithmrdquo in Proceedings of the Congress on EvolutionaryComputation pp 458ndash463 LA Jolla Calif USA July 2000
[30] L-X Zhang J-X Wang Y-N Zhao and Z-H Yang ldquoA novelhybrid feature selection algorithm using ReliefF estimationfor GA-Wrapper searchrdquo in Proceedings of the InternationalConference onMachine Learning andCybernetics vol 1 pp 380ndash384 IEEE Xirsquoan China November 2003
[31] Z Zhao L Wang and H Liu ldquoEfficient spectral featureselectionwithminimum redundancyrdquo in Proceedings of the 24thAAAI Conference on Artificial Intelligence AAAI Atlanta GaUSA 2010
[32] F Nie H Huang X Cai and C H Ding ldquoEfficient androbust feature selection via joint ℓ21-norms minimizationrdquo inAdvances in Neural Information Processing Systems pp 1813ndash1821 MIT Press 2010
Computational and Mathematical Methods in Medicine 11
[33] S-Y Jiang and L-X Wang ldquoEfficient feature selection based oncorrelation measure between continuous and discrete featuresrdquoInformation Processing Letters vol 116 no 2 pp 203ndash215 2016
[34] M J Kearns and U V Vazirani An Introduction to Compu-tational Learning Theory MIT Press Cambridge Mass USA1994
[35] R Bellman Adaptive Control Processes A Guided Tour RANDCorporation Research Studies Princeton University PressPrinceton NJ USA 1961
[36] Z Pawlak ldquoRough set theory and its applications to dataanalysisrdquo Cybernetics and Systems vol 29 no 7 pp 661ndash6881998
[37] A-E Hassanien ldquoRough set approach for attribute reductionand rule generation a case of patients with suspected breastcancerrdquo Journal of the American Society for Information Scienceand Technology vol 55 no 11 pp 954ndash962 2004
[38] A Skowron and C Rauszer ldquoThe discernibility matricesand functions in information systemsrdquo in Intelligent DecisionSupport-Handbook of Applications and Advances of the RoughSetsTheory SystemTheory Knowledge Engineering and ProblemSolving R Słowinski Ed vol 11 pp 331ndash362 KluwerAcademicDordrecht Netherlands 1992
[39] Z Pawlak ldquoRough set approach to knowledge-based decisionsupportrdquo European Journal of Operational Research vol 99 no1 pp 48ndash57 1997
[40] X Wang J Yang X Teng W Xia and R Jensen ldquoFeatureselection based on rough sets and particle swarm optimizationrdquoPattern Recognition Letters vol 28 no 4 pp 459ndash471 2007
[41] C Ma J Ouyang H-L Chen and X-H Zhao ldquoAn efficientdiagnosis system for Parkinsonrsquos disease using kernel-basedextreme learning machine with subtractive clustering featuresweighting approachrdquo Computational and Mathematical Meth-ods in Medicine vol 2014 Article ID 985789 14 pages 2014
[42] A H El-Baz ldquoHybrid intelligent system-based rough set andensemble classifier for breast cancer diagnosisrdquoNeural Comput-ing and Applications vol 26 no 2 pp 437ndash446 2015
[43] J-H Eom S-CKim andB-T Zhang ldquoAptaCDSS-E a classifierensemble-based clinical decision support system for cardiovas-cular disease level predictionrdquo Expert Systems with Applicationsvol 34 no 4 pp 2465ndash2479 2008
[44] Y Ma Ensemble Machine Learning Methods and ApplicationsSpringer New York NY USA 2012
[45] S A Etemad and A Arya ldquoClassification and translation ofstyle and affect in human motion using RBF neural networksrdquoNeurocomputing vol 129 pp 585ndash595 2014
[46] Z Pawlak ldquoRough setsrdquo International Journal of Computer ampInformation Sciences vol 11 no 5 pp 341ndash356 1982
[47] Z Pawlak ldquoRough sets and intelligent data analysisrdquo Informa-tion Sciences vol 147 no 1ndash4 pp 1ndash12 2002
[48] Y Huang P J McCullagh and N D Black ldquoAn optimization ofReliefF for classification in large datasetsrdquoData and KnowledgeEngineering vol 68 no 11 pp 1348ndash1356 2009
[49] J Sun M-Y Jia and H Li ldquoAdaBoost ensemble for financialdistress prediction an empirical comparison with data fromChinese listed companiesrdquo Expert Systems with Applicationsvol 38 no 8 pp 9305ndash9312 2011
[50] R Kohavi and F Provost ldquoGlossary of termsrdquo Machine Learn-ing vol 30 no 2-3 pp 271ndash274 1998
[51] W Chen and H Lin ldquoPrediction of midbody centrosome andkinetochore proteins based on gene ontology informationrdquoBiochemical and Biophysical Research Communications vol 401no 3 pp 382ndash384 2010
[52] K-C Chou and C-T Zhang ldquoPrediction of protein structuralclassesrdquo Critical Reviews in Biochemistry andMolecular Biologyvol 30 no 4 pp 275ndash349 1995
[53] W Chen and H Lin ldquoIdentification of voltage-gated potassiumchannel subfamilies from sequence information using supportvectormachinerdquoComputers in Biology andMedicine vol 42 no4 pp 504ndash507 2012
[54] K-C Chou and H-B Shen ldquoRecent progress in proteinsubcellular location predictionrdquo Analytical Biochemistry vol370 no 1 pp 1ndash16 2007
[55] P Feng H Lin W Chen and Y Zuo ldquoPredicting the typesof J-proteins using clustered amino acidsrdquo BioMed ResearchInternational vol 2014 Article ID 935719 8 pages 2014
[56] K-C Chou and H-B Shen ldquoProtIdent a web server foridentifying proteases and their types by fusing functionaldomain and sequential evolution informationrdquoBiochemical andBiophysical Research Communications vol 376 no 2 pp 321ndash325 2008
[57] K-C Chou and Y-D Cai ldquoPrediction of membrane proteintypes by incorporating amphipathic effectsrdquo Journal of ChemicalInformation and Modeling vol 45 no 2 pp 407ndash413 2005
[58] F Tom ldquoROC graphs notes and practical considerations forresearchersrdquo Machine Learning vol 31 pp 1ndash38 2004
[59] JHuang andCX Ling ldquoUsingAUCand accuracy in evaluatinglearning algorithmsrdquo IEEE Transactions on Knowledge andDataEngineering vol 17 no 3 pp 299ndash310 2005
[60] F D Foresee and M T Hagan ldquoGauss-Newton approximationto Bayesian learningrdquo in Proceedings of the International Con-ference on Neural Networks vol 3 pp 1930ndash1935 Houston TexUSA 1997
2 Computational and Mathematical Methods in Medicine
a classification accuracy of 8743 The Algebraic SigmoidMethod has also been proposed to classify heart diseasewith a reported accuracy of 8524 [12] Wang et al [13]used linear kernel SVM classifiers for heart disease detectionand obtained an accuracy of 8337 In [14] three distancecriteria were applied in simple AIS and the accuracyobtained on the Statlog (Heart) dataset was 8395 In[15] a hybrid neural network method was proposed andthe reported accuracy was 868 Yan et al [16] achieved an8375 classification accuracy using ICA and SVMclassifiersSahan et al [17] proposed a new artificial immune systemnamed the Attribute Weighted Artificial Immune System(AWAIS) and obtained an accuracy of 8259 using thek-fold cross-validation method In [18] the k-NN k-NN with Manhattan feature space mapping (FSM) andseparability split value (SSV) algorithms were used for heartdisease detection and the highest classification accuracy(856) was obtained by k-NN
From these works it can be observed that feature selec-tion methods can effectively increase the performance ofsingle classifier algorithms in diagnosing heart disease [19]Noisy features and dependency relationships in the heartdisease dataset can influence the diagnosis process Typicallythere are numerous records of accompanied syndromes in theoriginal datasets aswell as a large number of redundant symp-toms Consequently it is necessary to reduce the dimensionsof the original feature set by a feature selection method thatcan remove the irrelevant and redundant features
ReliefF is one of the most popular and successful featureestimation algorithms It can accurately estimate the qualityof features with strong dependencies and is not affected bytheir relations [20] There are two advantages to using theReliefF algorithm (i) it follows the filter approach and doesnot employ domain specific knowledge to set feature weights[21 22] and (ii) it is a feature weighting (FW) engineeringtechnique ReliefF assigns a weight to each feature thatrepresents the usefulness of that feature for distinguishingpattern classes First theweight vector can be used to improvethe performance of the lazy algorithms [21] Furthermorethe weight vector can also be used as a method for rankingfeatures to guide the search for the best subset of features[22ndash26] The ReliefF algorithm has proved its usefulness inFS [20 23] feature ranking [27] and building tree-basedmodels [22] with an association rules-based classifier [28]in improving the efficiencies of the genetic algorithms [29]and with lazy classifiers [21]
ReliefF has excellent performance in both supervised andunsupervised learning However it does not help identifyredundant features [30ndash32] ReliefF algorithm estimates thequality of each feature according to its weight When most ofthe given features are relevant to the concept this algorithmwill select most of them even though only some fractionis necessary for concept description [32] Furthermore theReliefF algorithm does not attempt to determine the usefulsubsets of these weakly relevant features [33]
Redundant features increase dimensionality unnecessar-ily [34] and adversely affect learning performancewhen facedwith shortage of data It has also been empirically shownthat removing redundant features can result in significant
performance improvement [35] Rough Set (RS) theory is anewmathematical approach to data analysis and data miningthat has been applied successfully to many real-life problemsin medicine pharmacology engineering banking financialand market analysis and others [36] The RS reductionalgorithm can reduce all redundant features of datasets andseek the minimum subset of features to attain a satisfactoryclassification [37]
There are three advantages to combining ReliefF and RS(RFRS) approach as an integrated feature selection system forheart disease diagnosis
(i)The RFRSmethod can remove superfluous and redun-dant features more effectively The ReliefF algorithm canselect relevant features for disease diagnosis however redun-dant featuresmay still exist in the selected relevant features Insuch cases the RS reduction algorithm can remove remainingredundant features to offset this limitation of the ReliefFalgorithm
(ii)TheRFRSmethod helps to accelerate the RS reductionprocess and guide the search of the reducts Finding aminimal reduct of a given information system is an NP-hardproblem as was demonstrated in [38] The complexity ofcomputing all reducts in an information system is ratherhigh [39] On one hand as a data preprocessing tool thefeatures revealed by the ReliefF method can accelerate theoperation process by serving as the input for the RS reductionalgorithm On the other hand the weight vector obtainedby the ReliefF algorithm can act as a heuristic to guide thesearch for the reducts [25 26] thus helping to improve theperformance of the heuristic algorithm [21]
(iii) The RFRS method can reduce the number andimprove the quality of reducts Usually more than one reductexists in the dataset and larger numbers of features result inlarger numbers of reducts [40] The number of reducts willdecrease if superfluous features are removed using the ReliefFalgorithm When unnecessary features are removed moreimportant features can be extracted which will also improvethe quality of reducts
It is obvious that the choice of an efficient feature selectionmethod and an excellent classifier is extremely important forthe heart disease diagnosis problem [41] Most of the com-mon classifiers from the machine learning community havebeen used for heart disease diagnosis It is now recognizedthat no single model exists that is superior for all patternrecognition problems andno single technique is applicable toall problems [42] One solution to overcome the limitations ofa single classifier is to use an ensemble model An ensemblemodel is a multiclassifier combination model that results inmore precise decisions because the same problem is solved byseveral different trained classifiers which reduces the vari-ance of error estimation [43] In recent years ensemble learn-ing has been employed to increase classification accuraciesbeyond the level that can be achieved by individual classifiers[44 45] In this paper we used an ensemble classifier toevaluate the feature selection model
To improve the efficiency and effectiveness of the classi-fication performance for the diagnosis of heart disease wepropose a hybrid classification system based on the ReliefFand RS (RFRS) approach in handling relevant and redundant
Computational and Mathematical Methods in Medicine 3
features The system contains two subsystems the RFRSfeature selection subsystem and a classification subsystemIn the RFRS feature selection subsystem we use a two-stage hybrid modeling procedure by integrating ReliefF withthe RS (RFRS) method First the proposed method adoptsthe ReliefF algorithm to obtain feature weights and selectmore relevant and important features from heart diseasedatasets Then the feature estimation obtained from the firstphase is used as the input for the RS reduction algorithmand guide the initialization of the necessary parameters forthe genetic algorithm We use a GA-based search engine tofind satisfactory reducts In the classification subsystem theresulting reducts serve as the input for the chosen classifiersFinally the optimal reduct and performance can be obtained
To evaluate the performance of the proposed hybridmethod a confusion matrix sensitivity specificity accuracyand ROC were used The experimental results show that theproposed method achieves very promising results using thejack knife test
The main contributions of this paper are summarized asfollows
(i) We propose a feature selection system to integrate theReliefF approach with the RS method (RFRS) to detect heartdisease in an efficient and effective way The idea is to use thefeature estimation from the ReliefF phase as the input andheuristics for the RS reduction phase
(ii) In the classification system we propose an ensembleclassifier using C45 as the base classifier Ensemble learningcan achieve better performance at the cost of computationthan single classifiers The experimental results show that theensemble classifier in this paper is superior to three commonclassifiers
(iii) Compared with three classifiers and previous studiesthe proposed diagnostic system achieved excellent classifi-cation results On the Statlog (Heart) dataset from the UCImachine learning database [3] the resulting classificationaccuracy was 9259 which is higher than that achieved byother studies
The rest of the paper is organized as follows Section 2offers brief background information concerning the ReliefFalgorithm and RS theory The details of the diagnosis sys-tem implementation are presented in Section 3 Section 4describes the experimental results and discusses the pro-posed method Finally conclusions and recommendationsfor future work are summarized in Section 5
2 Theoretical Background
21 Basic Concepts of Rough Set Theory Rough Set (RS)theory which was proposed by Pawlak in the early 1980sis a new mathematical approach to addressing vaguenessand uncertainty [46] RS theory has been applied in manydomains including classification system analysis patternreorganization and data mining [47] RS-based classificationalgorithms are based on equivalence relations and have beenused as classifiers in medical diagnosis [37 46] In this paperwe primarily focus on the RS reduction algorithm whichcan reduce all redundant features of datasets and seek the
minimum subset of features necessary to attain a satisfactoryclassification [37] A few basic concepts of RS theory aredefined [46 47] as follows
Definition 1 U is a certain set that is referred to as theuniverse R is an equivalence relation in U The pair 119860 =(119880 119877) is referred to as an approximation space
Definition 2 119875 sub 119877 cap119875 (the intersection of all equivalencerelations in P) is an equivalence relation which is referredto as the R-indiscernibility relation and it is represented byInd(119877)
Definition 3 Let X be a certain subset of U The leastcomposed set in R that contains X is referred to as the bestupper approximation ofX in R and represented by119877minus(119883) thegreatest composed set in R contained inX is referred to as thebest lower approximation of X in R and it is represented by119877minus(119883)
119877minus (119883) = 119909 isin 119880 [119909]119877 sub 119883
whereU is the universe that consists of a finite set of n objects119860 = 119862 cup 119863 in which C is a set of condition attributes andD is a set of decision attributes V is the set of domains ofattributes and F is the information function for each 119886 isin 119860119909 isin 119880 119865(119909 119886) isin 119881119886
Definition 5 In an information system C and D are sets ofattributes in119880119883 isin 119880ind(119876) and pos119901(119876) which is referredto as a positive region is defined as
pos119901 (119876) = cup119875minus (119883) (3)
Definition 6 P and Q are sets of attributes in U 119875119876 sube 119860and the dependency 119903119901(119876) is defined as
119903119901 (119876) =card (pos119901 (119876))
card (119880) (4)
Card (X) denotes the cardinality of X 0 le 119903119901(119876) le 1
Definition 7 P andQ are sets of attributes inU 119875119876 sube 119860 andthe significance of 119886119894 is defined as
sig (119886119894) = 119903119901 (119876) minus 119903119901minus119886119894 (119876) (5)
22 ReliefF Algorithm Many feature selection algorithmshave been developed ReliefF is one of the most widely usedand effective algorithms [48] ReliefF is a simple yet efficientprocedure for estimating the quality of features in problemswith dependencies between features [20] The pseudocode ofReliefF algorithm is listed in Algorithm 1
4 Computational and Mathematical Methods in Medicine
ReliefF algorithmInput A decision table 119878 = (119880 119875 119876)Output the vector119882 of estimations of the qualities of features(1) set all weights119882[119860] fl 00(2) for 119894 fl 1 to119898 do begin(3) randomly select a sample 119877119894(4) find 119896 nearest hits119867119895(5) for each class 119862 = class(119877119894) do(6) from class 119862 find 119896 nearest misses119872119895(119862)(7) for 119860 fl 1 to a do(8) 119882[119860] = 119882[119860] minus sum119896119895=1 diff(119860 119877119894 119867119895)119898119896 + sum119862 =class(119877119894)[119875(119862)1 minus 119875(class(119877119894)) sum
31 Overview The proposed hybrid classification systemconsists of two main components (i) feature selection usingthe RFRS subsystem and (ii) data classification using theclassification system A flow chart of the proposed systemis shown in Figure 1 We describe the preprocessing andclassification systems in the following subsections
32 RFRS Feature Selection Subsystem We propose a two-phase feature selection method based on the ReliefF algo-rithm and the RS (RFRS) algorithm The idea is to use thefeature estimation from the ReliefF phase as the input andheuristics for the subsequent RS reduction phase In thefirst phase we adopt the ReliefF algorithm to obtain featureweights and select important features in the second phasethe feature estimation obtained from the first phase is usedto guide the initialization of the parameters required for thegenetic algorithm We use a GA-based search engine to findsatisfactory reducts
The RFRS feature selection subsystem consists of threemain modules (i) data discretization (ii) feature extractionusing the ReliefF algorithm and (iii) feature reduction usingthe heuristic RS reduction algorithm we propose
321 Data Discretization RS reduction requires categoricaldata Consequently data discretization is the first step Weused an approximate equal interval binning method to binthe data variables into a small number of categories
322 Feature Extraction by the ReliefF Algorithm Module 2is used for feature extraction by the ReliefF algorithm To dealwith incomplete data we change the diff function Missingfeature values are treated probabilistically [20] We calculatethe probability that two given instances have different valuesfor a given feature conditioned over the class value [20]When one instance has an unknown value then
diff (119860 1198681 1198682) = 1 minus 119875 (value (119860 1198682) | class (1198681)) (6)
When both instances have unknown values then
diff (119860 1198681 1198682)
= 1
minusvalues(119860)sum119881
(119875 (119881 | class (1198681)) times 119875 (119881 | class (1198682)))
(7)
Conditional probabilities are approximated by relativefrequencies in the training set The process of feature extrac-tion is shown as follows
The Process of Feature Extraction Using ReliefF Algorithm
Input A decision table 119878 = (119880 119875 119876) 119875 = 1198861 1198862 119886119898119876 = 1198891 1198892 119889119899 (119898 ge 1 119899 ge 1)
Output The selected feature subset 119870 = 1198861 1198862 119886119896(1 le119896 le 119898)
Step 1 Obtain the weight matrix of each feature using ReliefFalgorithm119882 = 1199081 1199082 119908119894 119908119898 (1 le 119894 le 119898)
Step 2 Set a threshold 120575
Step 3 If 119908119894 gt 120575 then feature 119886119894 is selected
323 Feature Reduction by the Heuristic RS Reduction Algo-rithm The evaluation result obtained by the ReliefF algo-rithm is the feature rank A higher ranking means that thefeature has stronger distinguishing qualities and a higherweight [30] Consequently in the process of reduct searchingthe features in the front rank should have a higher probabilityof being selected
We proposed the RS reduction algorithmby using the fea-ture estimation as heuristics and a GA-based search engine tosearch for the satisfactory reducts The pseudocode of thealgorithm is provided in Algorithm 2 The algorithm wasimplemented in MATLAB R2014a
33 Classification Subsystem In the classification subsystemthe dataset is split into training sets and corresponding test
Computational and Mathematical Methods in Medicine 5
RFRSfeatureselectionsubsystem
Heart disease dataset Data discretization
Classificationsubsystem
An ensemble classifierCross-validation
ReliefF algorithm
Feature extraction
Feature reduction
Heuristic RS reduction algorithm
Optimal performance pi Optimal reduct rij
S = (UA V f) A = C cup D C = C1 D = D1 m ge 1 n ge 1C2 Cm D2 Dn
K = C1 (1 le k le m)C2 Ck
Reducts R = R1 Ri = ri1 i ge 1 1 le j le kR2 Ri ri2 rij
Training set T = T1 Ti = ti1 D i ge 1 1 le j le kti2 tijT2 Ti
Test set V = V1 Vi = i1 D i ge 1 1 le j le kV2 Vi i2 ij
Performance set P = p1 pi = pi1 i ge 1 1 le j le kpi2 pijp2 pi
Trained set T998400 = T9984001 T998400i = t998400i1 D i ge 1 1 le j le kT9984002 T998400i t998400i2 t
998400ij
Figure 1 Structure of RFRS-based classification system
sets The decision tree is a nonparametric learning algorithmthat does not need to search for optimal parameters in thetraining stage and thus is used as a weak learner for ensemblelearning [49] In this paper the ensemble classifier uses theC45 decision tree as the base classifier We use the boostingtechnique to construct ensemble classifiers Jackknife cross-validation is used to increase the amount of data for testingthe results The optimal reduct is the reduct that obtains thebest classification accuracy
4 Experimental Results
41 Dataset TheStatlog (Heart) dataset used in ourworkwasobtained from the UCI machine learning database [3] Thisdataset contains 270 observations and 2 classes the presence
and absence of heart disease The samples include 13 condi-tion features presented in Table 1 We denote the 13 featuresas C1 to C13
42 Performance Evaluation Methods
421 Confusion Matrix Sensitivity Specificity and AccuracyA confusion matrix [50] contains information about actualand predicted classifications performed by a classificationsystem The performance of such systems is commonlyevaluated using the data in the matrix Table 2 shows theconfusion matrix for a two-class classifier
In the confusion matrix TP is the number of true posi-tives representing the cases with heart disease that are cor-rectly classified into the heart disease class FN is the numberof false negatives representing cases with heart disease that
6 Computational and Mathematical Methods in Medicine
Heuristic RS reduction algorithmInput a decision table 119878 = (119880 119862119863) 119862 = 1198881 1198882 119888119898119863 = 1198891 1198892 119889119899Output RedStep 1 Return Core(1) Corelarr (2) For 119894 = 1 to119898(3) Select 119888119894 from 119862(4) Calculate Ind(119862) Ind(119862 minus 119888119894) and Ind(119863)(5) Calculate pos119862(119863) pos119862minus119888119894 (119863) and 119903119862(119863)(6) Calculate Sig(119886119894) Sig(119886119894) = 119903119862(119863) minus 119903119862minus119888119894 (119863)(7) If sig(119888119894) = 0(8) core = core cup 119888119894(9) End if(10) End forStep 2 Return Red(1) Red = Core(2) 1198621015840 = 119862 minus Red(3) While Sig(Red119863) = Sig(119862119863) do
Compute the weight of each feature 119888 in 1198621015840 using the ReliefF algorithmSelect a feature 119888 according to its weight let Red=Red cup 119888Initialize all the necessary parameters for the GA-based search engine according to theresults of the last step and search for satisfactory reducts
End while
Algorithm 2 Pseudocode of heuristic RS reduction algorithm
are classified into the healthy class TN is the number of truenegatives representing healthy cases that are correctly classi-fied into the healthy class Finally FP is the number of falsepositives representing the healthy cases that are incorrectlyclassified into the heart disease class [50]
The performance of the proposed system was evaluatedbased on sensitivity specificity and accuracy tests which usethe true positive (TP) true negative (TN) false negative (FN)and false positive (FP) terms [33]These criteria are calculatedas follows [41]
Sensitivity (Sn) = TPTP + FN
times 100
Specificity (Sp) = TNFP + TN
times 100
Accuracy (Acc) = TP + TNTP + TN + FP + FN
times 100
(8)
422 Cross-Validation Three cross-validation methodsnamely subsampling tests independent dataset tests andjackknife tests are often employed to evaluate the predictivecapability of a predictor [51] Among the three methodsthe jackknife test is deemed the least arbitrary and the mostobjective and rigorous [52 53] because it always yields aunique outcome as demonstrated by a penetrating analysis ina recent comprehensive review [54 55] Therefore thejackknife test has been widely and increasingly adopted inmany areas [56 57]
Accordingly the jackknife test was employed to examinethe performance of the model proposed in this paper Forjackknife cross-validation each sequence in the training
dataset is in turn singled out as an independent test sampleand all the parameter rules are calculated based on theremaining samples without including the one being treatedas the test sample
423 Receiver Operating Characteristics (ROC) The receiveroperating characteristic (ROC) curve is used for analyzing theprediction performance of a predictor [58] It is usually plot-ted using the true positive rate versus the false positive rate asthe discrimination threshold of classification algorithm isvaried The area under the ROC curve (AUC) is widely usedand relatively accepted in classification studies because itprovides a good summary of a classifierrsquos performance [59]
43 Results and Discussion
431 Results and Analysis on the Statlog (Heart) DatasetFirst we used the equal interval binningmethod to discretizethe original data In the feature extraction module the num-ber of k-nearest neighbors in the ReliefF algorithm was set to10 and the threshold 120575 was set to 002 Table 3 summarizesthe results of the ReliefF algorithm Based on these resultsC5and C6 were removed In Module 3 we obtained 15 reductsusing the heuristic RS reduction algorithm implemented inMATLAB 2014a
Trials were conducted using 70ndash30 training-test par-titions using all the reduced feature sets Jackknife cross-validation was performed on the dataset The number ofdesired base classifiers k was set to 50 100 and 150 Thecalculations were run 10 times and the highest classification
Computational and Mathematical Methods in Medicine 7
Table 1 Feature information of Statlog (Heart) dataset
Feature Code Description Domain Data type Mean Standard deviationAge 1198621 mdash 29ndash77 Real 54 9Sex 1198622 Male female 0 1 Binary mdash mdash
Chest pain type 1198623Angina
asymptomaticabnormal
1 2 3 4 Nominal mdash mdash
Resting blood pressure 1198624 mdash 94ndash200 Real 131344 17862Serum cholesterol in mgdl 1198625 mdash 126ndash564 Real 249659 51686Fasting blood sugar gt 120mgdl 1198626 mdash 0 1 Binary mdash mdash
Resting electrocardiographic results 1198627Norm
abnormal hyper 0 1 2 Nominal mdash mdash
Maximum heart rate achieved 1198628 mdash 71ndash202 Real 149678 231666Exercise-induced angina 1198629 mdash 0 1 Binary mdash mdashOld peak = ST depression induced by exercise relative to rest 11986210 mdash 0ndash62 Real 105 1145Slope of the peak exercise ST segment 11986211 Up flat down 1 2 3 Ordered mdash mdashNumber of major vessels (0ndash3) colored by fluoroscopy 11986212 mdash 0 1 2 3 Real mdash mdash
Thal 11986213Normal fixed
defectreversible defect
3 6 7 Nominal mdash mdash
Table 2 The confusion matrix
Predicted patientswith heart disease
Predicted healthypersons
Actual patients withheart disease True positive (TP) False negative (FN)
Actual healthypersons False positive (FP) True negative (TN)
performances for each training-test partition are provided inTable 4
In Table 4 1198772 obtains the best test set classificationaccuracy (9259) using the ensemble classifiers when 119896 =100 The training process is shown in Figure 2 The trainingand test ROC curves are shown in Figure 3
432 Comparison with Other Classifiers In this sectionour ensemble classification method is compared with theindividual C45 decision tree and Naıve Bayes and BayesianNeural Networks (BNN)methodsTheC45 decision tree andNaıve Bayes are common classifiers Bayesian Neural Net-works (BNN) is a classifier that uses Bayesian regularizationto train feed-forward neural networks [60] and has betterperformance than pure neural networks The classificationaccuracy results of the four classifiers are listed in Table 5Theensemble classification method has better performance thanthe individual C45 classifier and the other two classifiers
433 Comparison of the Results with Other Studies Wecompared our results with the results of other studies Table 6shows the classification accuracies of our study and previousmethods
0 10 20 30 40 50 60 70 80 90 100008
01
012
014
016
018
02
022
024
Number of trees
Test
class
ifica
tion
erro
r
Figure 2 Training process of 1198777
The results show that our proposed method obtainssuperior and promising results in classifying heart diseasepatients We believe that the proposed RFRS-based classi-fication system can be exceedingly beneficial in assistingphysicians in making accurate decisions
8 Computational and Mathematical Methods in Medicine
In this paper a novel ReliefF and Rough Set- (RFRS-)based classification system is proposed for heart diseasediagnosisThemain novelty of this paper lies in the proposedapproach the combination of the ReliefF and RS methodsto classify heart disease problems in an efficient and fastmanner The RFRS classification system consists of twosubsystems the RFRS feature selection subsystem and theclassification subsystemThe Statlog (Heart) dataset from theUCI machine learning database [3] was selected to test thesystemThe experimental results show that the reduct1198772 (11986211198623 1198627 1198628 11986211 11986212 11986213) achieves the highest classificationaccuracy (9259) using an ensemble classifier with the C45decision tree as the weak learner The results also show thatthe RFRS method has superior performance compared to
three common classifiers in terms of ACC sensitivity andspecificity In addition the performance of the proposedsystem is superior to that of existingmethods in the literatureBased on empirical analysis the results indicate that theproposed classification system can be used as a promisingalternative tool in medical decision making for heart diseasediagnosis
However the proposed method also has some weak-nesses The number of the nearest neighbors (119896) and theweight threshold (120579) are not stable in the ReliefF algorithm[20] One solution to this problem is to compute estimatesfor all possible numbers and take the highest estimate of eachfeature as the final result [20] We need to perform moreexperiments to find the optimal parameter values for theReliefF algorithm in the future
Computational and Mathematical Methods in Medicine 9
Table 6 Comparison of our results with those of other studies
Author Method Classification accuracy ()Our study RFRS classification system 9259Lee [4] Graphical characteristics of BSWFM combined with Euclidean distance 874Tomar and Agarwal [5] Feature selection-based LSTSVM 8559Buscema et al [6] TWIST algorithm 8414Subbulakshmi et al [7] ELM 875Karegowda et al [8] GA + Naıve Bayes 8587Srinivas et al [9] Naıve Bayes 8370Polat and Gunes [10] RBF kernel 119865-score + LS-SVM 8370Ozsen and Gunes [11] GA-AWAIS 8743Helmy and Rasheed [12] Algebraic Sigmoid 8524Wang et al [13] Linear kernel SVM classifiers 8337Ozsen and Gunes [14] Hybrid similarity measure 8395Kahramanli and Allahverdi [15] Hybrid neural network method 868Yan et al [16] ICA + SVM 8375Sahan et al [17] AWAIS 8259Duch et al [18] KNN classifier 856BSWFM bounded sumof weighted fuzzymembership functions LSTSVM Least Square Twin Support VectorMachine TWIST Training with Input Selectionand Testing ELM Extreme LearningMachine GA genetic algorithm SVM support vectormachine ICA imperialist competitive algorithm AWAIS attributeweighted artificial immune system119870NN 119896-nearest neighbor
Test ROC
0 10
01
02
02
03
04
04
05
06
06
07
08
08
09
1
False positive rate
True
pos
itive
rate
0 10
01
02
02
03
04
04
05
06
06
07
08
08
09
1
False positive rate
True
pos
itive
rate
Training ROC
Figure 3 ROC curves for training and test sets
Competing Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
This study was supported by the National Natural ScienceFoundation of China (Grant no 71432007)
References
[1] K D Kochanek J Xu S L Murphy A M Minino and H-C Kung ldquoDeaths final data for 2009rdquo National Vital StatisticsReports vol 60 no 3 pp 1ndash116 2011
[2] H Temurtas N Yumusak and F Temurtas ldquoA comparativestudy on diabetes disease diagnosis using neural networksrdquoExpert Systems with Applications vol 36 no 4 pp 8610ndash86152009
10 Computational and Mathematical Methods in Medicine
[3] UCI Repository of Machine Learning Databases httparchiveicsuciedumldatasetsStatlog+28Heart29
[4] S-H Lee ldquoFeature selection based on the center of gravity ofBSWFMs using NEWFMrdquo Engineering Applications of ArtificialIntelligence vol 45 pp 482ndash487 2015
[5] D Tomar and S Agarwal ldquoFeature selection based least squaretwin support vector machine for diagnosis of heart diseaserdquoInternational Journal of Bio-Science and Bio-Technology vol 6no 2 pp 69ndash82 2014
[6] M Buscema M Breda and W Lodwick ldquoTraining with InputSelection and Testing (TWIST) algorithm a significant advancein pattern recognition performance of machine learningrdquoJournal of Intelligent Learning Systems and Applications vol 5no 1 pp 29ndash38 2013
[7] C V Subbulakshmi S N Deepa and N Malathi ldquoExtremelearning machine for two category data classificationrdquo inProceedings of the IEEE International Conference on AdvancedCommunication Control and Computing Technologies (ICAC-CCT rsquo12) pp 458ndash461 Ramanathapuram India August 2012
[8] A G Karegowda A SManjunath andM A Jayaram ldquoFeaturesubset selection problem using wrapper approach in supervisedlearningrdquo International Journal of Computer Applications vol 1no 7 pp 13ndash17 2010
[9] K Srinivas B K Rani and A Govrdhan ldquoApplications ofdata mining techniques in healthcare and prediction of heartattacksrdquo International Journal on Computer Science and Engi-neering vol 2 pp 250ndash255 2010
[10] K Polat and S Gunes ldquoA new feature selection methodon classification of medical datasets kernel F-score featureselectionrdquo Expert Systems with Applications vol 36 no 7 pp10367ndash10373 2009
[11] S Ozsen and S Gunes ldquoAttribute weighting via genetic algo-rithms for attributeweighted artificial immune system (AWAIS)and its application to heart disease and liver disorders prob-lemsrdquo Expert Systems with Applications vol 36 no 1 pp 386ndash392 2009
[12] T Helmy and Z Rasheed ldquoMulti-category bioinformaticsdataset classification using extreme learning machinerdquo in Pro-ceedings of the IEEE Congress on Evolutionary Computation(CEC rsquo09) pp 3234ndash3240 Trondheim Norway May 2009
[13] S-J Wang A Mathew Y Chen L-F Xi L Ma and JLee ldquoEmpirical analysis of support vector machine ensembleclassifiersrdquo Expert Systems with Applications vol 36 no 3 pp6466ndash6476 2009
[14] S Ozsen and S Gunes ldquoEffect of feature-type in selectingdistance measure for an artificial immune system as a patternrecognizerrdquoDigital Signal Processing vol 18 no 4 pp 635ndash6452008
[15] H Kahramanli and N Allahverdi ldquoDesign of a hybrid systemfor the diabetes and heart diseasesrdquo Expert Systems withApplications vol 35 no 1-2 pp 82ndash89 2008
[16] G Yan G Ma J Lv and B Song ldquoCombining independentcomponent analysis with support vector machinesrdquo in Pro-ceedings of the in 1st International Symposium on Systems andControl in Aerospace and Astronautics (ISSCAA rsquo06) pp 493ndash496 Harbin China January 2006
[17] S Sahan K Polat H Kodaz and S Gunes ldquoThe medicalapplications of attribute weighted artificial immune system(AWAIS) diagnosis of heart and diabetes diseasesrdquo ArtificialImmune Systems vol 3627 pp 456ndash468 2005
[18] W Duch R Adamczak and K Grabczewski ldquoA new method-ology of extraction optimization and application of crisp and
fuzzy logical rulesrdquo IEEE Transactions on Neural Networks vol12 no 2 pp 277ndash306 2001
[19] M P McRae B Bozkurt C M Ballantyne et al ldquoCardiacScoreCard a diagnostic multivariate index assay system forpredicting a spectrumof cardiovascular diseaserdquoExpert Systemswith Applications vol 54 pp 136ndash147 2016
[20] M Robnik-Sikonja and I Kononenko ldquoTheoretical and empir-ical analysis of ReliefF and RReliefFrdquoMachine Learning vol 53no 1-2 pp 23ndash69 2003
[21] D Wettschereck D W Aha and T Mohri ldquoA review andempirical evaluation of feature weighting methods for a classof lazy learning algorithmsrdquo Artificial Intelligence Review vol11 no 1ndash5 pp 273ndash314 1997
[22] I Kononenko E Simec and M R Sikonja ldquoOvercomingthe myopia of inductive learning algorithms with RELIEFFrdquoApplied Intelligence vol 7 no 1 pp 39ndash55 1997
[23] K Kira and L Rendell ldquoA practical approach to featureselectionrdquo in Proceedings of the International Conference onMachine Learning pp 249ndash256 Morgan Kaufmann AberdeenScotland 1992
[24] I Kononenko ldquoEstimating attributes analysis and extension ofReliefFrdquo in Machine Learning ECML-94 European ConferenceonMachine Learning Catania Italy April 6ndash8 1994 Proceedingsvol 784 of Lecture Notes in Computer Science pp 171ndash182Springer Berlin Germany 1994
[25] R Ruiz J C Riquelme and J S Aguilar-Ruiz ldquoHeuristicsearch over a ranking for feature selectionrdquo in Proceedings of8th InternationalWork-Conference onArtificial NeuralNetworks(IWANN rsquo05) Barcelona Spain June 2005 vol 3512 of LecturesNotes in Computer Science pp 742ndash749 Springer Berlin Ger-many 2005
[26] N Spolaor E A ChermanM CMonard andHD Lee ldquoFilterapproach feature selection methods to support multi-labellearning based on ReliefF and information gainrdquo in Advancesin Artificial IntelligencemdashSBIA 2012 21th Brazilian Symposiumon Artificial Intelligence Curitiba Brazil October 20ndash25 2012Proceedings vol 7589 of Lectures Notes in Computer Science pp72ndash81 Springer Berlin Germany 2012
[27] R Ruiz J C Riquelme and J S Aguilar-Ruiz ldquoFast featureranking algorithmrdquo in Proceedings of the Knowledge-BasedIntelligent Information and Engineering Systems (KES rsquo03) pp325ndash331 Springer Berlin Germany 2003
[28] V Jovanoski and N Lavrac ldquoFeature subset selection inassociation rules learning systemsrdquo in Proceedings of AnalysisWarehousing and Mining the Data pp 74ndash77 1999
[29] J J Liu and J T Y Kwok ldquoAn extended genetic rule inductionalgorithmrdquo in Proceedings of the Congress on EvolutionaryComputation pp 458ndash463 LA Jolla Calif USA July 2000
[30] L-X Zhang J-X Wang Y-N Zhao and Z-H Yang ldquoA novelhybrid feature selection algorithm using ReliefF estimationfor GA-Wrapper searchrdquo in Proceedings of the InternationalConference onMachine Learning andCybernetics vol 1 pp 380ndash384 IEEE Xirsquoan China November 2003
[31] Z Zhao L Wang and H Liu ldquoEfficient spectral featureselectionwithminimum redundancyrdquo in Proceedings of the 24thAAAI Conference on Artificial Intelligence AAAI Atlanta GaUSA 2010
[32] F Nie H Huang X Cai and C H Ding ldquoEfficient androbust feature selection via joint ℓ21-norms minimizationrdquo inAdvances in Neural Information Processing Systems pp 1813ndash1821 MIT Press 2010
Computational and Mathematical Methods in Medicine 11
[33] S-Y Jiang and L-X Wang ldquoEfficient feature selection based oncorrelation measure between continuous and discrete featuresrdquoInformation Processing Letters vol 116 no 2 pp 203ndash215 2016
[34] M J Kearns and U V Vazirani An Introduction to Compu-tational Learning Theory MIT Press Cambridge Mass USA1994
[35] R Bellman Adaptive Control Processes A Guided Tour RANDCorporation Research Studies Princeton University PressPrinceton NJ USA 1961
[36] Z Pawlak ldquoRough set theory and its applications to dataanalysisrdquo Cybernetics and Systems vol 29 no 7 pp 661ndash6881998
[37] A-E Hassanien ldquoRough set approach for attribute reductionand rule generation a case of patients with suspected breastcancerrdquo Journal of the American Society for Information Scienceand Technology vol 55 no 11 pp 954ndash962 2004
[38] A Skowron and C Rauszer ldquoThe discernibility matricesand functions in information systemsrdquo in Intelligent DecisionSupport-Handbook of Applications and Advances of the RoughSetsTheory SystemTheory Knowledge Engineering and ProblemSolving R Słowinski Ed vol 11 pp 331ndash362 KluwerAcademicDordrecht Netherlands 1992
[39] Z Pawlak ldquoRough set approach to knowledge-based decisionsupportrdquo European Journal of Operational Research vol 99 no1 pp 48ndash57 1997
[40] X Wang J Yang X Teng W Xia and R Jensen ldquoFeatureselection based on rough sets and particle swarm optimizationrdquoPattern Recognition Letters vol 28 no 4 pp 459ndash471 2007
[41] C Ma J Ouyang H-L Chen and X-H Zhao ldquoAn efficientdiagnosis system for Parkinsonrsquos disease using kernel-basedextreme learning machine with subtractive clustering featuresweighting approachrdquo Computational and Mathematical Meth-ods in Medicine vol 2014 Article ID 985789 14 pages 2014
[42] A H El-Baz ldquoHybrid intelligent system-based rough set andensemble classifier for breast cancer diagnosisrdquoNeural Comput-ing and Applications vol 26 no 2 pp 437ndash446 2015
[43] J-H Eom S-CKim andB-T Zhang ldquoAptaCDSS-E a classifierensemble-based clinical decision support system for cardiovas-cular disease level predictionrdquo Expert Systems with Applicationsvol 34 no 4 pp 2465ndash2479 2008
[44] Y Ma Ensemble Machine Learning Methods and ApplicationsSpringer New York NY USA 2012
[45] S A Etemad and A Arya ldquoClassification and translation ofstyle and affect in human motion using RBF neural networksrdquoNeurocomputing vol 129 pp 585ndash595 2014
[46] Z Pawlak ldquoRough setsrdquo International Journal of Computer ampInformation Sciences vol 11 no 5 pp 341ndash356 1982
[47] Z Pawlak ldquoRough sets and intelligent data analysisrdquo Informa-tion Sciences vol 147 no 1ndash4 pp 1ndash12 2002
[48] Y Huang P J McCullagh and N D Black ldquoAn optimization ofReliefF for classification in large datasetsrdquoData and KnowledgeEngineering vol 68 no 11 pp 1348ndash1356 2009
[49] J Sun M-Y Jia and H Li ldquoAdaBoost ensemble for financialdistress prediction an empirical comparison with data fromChinese listed companiesrdquo Expert Systems with Applicationsvol 38 no 8 pp 9305ndash9312 2011
[50] R Kohavi and F Provost ldquoGlossary of termsrdquo Machine Learn-ing vol 30 no 2-3 pp 271ndash274 1998
[51] W Chen and H Lin ldquoPrediction of midbody centrosome andkinetochore proteins based on gene ontology informationrdquoBiochemical and Biophysical Research Communications vol 401no 3 pp 382ndash384 2010
[52] K-C Chou and C-T Zhang ldquoPrediction of protein structuralclassesrdquo Critical Reviews in Biochemistry andMolecular Biologyvol 30 no 4 pp 275ndash349 1995
[53] W Chen and H Lin ldquoIdentification of voltage-gated potassiumchannel subfamilies from sequence information using supportvectormachinerdquoComputers in Biology andMedicine vol 42 no4 pp 504ndash507 2012
[54] K-C Chou and H-B Shen ldquoRecent progress in proteinsubcellular location predictionrdquo Analytical Biochemistry vol370 no 1 pp 1ndash16 2007
[55] P Feng H Lin W Chen and Y Zuo ldquoPredicting the typesof J-proteins using clustered amino acidsrdquo BioMed ResearchInternational vol 2014 Article ID 935719 8 pages 2014
[56] K-C Chou and H-B Shen ldquoProtIdent a web server foridentifying proteases and their types by fusing functionaldomain and sequential evolution informationrdquoBiochemical andBiophysical Research Communications vol 376 no 2 pp 321ndash325 2008
[57] K-C Chou and Y-D Cai ldquoPrediction of membrane proteintypes by incorporating amphipathic effectsrdquo Journal of ChemicalInformation and Modeling vol 45 no 2 pp 407ndash413 2005
[58] F Tom ldquoROC graphs notes and practical considerations forresearchersrdquo Machine Learning vol 31 pp 1ndash38 2004
[59] JHuang andCX Ling ldquoUsingAUCand accuracy in evaluatinglearning algorithmsrdquo IEEE Transactions on Knowledge andDataEngineering vol 17 no 3 pp 299ndash310 2005
[60] F D Foresee and M T Hagan ldquoGauss-Newton approximationto Bayesian learningrdquo in Proceedings of the International Con-ference on Neural Networks vol 3 pp 1930ndash1935 Houston TexUSA 1997
Computational and Mathematical Methods in Medicine 3
features The system contains two subsystems the RFRSfeature selection subsystem and a classification subsystemIn the RFRS feature selection subsystem we use a two-stage hybrid modeling procedure by integrating ReliefF withthe RS (RFRS) method First the proposed method adoptsthe ReliefF algorithm to obtain feature weights and selectmore relevant and important features from heart diseasedatasets Then the feature estimation obtained from the firstphase is used as the input for the RS reduction algorithmand guide the initialization of the necessary parameters forthe genetic algorithm We use a GA-based search engine tofind satisfactory reducts In the classification subsystem theresulting reducts serve as the input for the chosen classifiersFinally the optimal reduct and performance can be obtained
To evaluate the performance of the proposed hybridmethod a confusion matrix sensitivity specificity accuracyand ROC were used The experimental results show that theproposed method achieves very promising results using thejack knife test
The main contributions of this paper are summarized asfollows
(i) We propose a feature selection system to integrate theReliefF approach with the RS method (RFRS) to detect heartdisease in an efficient and effective way The idea is to use thefeature estimation from the ReliefF phase as the input andheuristics for the RS reduction phase
(ii) In the classification system we propose an ensembleclassifier using C45 as the base classifier Ensemble learningcan achieve better performance at the cost of computationthan single classifiers The experimental results show that theensemble classifier in this paper is superior to three commonclassifiers
(iii) Compared with three classifiers and previous studiesthe proposed diagnostic system achieved excellent classifi-cation results On the Statlog (Heart) dataset from the UCImachine learning database [3] the resulting classificationaccuracy was 9259 which is higher than that achieved byother studies
The rest of the paper is organized as follows Section 2offers brief background information concerning the ReliefFalgorithm and RS theory The details of the diagnosis sys-tem implementation are presented in Section 3 Section 4describes the experimental results and discusses the pro-posed method Finally conclusions and recommendationsfor future work are summarized in Section 5
2 Theoretical Background
21 Basic Concepts of Rough Set Theory Rough Set (RS)theory which was proposed by Pawlak in the early 1980sis a new mathematical approach to addressing vaguenessand uncertainty [46] RS theory has been applied in manydomains including classification system analysis patternreorganization and data mining [47] RS-based classificationalgorithms are based on equivalence relations and have beenused as classifiers in medical diagnosis [37 46] In this paperwe primarily focus on the RS reduction algorithm whichcan reduce all redundant features of datasets and seek the
minimum subset of features necessary to attain a satisfactoryclassification [37] A few basic concepts of RS theory aredefined [46 47] as follows
Definition 1 U is a certain set that is referred to as theuniverse R is an equivalence relation in U The pair 119860 =(119880 119877) is referred to as an approximation space
Definition 2 119875 sub 119877 cap119875 (the intersection of all equivalencerelations in P) is an equivalence relation which is referredto as the R-indiscernibility relation and it is represented byInd(119877)
Definition 3 Let X be a certain subset of U The leastcomposed set in R that contains X is referred to as the bestupper approximation ofX in R and represented by119877minus(119883) thegreatest composed set in R contained inX is referred to as thebest lower approximation of X in R and it is represented by119877minus(119883)
119877minus (119883) = 119909 isin 119880 [119909]119877 sub 119883
whereU is the universe that consists of a finite set of n objects119860 = 119862 cup 119863 in which C is a set of condition attributes andD is a set of decision attributes V is the set of domains ofattributes and F is the information function for each 119886 isin 119860119909 isin 119880 119865(119909 119886) isin 119881119886
Definition 5 In an information system C and D are sets ofattributes in119880119883 isin 119880ind(119876) and pos119901(119876) which is referredto as a positive region is defined as
pos119901 (119876) = cup119875minus (119883) (3)
Definition 6 P and Q are sets of attributes in U 119875119876 sube 119860and the dependency 119903119901(119876) is defined as
119903119901 (119876) =card (pos119901 (119876))
card (119880) (4)
Card (X) denotes the cardinality of X 0 le 119903119901(119876) le 1
Definition 7 P andQ are sets of attributes inU 119875119876 sube 119860 andthe significance of 119886119894 is defined as
sig (119886119894) = 119903119901 (119876) minus 119903119901minus119886119894 (119876) (5)
22 ReliefF Algorithm Many feature selection algorithmshave been developed ReliefF is one of the most widely usedand effective algorithms [48] ReliefF is a simple yet efficientprocedure for estimating the quality of features in problemswith dependencies between features [20] The pseudocode ofReliefF algorithm is listed in Algorithm 1
4 Computational and Mathematical Methods in Medicine
ReliefF algorithmInput A decision table 119878 = (119880 119875 119876)Output the vector119882 of estimations of the qualities of features(1) set all weights119882[119860] fl 00(2) for 119894 fl 1 to119898 do begin(3) randomly select a sample 119877119894(4) find 119896 nearest hits119867119895(5) for each class 119862 = class(119877119894) do(6) from class 119862 find 119896 nearest misses119872119895(119862)(7) for 119860 fl 1 to a do(8) 119882[119860] = 119882[119860] minus sum119896119895=1 diff(119860 119877119894 119867119895)119898119896 + sum119862 =class(119877119894)[119875(119862)1 minus 119875(class(119877119894)) sum
31 Overview The proposed hybrid classification systemconsists of two main components (i) feature selection usingthe RFRS subsystem and (ii) data classification using theclassification system A flow chart of the proposed systemis shown in Figure 1 We describe the preprocessing andclassification systems in the following subsections
32 RFRS Feature Selection Subsystem We propose a two-phase feature selection method based on the ReliefF algo-rithm and the RS (RFRS) algorithm The idea is to use thefeature estimation from the ReliefF phase as the input andheuristics for the subsequent RS reduction phase In thefirst phase we adopt the ReliefF algorithm to obtain featureweights and select important features in the second phasethe feature estimation obtained from the first phase is usedto guide the initialization of the parameters required for thegenetic algorithm We use a GA-based search engine to findsatisfactory reducts
The RFRS feature selection subsystem consists of threemain modules (i) data discretization (ii) feature extractionusing the ReliefF algorithm and (iii) feature reduction usingthe heuristic RS reduction algorithm we propose
321 Data Discretization RS reduction requires categoricaldata Consequently data discretization is the first step Weused an approximate equal interval binning method to binthe data variables into a small number of categories
322 Feature Extraction by the ReliefF Algorithm Module 2is used for feature extraction by the ReliefF algorithm To dealwith incomplete data we change the diff function Missingfeature values are treated probabilistically [20] We calculatethe probability that two given instances have different valuesfor a given feature conditioned over the class value [20]When one instance has an unknown value then
diff (119860 1198681 1198682) = 1 minus 119875 (value (119860 1198682) | class (1198681)) (6)
When both instances have unknown values then
diff (119860 1198681 1198682)
= 1
minusvalues(119860)sum119881
(119875 (119881 | class (1198681)) times 119875 (119881 | class (1198682)))
(7)
Conditional probabilities are approximated by relativefrequencies in the training set The process of feature extrac-tion is shown as follows
The Process of Feature Extraction Using ReliefF Algorithm
Input A decision table 119878 = (119880 119875 119876) 119875 = 1198861 1198862 119886119898119876 = 1198891 1198892 119889119899 (119898 ge 1 119899 ge 1)
Output The selected feature subset 119870 = 1198861 1198862 119886119896(1 le119896 le 119898)
Step 1 Obtain the weight matrix of each feature using ReliefFalgorithm119882 = 1199081 1199082 119908119894 119908119898 (1 le 119894 le 119898)
Step 2 Set a threshold 120575
Step 3 If 119908119894 gt 120575 then feature 119886119894 is selected
323 Feature Reduction by the Heuristic RS Reduction Algo-rithm The evaluation result obtained by the ReliefF algo-rithm is the feature rank A higher ranking means that thefeature has stronger distinguishing qualities and a higherweight [30] Consequently in the process of reduct searchingthe features in the front rank should have a higher probabilityof being selected
We proposed the RS reduction algorithmby using the fea-ture estimation as heuristics and a GA-based search engine tosearch for the satisfactory reducts The pseudocode of thealgorithm is provided in Algorithm 2 The algorithm wasimplemented in MATLAB R2014a
33 Classification Subsystem In the classification subsystemthe dataset is split into training sets and corresponding test
Computational and Mathematical Methods in Medicine 5
RFRSfeatureselectionsubsystem
Heart disease dataset Data discretization
Classificationsubsystem
An ensemble classifierCross-validation
ReliefF algorithm
Feature extraction
Feature reduction
Heuristic RS reduction algorithm
Optimal performance pi Optimal reduct rij
S = (UA V f) A = C cup D C = C1 D = D1 m ge 1 n ge 1C2 Cm D2 Dn
K = C1 (1 le k le m)C2 Ck
Reducts R = R1 Ri = ri1 i ge 1 1 le j le kR2 Ri ri2 rij
Training set T = T1 Ti = ti1 D i ge 1 1 le j le kti2 tijT2 Ti
Test set V = V1 Vi = i1 D i ge 1 1 le j le kV2 Vi i2 ij
Performance set P = p1 pi = pi1 i ge 1 1 le j le kpi2 pijp2 pi
Trained set T998400 = T9984001 T998400i = t998400i1 D i ge 1 1 le j le kT9984002 T998400i t998400i2 t
998400ij
Figure 1 Structure of RFRS-based classification system
sets The decision tree is a nonparametric learning algorithmthat does not need to search for optimal parameters in thetraining stage and thus is used as a weak learner for ensemblelearning [49] In this paper the ensemble classifier uses theC45 decision tree as the base classifier We use the boostingtechnique to construct ensemble classifiers Jackknife cross-validation is used to increase the amount of data for testingthe results The optimal reduct is the reduct that obtains thebest classification accuracy
4 Experimental Results
41 Dataset TheStatlog (Heart) dataset used in ourworkwasobtained from the UCI machine learning database [3] Thisdataset contains 270 observations and 2 classes the presence
and absence of heart disease The samples include 13 condi-tion features presented in Table 1 We denote the 13 featuresas C1 to C13
42 Performance Evaluation Methods
421 Confusion Matrix Sensitivity Specificity and AccuracyA confusion matrix [50] contains information about actualand predicted classifications performed by a classificationsystem The performance of such systems is commonlyevaluated using the data in the matrix Table 2 shows theconfusion matrix for a two-class classifier
In the confusion matrix TP is the number of true posi-tives representing the cases with heart disease that are cor-rectly classified into the heart disease class FN is the numberof false negatives representing cases with heart disease that
6 Computational and Mathematical Methods in Medicine
Heuristic RS reduction algorithmInput a decision table 119878 = (119880 119862119863) 119862 = 1198881 1198882 119888119898119863 = 1198891 1198892 119889119899Output RedStep 1 Return Core(1) Corelarr (2) For 119894 = 1 to119898(3) Select 119888119894 from 119862(4) Calculate Ind(119862) Ind(119862 minus 119888119894) and Ind(119863)(5) Calculate pos119862(119863) pos119862minus119888119894 (119863) and 119903119862(119863)(6) Calculate Sig(119886119894) Sig(119886119894) = 119903119862(119863) minus 119903119862minus119888119894 (119863)(7) If sig(119888119894) = 0(8) core = core cup 119888119894(9) End if(10) End forStep 2 Return Red(1) Red = Core(2) 1198621015840 = 119862 minus Red(3) While Sig(Red119863) = Sig(119862119863) do
Compute the weight of each feature 119888 in 1198621015840 using the ReliefF algorithmSelect a feature 119888 according to its weight let Red=Red cup 119888Initialize all the necessary parameters for the GA-based search engine according to theresults of the last step and search for satisfactory reducts
End while
Algorithm 2 Pseudocode of heuristic RS reduction algorithm
are classified into the healthy class TN is the number of truenegatives representing healthy cases that are correctly classi-fied into the healthy class Finally FP is the number of falsepositives representing the healthy cases that are incorrectlyclassified into the heart disease class [50]
The performance of the proposed system was evaluatedbased on sensitivity specificity and accuracy tests which usethe true positive (TP) true negative (TN) false negative (FN)and false positive (FP) terms [33]These criteria are calculatedas follows [41]
Sensitivity (Sn) = TPTP + FN
times 100
Specificity (Sp) = TNFP + TN
times 100
Accuracy (Acc) = TP + TNTP + TN + FP + FN
times 100
(8)
422 Cross-Validation Three cross-validation methodsnamely subsampling tests independent dataset tests andjackknife tests are often employed to evaluate the predictivecapability of a predictor [51] Among the three methodsthe jackknife test is deemed the least arbitrary and the mostobjective and rigorous [52 53] because it always yields aunique outcome as demonstrated by a penetrating analysis ina recent comprehensive review [54 55] Therefore thejackknife test has been widely and increasingly adopted inmany areas [56 57]
Accordingly the jackknife test was employed to examinethe performance of the model proposed in this paper Forjackknife cross-validation each sequence in the training
dataset is in turn singled out as an independent test sampleand all the parameter rules are calculated based on theremaining samples without including the one being treatedas the test sample
423 Receiver Operating Characteristics (ROC) The receiveroperating characteristic (ROC) curve is used for analyzing theprediction performance of a predictor [58] It is usually plot-ted using the true positive rate versus the false positive rate asthe discrimination threshold of classification algorithm isvaried The area under the ROC curve (AUC) is widely usedand relatively accepted in classification studies because itprovides a good summary of a classifierrsquos performance [59]
43 Results and Discussion
431 Results and Analysis on the Statlog (Heart) DatasetFirst we used the equal interval binningmethod to discretizethe original data In the feature extraction module the num-ber of k-nearest neighbors in the ReliefF algorithm was set to10 and the threshold 120575 was set to 002 Table 3 summarizesthe results of the ReliefF algorithm Based on these resultsC5and C6 were removed In Module 3 we obtained 15 reductsusing the heuristic RS reduction algorithm implemented inMATLAB 2014a
Trials were conducted using 70ndash30 training-test par-titions using all the reduced feature sets Jackknife cross-validation was performed on the dataset The number ofdesired base classifiers k was set to 50 100 and 150 Thecalculations were run 10 times and the highest classification
Computational and Mathematical Methods in Medicine 7
Table 1 Feature information of Statlog (Heart) dataset
Feature Code Description Domain Data type Mean Standard deviationAge 1198621 mdash 29ndash77 Real 54 9Sex 1198622 Male female 0 1 Binary mdash mdash
Chest pain type 1198623Angina
asymptomaticabnormal
1 2 3 4 Nominal mdash mdash
Resting blood pressure 1198624 mdash 94ndash200 Real 131344 17862Serum cholesterol in mgdl 1198625 mdash 126ndash564 Real 249659 51686Fasting blood sugar gt 120mgdl 1198626 mdash 0 1 Binary mdash mdash
Resting electrocardiographic results 1198627Norm
abnormal hyper 0 1 2 Nominal mdash mdash
Maximum heart rate achieved 1198628 mdash 71ndash202 Real 149678 231666Exercise-induced angina 1198629 mdash 0 1 Binary mdash mdashOld peak = ST depression induced by exercise relative to rest 11986210 mdash 0ndash62 Real 105 1145Slope of the peak exercise ST segment 11986211 Up flat down 1 2 3 Ordered mdash mdashNumber of major vessels (0ndash3) colored by fluoroscopy 11986212 mdash 0 1 2 3 Real mdash mdash
Thal 11986213Normal fixed
defectreversible defect
3 6 7 Nominal mdash mdash
Table 2 The confusion matrix
Predicted patientswith heart disease
Predicted healthypersons
Actual patients withheart disease True positive (TP) False negative (FN)
Actual healthypersons False positive (FP) True negative (TN)
performances for each training-test partition are provided inTable 4
In Table 4 1198772 obtains the best test set classificationaccuracy (9259) using the ensemble classifiers when 119896 =100 The training process is shown in Figure 2 The trainingand test ROC curves are shown in Figure 3
432 Comparison with Other Classifiers In this sectionour ensemble classification method is compared with theindividual C45 decision tree and Naıve Bayes and BayesianNeural Networks (BNN)methodsTheC45 decision tree andNaıve Bayes are common classifiers Bayesian Neural Net-works (BNN) is a classifier that uses Bayesian regularizationto train feed-forward neural networks [60] and has betterperformance than pure neural networks The classificationaccuracy results of the four classifiers are listed in Table 5Theensemble classification method has better performance thanthe individual C45 classifier and the other two classifiers
433 Comparison of the Results with Other Studies Wecompared our results with the results of other studies Table 6shows the classification accuracies of our study and previousmethods
0 10 20 30 40 50 60 70 80 90 100008
01
012
014
016
018
02
022
024
Number of trees
Test
class
ifica
tion
erro
r
Figure 2 Training process of 1198777
The results show that our proposed method obtainssuperior and promising results in classifying heart diseasepatients We believe that the proposed RFRS-based classi-fication system can be exceedingly beneficial in assistingphysicians in making accurate decisions
8 Computational and Mathematical Methods in Medicine
In this paper a novel ReliefF and Rough Set- (RFRS-)based classification system is proposed for heart diseasediagnosisThemain novelty of this paper lies in the proposedapproach the combination of the ReliefF and RS methodsto classify heart disease problems in an efficient and fastmanner The RFRS classification system consists of twosubsystems the RFRS feature selection subsystem and theclassification subsystemThe Statlog (Heart) dataset from theUCI machine learning database [3] was selected to test thesystemThe experimental results show that the reduct1198772 (11986211198623 1198627 1198628 11986211 11986212 11986213) achieves the highest classificationaccuracy (9259) using an ensemble classifier with the C45decision tree as the weak learner The results also show thatthe RFRS method has superior performance compared to
three common classifiers in terms of ACC sensitivity andspecificity In addition the performance of the proposedsystem is superior to that of existingmethods in the literatureBased on empirical analysis the results indicate that theproposed classification system can be used as a promisingalternative tool in medical decision making for heart diseasediagnosis
However the proposed method also has some weak-nesses The number of the nearest neighbors (119896) and theweight threshold (120579) are not stable in the ReliefF algorithm[20] One solution to this problem is to compute estimatesfor all possible numbers and take the highest estimate of eachfeature as the final result [20] We need to perform moreexperiments to find the optimal parameter values for theReliefF algorithm in the future
Computational and Mathematical Methods in Medicine 9
Table 6 Comparison of our results with those of other studies
Author Method Classification accuracy ()Our study RFRS classification system 9259Lee [4] Graphical characteristics of BSWFM combined with Euclidean distance 874Tomar and Agarwal [5] Feature selection-based LSTSVM 8559Buscema et al [6] TWIST algorithm 8414Subbulakshmi et al [7] ELM 875Karegowda et al [8] GA + Naıve Bayes 8587Srinivas et al [9] Naıve Bayes 8370Polat and Gunes [10] RBF kernel 119865-score + LS-SVM 8370Ozsen and Gunes [11] GA-AWAIS 8743Helmy and Rasheed [12] Algebraic Sigmoid 8524Wang et al [13] Linear kernel SVM classifiers 8337Ozsen and Gunes [14] Hybrid similarity measure 8395Kahramanli and Allahverdi [15] Hybrid neural network method 868Yan et al [16] ICA + SVM 8375Sahan et al [17] AWAIS 8259Duch et al [18] KNN classifier 856BSWFM bounded sumof weighted fuzzymembership functions LSTSVM Least Square Twin Support VectorMachine TWIST Training with Input Selectionand Testing ELM Extreme LearningMachine GA genetic algorithm SVM support vectormachine ICA imperialist competitive algorithm AWAIS attributeweighted artificial immune system119870NN 119896-nearest neighbor
Test ROC
0 10
01
02
02
03
04
04
05
06
06
07
08
08
09
1
False positive rate
True
pos
itive
rate
0 10
01
02
02
03
04
04
05
06
06
07
08
08
09
1
False positive rate
True
pos
itive
rate
Training ROC
Figure 3 ROC curves for training and test sets
Competing Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
This study was supported by the National Natural ScienceFoundation of China (Grant no 71432007)
References
[1] K D Kochanek J Xu S L Murphy A M Minino and H-C Kung ldquoDeaths final data for 2009rdquo National Vital StatisticsReports vol 60 no 3 pp 1ndash116 2011
[2] H Temurtas N Yumusak and F Temurtas ldquoA comparativestudy on diabetes disease diagnosis using neural networksrdquoExpert Systems with Applications vol 36 no 4 pp 8610ndash86152009
10 Computational and Mathematical Methods in Medicine
[3] UCI Repository of Machine Learning Databases httparchiveicsuciedumldatasetsStatlog+28Heart29
[4] S-H Lee ldquoFeature selection based on the center of gravity ofBSWFMs using NEWFMrdquo Engineering Applications of ArtificialIntelligence vol 45 pp 482ndash487 2015
[5] D Tomar and S Agarwal ldquoFeature selection based least squaretwin support vector machine for diagnosis of heart diseaserdquoInternational Journal of Bio-Science and Bio-Technology vol 6no 2 pp 69ndash82 2014
[6] M Buscema M Breda and W Lodwick ldquoTraining with InputSelection and Testing (TWIST) algorithm a significant advancein pattern recognition performance of machine learningrdquoJournal of Intelligent Learning Systems and Applications vol 5no 1 pp 29ndash38 2013
[7] C V Subbulakshmi S N Deepa and N Malathi ldquoExtremelearning machine for two category data classificationrdquo inProceedings of the IEEE International Conference on AdvancedCommunication Control and Computing Technologies (ICAC-CCT rsquo12) pp 458ndash461 Ramanathapuram India August 2012
[8] A G Karegowda A SManjunath andM A Jayaram ldquoFeaturesubset selection problem using wrapper approach in supervisedlearningrdquo International Journal of Computer Applications vol 1no 7 pp 13ndash17 2010
[9] K Srinivas B K Rani and A Govrdhan ldquoApplications ofdata mining techniques in healthcare and prediction of heartattacksrdquo International Journal on Computer Science and Engi-neering vol 2 pp 250ndash255 2010
[10] K Polat and S Gunes ldquoA new feature selection methodon classification of medical datasets kernel F-score featureselectionrdquo Expert Systems with Applications vol 36 no 7 pp10367ndash10373 2009
[11] S Ozsen and S Gunes ldquoAttribute weighting via genetic algo-rithms for attributeweighted artificial immune system (AWAIS)and its application to heart disease and liver disorders prob-lemsrdquo Expert Systems with Applications vol 36 no 1 pp 386ndash392 2009
[12] T Helmy and Z Rasheed ldquoMulti-category bioinformaticsdataset classification using extreme learning machinerdquo in Pro-ceedings of the IEEE Congress on Evolutionary Computation(CEC rsquo09) pp 3234ndash3240 Trondheim Norway May 2009
[13] S-J Wang A Mathew Y Chen L-F Xi L Ma and JLee ldquoEmpirical analysis of support vector machine ensembleclassifiersrdquo Expert Systems with Applications vol 36 no 3 pp6466ndash6476 2009
[14] S Ozsen and S Gunes ldquoEffect of feature-type in selectingdistance measure for an artificial immune system as a patternrecognizerrdquoDigital Signal Processing vol 18 no 4 pp 635ndash6452008
[15] H Kahramanli and N Allahverdi ldquoDesign of a hybrid systemfor the diabetes and heart diseasesrdquo Expert Systems withApplications vol 35 no 1-2 pp 82ndash89 2008
[16] G Yan G Ma J Lv and B Song ldquoCombining independentcomponent analysis with support vector machinesrdquo in Pro-ceedings of the in 1st International Symposium on Systems andControl in Aerospace and Astronautics (ISSCAA rsquo06) pp 493ndash496 Harbin China January 2006
[17] S Sahan K Polat H Kodaz and S Gunes ldquoThe medicalapplications of attribute weighted artificial immune system(AWAIS) diagnosis of heart and diabetes diseasesrdquo ArtificialImmune Systems vol 3627 pp 456ndash468 2005
[18] W Duch R Adamczak and K Grabczewski ldquoA new method-ology of extraction optimization and application of crisp and
fuzzy logical rulesrdquo IEEE Transactions on Neural Networks vol12 no 2 pp 277ndash306 2001
[19] M P McRae B Bozkurt C M Ballantyne et al ldquoCardiacScoreCard a diagnostic multivariate index assay system forpredicting a spectrumof cardiovascular diseaserdquoExpert Systemswith Applications vol 54 pp 136ndash147 2016
[20] M Robnik-Sikonja and I Kononenko ldquoTheoretical and empir-ical analysis of ReliefF and RReliefFrdquoMachine Learning vol 53no 1-2 pp 23ndash69 2003
[21] D Wettschereck D W Aha and T Mohri ldquoA review andempirical evaluation of feature weighting methods for a classof lazy learning algorithmsrdquo Artificial Intelligence Review vol11 no 1ndash5 pp 273ndash314 1997
[22] I Kononenko E Simec and M R Sikonja ldquoOvercomingthe myopia of inductive learning algorithms with RELIEFFrdquoApplied Intelligence vol 7 no 1 pp 39ndash55 1997
[23] K Kira and L Rendell ldquoA practical approach to featureselectionrdquo in Proceedings of the International Conference onMachine Learning pp 249ndash256 Morgan Kaufmann AberdeenScotland 1992
[24] I Kononenko ldquoEstimating attributes analysis and extension ofReliefFrdquo in Machine Learning ECML-94 European ConferenceonMachine Learning Catania Italy April 6ndash8 1994 Proceedingsvol 784 of Lecture Notes in Computer Science pp 171ndash182Springer Berlin Germany 1994
[25] R Ruiz J C Riquelme and J S Aguilar-Ruiz ldquoHeuristicsearch over a ranking for feature selectionrdquo in Proceedings of8th InternationalWork-Conference onArtificial NeuralNetworks(IWANN rsquo05) Barcelona Spain June 2005 vol 3512 of LecturesNotes in Computer Science pp 742ndash749 Springer Berlin Ger-many 2005
[26] N Spolaor E A ChermanM CMonard andHD Lee ldquoFilterapproach feature selection methods to support multi-labellearning based on ReliefF and information gainrdquo in Advancesin Artificial IntelligencemdashSBIA 2012 21th Brazilian Symposiumon Artificial Intelligence Curitiba Brazil October 20ndash25 2012Proceedings vol 7589 of Lectures Notes in Computer Science pp72ndash81 Springer Berlin Germany 2012
[27] R Ruiz J C Riquelme and J S Aguilar-Ruiz ldquoFast featureranking algorithmrdquo in Proceedings of the Knowledge-BasedIntelligent Information and Engineering Systems (KES rsquo03) pp325ndash331 Springer Berlin Germany 2003
[28] V Jovanoski and N Lavrac ldquoFeature subset selection inassociation rules learning systemsrdquo in Proceedings of AnalysisWarehousing and Mining the Data pp 74ndash77 1999
[29] J J Liu and J T Y Kwok ldquoAn extended genetic rule inductionalgorithmrdquo in Proceedings of the Congress on EvolutionaryComputation pp 458ndash463 LA Jolla Calif USA July 2000
[30] L-X Zhang J-X Wang Y-N Zhao and Z-H Yang ldquoA novelhybrid feature selection algorithm using ReliefF estimationfor GA-Wrapper searchrdquo in Proceedings of the InternationalConference onMachine Learning andCybernetics vol 1 pp 380ndash384 IEEE Xirsquoan China November 2003
[31] Z Zhao L Wang and H Liu ldquoEfficient spectral featureselectionwithminimum redundancyrdquo in Proceedings of the 24thAAAI Conference on Artificial Intelligence AAAI Atlanta GaUSA 2010
[32] F Nie H Huang X Cai and C H Ding ldquoEfficient androbust feature selection via joint ℓ21-norms minimizationrdquo inAdvances in Neural Information Processing Systems pp 1813ndash1821 MIT Press 2010
Computational and Mathematical Methods in Medicine 11
[33] S-Y Jiang and L-X Wang ldquoEfficient feature selection based oncorrelation measure between continuous and discrete featuresrdquoInformation Processing Letters vol 116 no 2 pp 203ndash215 2016
[34] M J Kearns and U V Vazirani An Introduction to Compu-tational Learning Theory MIT Press Cambridge Mass USA1994
[35] R Bellman Adaptive Control Processes A Guided Tour RANDCorporation Research Studies Princeton University PressPrinceton NJ USA 1961
[36] Z Pawlak ldquoRough set theory and its applications to dataanalysisrdquo Cybernetics and Systems vol 29 no 7 pp 661ndash6881998
[37] A-E Hassanien ldquoRough set approach for attribute reductionand rule generation a case of patients with suspected breastcancerrdquo Journal of the American Society for Information Scienceand Technology vol 55 no 11 pp 954ndash962 2004
[38] A Skowron and C Rauszer ldquoThe discernibility matricesand functions in information systemsrdquo in Intelligent DecisionSupport-Handbook of Applications and Advances of the RoughSetsTheory SystemTheory Knowledge Engineering and ProblemSolving R Słowinski Ed vol 11 pp 331ndash362 KluwerAcademicDordrecht Netherlands 1992
[39] Z Pawlak ldquoRough set approach to knowledge-based decisionsupportrdquo European Journal of Operational Research vol 99 no1 pp 48ndash57 1997
[40] X Wang J Yang X Teng W Xia and R Jensen ldquoFeatureselection based on rough sets and particle swarm optimizationrdquoPattern Recognition Letters vol 28 no 4 pp 459ndash471 2007
[41] C Ma J Ouyang H-L Chen and X-H Zhao ldquoAn efficientdiagnosis system for Parkinsonrsquos disease using kernel-basedextreme learning machine with subtractive clustering featuresweighting approachrdquo Computational and Mathematical Meth-ods in Medicine vol 2014 Article ID 985789 14 pages 2014
[42] A H El-Baz ldquoHybrid intelligent system-based rough set andensemble classifier for breast cancer diagnosisrdquoNeural Comput-ing and Applications vol 26 no 2 pp 437ndash446 2015
[43] J-H Eom S-CKim andB-T Zhang ldquoAptaCDSS-E a classifierensemble-based clinical decision support system for cardiovas-cular disease level predictionrdquo Expert Systems with Applicationsvol 34 no 4 pp 2465ndash2479 2008
[44] Y Ma Ensemble Machine Learning Methods and ApplicationsSpringer New York NY USA 2012
[45] S A Etemad and A Arya ldquoClassification and translation ofstyle and affect in human motion using RBF neural networksrdquoNeurocomputing vol 129 pp 585ndash595 2014
[46] Z Pawlak ldquoRough setsrdquo International Journal of Computer ampInformation Sciences vol 11 no 5 pp 341ndash356 1982
[47] Z Pawlak ldquoRough sets and intelligent data analysisrdquo Informa-tion Sciences vol 147 no 1ndash4 pp 1ndash12 2002
[48] Y Huang P J McCullagh and N D Black ldquoAn optimization ofReliefF for classification in large datasetsrdquoData and KnowledgeEngineering vol 68 no 11 pp 1348ndash1356 2009
[49] J Sun M-Y Jia and H Li ldquoAdaBoost ensemble for financialdistress prediction an empirical comparison with data fromChinese listed companiesrdquo Expert Systems with Applicationsvol 38 no 8 pp 9305ndash9312 2011
[50] R Kohavi and F Provost ldquoGlossary of termsrdquo Machine Learn-ing vol 30 no 2-3 pp 271ndash274 1998
[51] W Chen and H Lin ldquoPrediction of midbody centrosome andkinetochore proteins based on gene ontology informationrdquoBiochemical and Biophysical Research Communications vol 401no 3 pp 382ndash384 2010
[52] K-C Chou and C-T Zhang ldquoPrediction of protein structuralclassesrdquo Critical Reviews in Biochemistry andMolecular Biologyvol 30 no 4 pp 275ndash349 1995
[53] W Chen and H Lin ldquoIdentification of voltage-gated potassiumchannel subfamilies from sequence information using supportvectormachinerdquoComputers in Biology andMedicine vol 42 no4 pp 504ndash507 2012
[54] K-C Chou and H-B Shen ldquoRecent progress in proteinsubcellular location predictionrdquo Analytical Biochemistry vol370 no 1 pp 1ndash16 2007
[55] P Feng H Lin W Chen and Y Zuo ldquoPredicting the typesof J-proteins using clustered amino acidsrdquo BioMed ResearchInternational vol 2014 Article ID 935719 8 pages 2014
[56] K-C Chou and H-B Shen ldquoProtIdent a web server foridentifying proteases and their types by fusing functionaldomain and sequential evolution informationrdquoBiochemical andBiophysical Research Communications vol 376 no 2 pp 321ndash325 2008
[57] K-C Chou and Y-D Cai ldquoPrediction of membrane proteintypes by incorporating amphipathic effectsrdquo Journal of ChemicalInformation and Modeling vol 45 no 2 pp 407ndash413 2005
[58] F Tom ldquoROC graphs notes and practical considerations forresearchersrdquo Machine Learning vol 31 pp 1ndash38 2004
[59] JHuang andCX Ling ldquoUsingAUCand accuracy in evaluatinglearning algorithmsrdquo IEEE Transactions on Knowledge andDataEngineering vol 17 no 3 pp 299ndash310 2005
[60] F D Foresee and M T Hagan ldquoGauss-Newton approximationto Bayesian learningrdquo in Proceedings of the International Con-ference on Neural Networks vol 3 pp 1930ndash1935 Houston TexUSA 1997
4 Computational and Mathematical Methods in Medicine
ReliefF algorithmInput A decision table 119878 = (119880 119875 119876)Output the vector119882 of estimations of the qualities of features(1) set all weights119882[119860] fl 00(2) for 119894 fl 1 to119898 do begin(3) randomly select a sample 119877119894(4) find 119896 nearest hits119867119895(5) for each class 119862 = class(119877119894) do(6) from class 119862 find 119896 nearest misses119872119895(119862)(7) for 119860 fl 1 to a do(8) 119882[119860] = 119882[119860] minus sum119896119895=1 diff(119860 119877119894 119867119895)119898119896 + sum119862 =class(119877119894)[119875(119862)1 minus 119875(class(119877119894)) sum
31 Overview The proposed hybrid classification systemconsists of two main components (i) feature selection usingthe RFRS subsystem and (ii) data classification using theclassification system A flow chart of the proposed systemis shown in Figure 1 We describe the preprocessing andclassification systems in the following subsections
32 RFRS Feature Selection Subsystem We propose a two-phase feature selection method based on the ReliefF algo-rithm and the RS (RFRS) algorithm The idea is to use thefeature estimation from the ReliefF phase as the input andheuristics for the subsequent RS reduction phase In thefirst phase we adopt the ReliefF algorithm to obtain featureweights and select important features in the second phasethe feature estimation obtained from the first phase is usedto guide the initialization of the parameters required for thegenetic algorithm We use a GA-based search engine to findsatisfactory reducts
The RFRS feature selection subsystem consists of threemain modules (i) data discretization (ii) feature extractionusing the ReliefF algorithm and (iii) feature reduction usingthe heuristic RS reduction algorithm we propose
321 Data Discretization RS reduction requires categoricaldata Consequently data discretization is the first step Weused an approximate equal interval binning method to binthe data variables into a small number of categories
322 Feature Extraction by the ReliefF Algorithm Module 2is used for feature extraction by the ReliefF algorithm To dealwith incomplete data we change the diff function Missingfeature values are treated probabilistically [20] We calculatethe probability that two given instances have different valuesfor a given feature conditioned over the class value [20]When one instance has an unknown value then
diff (119860 1198681 1198682) = 1 minus 119875 (value (119860 1198682) | class (1198681)) (6)
When both instances have unknown values then
diff (119860 1198681 1198682)
= 1
minusvalues(119860)sum119881
(119875 (119881 | class (1198681)) times 119875 (119881 | class (1198682)))
(7)
Conditional probabilities are approximated by relativefrequencies in the training set The process of feature extrac-tion is shown as follows
The Process of Feature Extraction Using ReliefF Algorithm
Input A decision table 119878 = (119880 119875 119876) 119875 = 1198861 1198862 119886119898119876 = 1198891 1198892 119889119899 (119898 ge 1 119899 ge 1)
Output The selected feature subset 119870 = 1198861 1198862 119886119896(1 le119896 le 119898)
Step 1 Obtain the weight matrix of each feature using ReliefFalgorithm119882 = 1199081 1199082 119908119894 119908119898 (1 le 119894 le 119898)
Step 2 Set a threshold 120575
Step 3 If 119908119894 gt 120575 then feature 119886119894 is selected
323 Feature Reduction by the Heuristic RS Reduction Algo-rithm The evaluation result obtained by the ReliefF algo-rithm is the feature rank A higher ranking means that thefeature has stronger distinguishing qualities and a higherweight [30] Consequently in the process of reduct searchingthe features in the front rank should have a higher probabilityof being selected
We proposed the RS reduction algorithmby using the fea-ture estimation as heuristics and a GA-based search engine tosearch for the satisfactory reducts The pseudocode of thealgorithm is provided in Algorithm 2 The algorithm wasimplemented in MATLAB R2014a
33 Classification Subsystem In the classification subsystemthe dataset is split into training sets and corresponding test
Computational and Mathematical Methods in Medicine 5
RFRSfeatureselectionsubsystem
Heart disease dataset Data discretization
Classificationsubsystem
An ensemble classifierCross-validation
ReliefF algorithm
Feature extraction
Feature reduction
Heuristic RS reduction algorithm
Optimal performance pi Optimal reduct rij
S = (UA V f) A = C cup D C = C1 D = D1 m ge 1 n ge 1C2 Cm D2 Dn
K = C1 (1 le k le m)C2 Ck
Reducts R = R1 Ri = ri1 i ge 1 1 le j le kR2 Ri ri2 rij
Training set T = T1 Ti = ti1 D i ge 1 1 le j le kti2 tijT2 Ti
Test set V = V1 Vi = i1 D i ge 1 1 le j le kV2 Vi i2 ij
Performance set P = p1 pi = pi1 i ge 1 1 le j le kpi2 pijp2 pi
Trained set T998400 = T9984001 T998400i = t998400i1 D i ge 1 1 le j le kT9984002 T998400i t998400i2 t
998400ij
Figure 1 Structure of RFRS-based classification system
sets The decision tree is a nonparametric learning algorithmthat does not need to search for optimal parameters in thetraining stage and thus is used as a weak learner for ensemblelearning [49] In this paper the ensemble classifier uses theC45 decision tree as the base classifier We use the boostingtechnique to construct ensemble classifiers Jackknife cross-validation is used to increase the amount of data for testingthe results The optimal reduct is the reduct that obtains thebest classification accuracy
4 Experimental Results
41 Dataset TheStatlog (Heart) dataset used in ourworkwasobtained from the UCI machine learning database [3] Thisdataset contains 270 observations and 2 classes the presence
and absence of heart disease The samples include 13 condi-tion features presented in Table 1 We denote the 13 featuresas C1 to C13
42 Performance Evaluation Methods
421 Confusion Matrix Sensitivity Specificity and AccuracyA confusion matrix [50] contains information about actualand predicted classifications performed by a classificationsystem The performance of such systems is commonlyevaluated using the data in the matrix Table 2 shows theconfusion matrix for a two-class classifier
In the confusion matrix TP is the number of true posi-tives representing the cases with heart disease that are cor-rectly classified into the heart disease class FN is the numberof false negatives representing cases with heart disease that
6 Computational and Mathematical Methods in Medicine
Heuristic RS reduction algorithmInput a decision table 119878 = (119880 119862119863) 119862 = 1198881 1198882 119888119898119863 = 1198891 1198892 119889119899Output RedStep 1 Return Core(1) Corelarr (2) For 119894 = 1 to119898(3) Select 119888119894 from 119862(4) Calculate Ind(119862) Ind(119862 minus 119888119894) and Ind(119863)(5) Calculate pos119862(119863) pos119862minus119888119894 (119863) and 119903119862(119863)(6) Calculate Sig(119886119894) Sig(119886119894) = 119903119862(119863) minus 119903119862minus119888119894 (119863)(7) If sig(119888119894) = 0(8) core = core cup 119888119894(9) End if(10) End forStep 2 Return Red(1) Red = Core(2) 1198621015840 = 119862 minus Red(3) While Sig(Red119863) = Sig(119862119863) do
Compute the weight of each feature 119888 in 1198621015840 using the ReliefF algorithmSelect a feature 119888 according to its weight let Red=Red cup 119888Initialize all the necessary parameters for the GA-based search engine according to theresults of the last step and search for satisfactory reducts
End while
Algorithm 2 Pseudocode of heuristic RS reduction algorithm
are classified into the healthy class TN is the number of truenegatives representing healthy cases that are correctly classi-fied into the healthy class Finally FP is the number of falsepositives representing the healthy cases that are incorrectlyclassified into the heart disease class [50]
The performance of the proposed system was evaluatedbased on sensitivity specificity and accuracy tests which usethe true positive (TP) true negative (TN) false negative (FN)and false positive (FP) terms [33]These criteria are calculatedas follows [41]
Sensitivity (Sn) = TPTP + FN
times 100
Specificity (Sp) = TNFP + TN
times 100
Accuracy (Acc) = TP + TNTP + TN + FP + FN
times 100
(8)
422 Cross-Validation Three cross-validation methodsnamely subsampling tests independent dataset tests andjackknife tests are often employed to evaluate the predictivecapability of a predictor [51] Among the three methodsthe jackknife test is deemed the least arbitrary and the mostobjective and rigorous [52 53] because it always yields aunique outcome as demonstrated by a penetrating analysis ina recent comprehensive review [54 55] Therefore thejackknife test has been widely and increasingly adopted inmany areas [56 57]
Accordingly the jackknife test was employed to examinethe performance of the model proposed in this paper Forjackknife cross-validation each sequence in the training
dataset is in turn singled out as an independent test sampleand all the parameter rules are calculated based on theremaining samples without including the one being treatedas the test sample
423 Receiver Operating Characteristics (ROC) The receiveroperating characteristic (ROC) curve is used for analyzing theprediction performance of a predictor [58] It is usually plot-ted using the true positive rate versus the false positive rate asthe discrimination threshold of classification algorithm isvaried The area under the ROC curve (AUC) is widely usedand relatively accepted in classification studies because itprovides a good summary of a classifierrsquos performance [59]
43 Results and Discussion
431 Results and Analysis on the Statlog (Heart) DatasetFirst we used the equal interval binningmethod to discretizethe original data In the feature extraction module the num-ber of k-nearest neighbors in the ReliefF algorithm was set to10 and the threshold 120575 was set to 002 Table 3 summarizesthe results of the ReliefF algorithm Based on these resultsC5and C6 were removed In Module 3 we obtained 15 reductsusing the heuristic RS reduction algorithm implemented inMATLAB 2014a
Trials were conducted using 70ndash30 training-test par-titions using all the reduced feature sets Jackknife cross-validation was performed on the dataset The number ofdesired base classifiers k was set to 50 100 and 150 Thecalculations were run 10 times and the highest classification
Computational and Mathematical Methods in Medicine 7
Table 1 Feature information of Statlog (Heart) dataset
Feature Code Description Domain Data type Mean Standard deviationAge 1198621 mdash 29ndash77 Real 54 9Sex 1198622 Male female 0 1 Binary mdash mdash
Chest pain type 1198623Angina
asymptomaticabnormal
1 2 3 4 Nominal mdash mdash
Resting blood pressure 1198624 mdash 94ndash200 Real 131344 17862Serum cholesterol in mgdl 1198625 mdash 126ndash564 Real 249659 51686Fasting blood sugar gt 120mgdl 1198626 mdash 0 1 Binary mdash mdash
Resting electrocardiographic results 1198627Norm
abnormal hyper 0 1 2 Nominal mdash mdash
Maximum heart rate achieved 1198628 mdash 71ndash202 Real 149678 231666Exercise-induced angina 1198629 mdash 0 1 Binary mdash mdashOld peak = ST depression induced by exercise relative to rest 11986210 mdash 0ndash62 Real 105 1145Slope of the peak exercise ST segment 11986211 Up flat down 1 2 3 Ordered mdash mdashNumber of major vessels (0ndash3) colored by fluoroscopy 11986212 mdash 0 1 2 3 Real mdash mdash
Thal 11986213Normal fixed
defectreversible defect
3 6 7 Nominal mdash mdash
Table 2 The confusion matrix
Predicted patientswith heart disease
Predicted healthypersons
Actual patients withheart disease True positive (TP) False negative (FN)
Actual healthypersons False positive (FP) True negative (TN)
performances for each training-test partition are provided inTable 4
In Table 4 1198772 obtains the best test set classificationaccuracy (9259) using the ensemble classifiers when 119896 =100 The training process is shown in Figure 2 The trainingand test ROC curves are shown in Figure 3
432 Comparison with Other Classifiers In this sectionour ensemble classification method is compared with theindividual C45 decision tree and Naıve Bayes and BayesianNeural Networks (BNN)methodsTheC45 decision tree andNaıve Bayes are common classifiers Bayesian Neural Net-works (BNN) is a classifier that uses Bayesian regularizationto train feed-forward neural networks [60] and has betterperformance than pure neural networks The classificationaccuracy results of the four classifiers are listed in Table 5Theensemble classification method has better performance thanthe individual C45 classifier and the other two classifiers
433 Comparison of the Results with Other Studies Wecompared our results with the results of other studies Table 6shows the classification accuracies of our study and previousmethods
0 10 20 30 40 50 60 70 80 90 100008
01
012
014
016
018
02
022
024
Number of trees
Test
class
ifica
tion
erro
r
Figure 2 Training process of 1198777
The results show that our proposed method obtainssuperior and promising results in classifying heart diseasepatients We believe that the proposed RFRS-based classi-fication system can be exceedingly beneficial in assistingphysicians in making accurate decisions
8 Computational and Mathematical Methods in Medicine
In this paper a novel ReliefF and Rough Set- (RFRS-)based classification system is proposed for heart diseasediagnosisThemain novelty of this paper lies in the proposedapproach the combination of the ReliefF and RS methodsto classify heart disease problems in an efficient and fastmanner The RFRS classification system consists of twosubsystems the RFRS feature selection subsystem and theclassification subsystemThe Statlog (Heart) dataset from theUCI machine learning database [3] was selected to test thesystemThe experimental results show that the reduct1198772 (11986211198623 1198627 1198628 11986211 11986212 11986213) achieves the highest classificationaccuracy (9259) using an ensemble classifier with the C45decision tree as the weak learner The results also show thatthe RFRS method has superior performance compared to
three common classifiers in terms of ACC sensitivity andspecificity In addition the performance of the proposedsystem is superior to that of existingmethods in the literatureBased on empirical analysis the results indicate that theproposed classification system can be used as a promisingalternative tool in medical decision making for heart diseasediagnosis
However the proposed method also has some weak-nesses The number of the nearest neighbors (119896) and theweight threshold (120579) are not stable in the ReliefF algorithm[20] One solution to this problem is to compute estimatesfor all possible numbers and take the highest estimate of eachfeature as the final result [20] We need to perform moreexperiments to find the optimal parameter values for theReliefF algorithm in the future
Computational and Mathematical Methods in Medicine 9
Table 6 Comparison of our results with those of other studies
Author Method Classification accuracy ()Our study RFRS classification system 9259Lee [4] Graphical characteristics of BSWFM combined with Euclidean distance 874Tomar and Agarwal [5] Feature selection-based LSTSVM 8559Buscema et al [6] TWIST algorithm 8414Subbulakshmi et al [7] ELM 875Karegowda et al [8] GA + Naıve Bayes 8587Srinivas et al [9] Naıve Bayes 8370Polat and Gunes [10] RBF kernel 119865-score + LS-SVM 8370Ozsen and Gunes [11] GA-AWAIS 8743Helmy and Rasheed [12] Algebraic Sigmoid 8524Wang et al [13] Linear kernel SVM classifiers 8337Ozsen and Gunes [14] Hybrid similarity measure 8395Kahramanli and Allahverdi [15] Hybrid neural network method 868Yan et al [16] ICA + SVM 8375Sahan et al [17] AWAIS 8259Duch et al [18] KNN classifier 856BSWFM bounded sumof weighted fuzzymembership functions LSTSVM Least Square Twin Support VectorMachine TWIST Training with Input Selectionand Testing ELM Extreme LearningMachine GA genetic algorithm SVM support vectormachine ICA imperialist competitive algorithm AWAIS attributeweighted artificial immune system119870NN 119896-nearest neighbor
Test ROC
0 10
01
02
02
03
04
04
05
06
06
07
08
08
09
1
False positive rate
True
pos
itive
rate
0 10
01
02
02
03
04
04
05
06
06
07
08
08
09
1
False positive rate
True
pos
itive
rate
Training ROC
Figure 3 ROC curves for training and test sets
Competing Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
This study was supported by the National Natural ScienceFoundation of China (Grant no 71432007)
References
[1] K D Kochanek J Xu S L Murphy A M Minino and H-C Kung ldquoDeaths final data for 2009rdquo National Vital StatisticsReports vol 60 no 3 pp 1ndash116 2011
[2] H Temurtas N Yumusak and F Temurtas ldquoA comparativestudy on diabetes disease diagnosis using neural networksrdquoExpert Systems with Applications vol 36 no 4 pp 8610ndash86152009
10 Computational and Mathematical Methods in Medicine
[3] UCI Repository of Machine Learning Databases httparchiveicsuciedumldatasetsStatlog+28Heart29
[4] S-H Lee ldquoFeature selection based on the center of gravity ofBSWFMs using NEWFMrdquo Engineering Applications of ArtificialIntelligence vol 45 pp 482ndash487 2015
[5] D Tomar and S Agarwal ldquoFeature selection based least squaretwin support vector machine for diagnosis of heart diseaserdquoInternational Journal of Bio-Science and Bio-Technology vol 6no 2 pp 69ndash82 2014
[6] M Buscema M Breda and W Lodwick ldquoTraining with InputSelection and Testing (TWIST) algorithm a significant advancein pattern recognition performance of machine learningrdquoJournal of Intelligent Learning Systems and Applications vol 5no 1 pp 29ndash38 2013
[7] C V Subbulakshmi S N Deepa and N Malathi ldquoExtremelearning machine for two category data classificationrdquo inProceedings of the IEEE International Conference on AdvancedCommunication Control and Computing Technologies (ICAC-CCT rsquo12) pp 458ndash461 Ramanathapuram India August 2012
[8] A G Karegowda A SManjunath andM A Jayaram ldquoFeaturesubset selection problem using wrapper approach in supervisedlearningrdquo International Journal of Computer Applications vol 1no 7 pp 13ndash17 2010
[9] K Srinivas B K Rani and A Govrdhan ldquoApplications ofdata mining techniques in healthcare and prediction of heartattacksrdquo International Journal on Computer Science and Engi-neering vol 2 pp 250ndash255 2010
[10] K Polat and S Gunes ldquoA new feature selection methodon classification of medical datasets kernel F-score featureselectionrdquo Expert Systems with Applications vol 36 no 7 pp10367ndash10373 2009
[11] S Ozsen and S Gunes ldquoAttribute weighting via genetic algo-rithms for attributeweighted artificial immune system (AWAIS)and its application to heart disease and liver disorders prob-lemsrdquo Expert Systems with Applications vol 36 no 1 pp 386ndash392 2009
[12] T Helmy and Z Rasheed ldquoMulti-category bioinformaticsdataset classification using extreme learning machinerdquo in Pro-ceedings of the IEEE Congress on Evolutionary Computation(CEC rsquo09) pp 3234ndash3240 Trondheim Norway May 2009
[13] S-J Wang A Mathew Y Chen L-F Xi L Ma and JLee ldquoEmpirical analysis of support vector machine ensembleclassifiersrdquo Expert Systems with Applications vol 36 no 3 pp6466ndash6476 2009
[14] S Ozsen and S Gunes ldquoEffect of feature-type in selectingdistance measure for an artificial immune system as a patternrecognizerrdquoDigital Signal Processing vol 18 no 4 pp 635ndash6452008
[15] H Kahramanli and N Allahverdi ldquoDesign of a hybrid systemfor the diabetes and heart diseasesrdquo Expert Systems withApplications vol 35 no 1-2 pp 82ndash89 2008
[16] G Yan G Ma J Lv and B Song ldquoCombining independentcomponent analysis with support vector machinesrdquo in Pro-ceedings of the in 1st International Symposium on Systems andControl in Aerospace and Astronautics (ISSCAA rsquo06) pp 493ndash496 Harbin China January 2006
[17] S Sahan K Polat H Kodaz and S Gunes ldquoThe medicalapplications of attribute weighted artificial immune system(AWAIS) diagnosis of heart and diabetes diseasesrdquo ArtificialImmune Systems vol 3627 pp 456ndash468 2005
[18] W Duch R Adamczak and K Grabczewski ldquoA new method-ology of extraction optimization and application of crisp and
fuzzy logical rulesrdquo IEEE Transactions on Neural Networks vol12 no 2 pp 277ndash306 2001
[19] M P McRae B Bozkurt C M Ballantyne et al ldquoCardiacScoreCard a diagnostic multivariate index assay system forpredicting a spectrumof cardiovascular diseaserdquoExpert Systemswith Applications vol 54 pp 136ndash147 2016
[20] M Robnik-Sikonja and I Kononenko ldquoTheoretical and empir-ical analysis of ReliefF and RReliefFrdquoMachine Learning vol 53no 1-2 pp 23ndash69 2003
[21] D Wettschereck D W Aha and T Mohri ldquoA review andempirical evaluation of feature weighting methods for a classof lazy learning algorithmsrdquo Artificial Intelligence Review vol11 no 1ndash5 pp 273ndash314 1997
[22] I Kononenko E Simec and M R Sikonja ldquoOvercomingthe myopia of inductive learning algorithms with RELIEFFrdquoApplied Intelligence vol 7 no 1 pp 39ndash55 1997
[23] K Kira and L Rendell ldquoA practical approach to featureselectionrdquo in Proceedings of the International Conference onMachine Learning pp 249ndash256 Morgan Kaufmann AberdeenScotland 1992
[24] I Kononenko ldquoEstimating attributes analysis and extension ofReliefFrdquo in Machine Learning ECML-94 European ConferenceonMachine Learning Catania Italy April 6ndash8 1994 Proceedingsvol 784 of Lecture Notes in Computer Science pp 171ndash182Springer Berlin Germany 1994
[25] R Ruiz J C Riquelme and J S Aguilar-Ruiz ldquoHeuristicsearch over a ranking for feature selectionrdquo in Proceedings of8th InternationalWork-Conference onArtificial NeuralNetworks(IWANN rsquo05) Barcelona Spain June 2005 vol 3512 of LecturesNotes in Computer Science pp 742ndash749 Springer Berlin Ger-many 2005
[26] N Spolaor E A ChermanM CMonard andHD Lee ldquoFilterapproach feature selection methods to support multi-labellearning based on ReliefF and information gainrdquo in Advancesin Artificial IntelligencemdashSBIA 2012 21th Brazilian Symposiumon Artificial Intelligence Curitiba Brazil October 20ndash25 2012Proceedings vol 7589 of Lectures Notes in Computer Science pp72ndash81 Springer Berlin Germany 2012
[27] R Ruiz J C Riquelme and J S Aguilar-Ruiz ldquoFast featureranking algorithmrdquo in Proceedings of the Knowledge-BasedIntelligent Information and Engineering Systems (KES rsquo03) pp325ndash331 Springer Berlin Germany 2003
[28] V Jovanoski and N Lavrac ldquoFeature subset selection inassociation rules learning systemsrdquo in Proceedings of AnalysisWarehousing and Mining the Data pp 74ndash77 1999
[29] J J Liu and J T Y Kwok ldquoAn extended genetic rule inductionalgorithmrdquo in Proceedings of the Congress on EvolutionaryComputation pp 458ndash463 LA Jolla Calif USA July 2000
[30] L-X Zhang J-X Wang Y-N Zhao and Z-H Yang ldquoA novelhybrid feature selection algorithm using ReliefF estimationfor GA-Wrapper searchrdquo in Proceedings of the InternationalConference onMachine Learning andCybernetics vol 1 pp 380ndash384 IEEE Xirsquoan China November 2003
[31] Z Zhao L Wang and H Liu ldquoEfficient spectral featureselectionwithminimum redundancyrdquo in Proceedings of the 24thAAAI Conference on Artificial Intelligence AAAI Atlanta GaUSA 2010
[32] F Nie H Huang X Cai and C H Ding ldquoEfficient androbust feature selection via joint ℓ21-norms minimizationrdquo inAdvances in Neural Information Processing Systems pp 1813ndash1821 MIT Press 2010
Computational and Mathematical Methods in Medicine 11
[33] S-Y Jiang and L-X Wang ldquoEfficient feature selection based oncorrelation measure between continuous and discrete featuresrdquoInformation Processing Letters vol 116 no 2 pp 203ndash215 2016
[34] M J Kearns and U V Vazirani An Introduction to Compu-tational Learning Theory MIT Press Cambridge Mass USA1994
[35] R Bellman Adaptive Control Processes A Guided Tour RANDCorporation Research Studies Princeton University PressPrinceton NJ USA 1961
[36] Z Pawlak ldquoRough set theory and its applications to dataanalysisrdquo Cybernetics and Systems vol 29 no 7 pp 661ndash6881998
[37] A-E Hassanien ldquoRough set approach for attribute reductionand rule generation a case of patients with suspected breastcancerrdquo Journal of the American Society for Information Scienceand Technology vol 55 no 11 pp 954ndash962 2004
[38] A Skowron and C Rauszer ldquoThe discernibility matricesand functions in information systemsrdquo in Intelligent DecisionSupport-Handbook of Applications and Advances of the RoughSetsTheory SystemTheory Knowledge Engineering and ProblemSolving R Słowinski Ed vol 11 pp 331ndash362 KluwerAcademicDordrecht Netherlands 1992
[39] Z Pawlak ldquoRough set approach to knowledge-based decisionsupportrdquo European Journal of Operational Research vol 99 no1 pp 48ndash57 1997
[40] X Wang J Yang X Teng W Xia and R Jensen ldquoFeatureselection based on rough sets and particle swarm optimizationrdquoPattern Recognition Letters vol 28 no 4 pp 459ndash471 2007
[41] C Ma J Ouyang H-L Chen and X-H Zhao ldquoAn efficientdiagnosis system for Parkinsonrsquos disease using kernel-basedextreme learning machine with subtractive clustering featuresweighting approachrdquo Computational and Mathematical Meth-ods in Medicine vol 2014 Article ID 985789 14 pages 2014
[42] A H El-Baz ldquoHybrid intelligent system-based rough set andensemble classifier for breast cancer diagnosisrdquoNeural Comput-ing and Applications vol 26 no 2 pp 437ndash446 2015
[43] J-H Eom S-CKim andB-T Zhang ldquoAptaCDSS-E a classifierensemble-based clinical decision support system for cardiovas-cular disease level predictionrdquo Expert Systems with Applicationsvol 34 no 4 pp 2465ndash2479 2008
[44] Y Ma Ensemble Machine Learning Methods and ApplicationsSpringer New York NY USA 2012
[45] S A Etemad and A Arya ldquoClassification and translation ofstyle and affect in human motion using RBF neural networksrdquoNeurocomputing vol 129 pp 585ndash595 2014
[46] Z Pawlak ldquoRough setsrdquo International Journal of Computer ampInformation Sciences vol 11 no 5 pp 341ndash356 1982
[47] Z Pawlak ldquoRough sets and intelligent data analysisrdquo Informa-tion Sciences vol 147 no 1ndash4 pp 1ndash12 2002
[48] Y Huang P J McCullagh and N D Black ldquoAn optimization ofReliefF for classification in large datasetsrdquoData and KnowledgeEngineering vol 68 no 11 pp 1348ndash1356 2009
[49] J Sun M-Y Jia and H Li ldquoAdaBoost ensemble for financialdistress prediction an empirical comparison with data fromChinese listed companiesrdquo Expert Systems with Applicationsvol 38 no 8 pp 9305ndash9312 2011
[50] R Kohavi and F Provost ldquoGlossary of termsrdquo Machine Learn-ing vol 30 no 2-3 pp 271ndash274 1998
[51] W Chen and H Lin ldquoPrediction of midbody centrosome andkinetochore proteins based on gene ontology informationrdquoBiochemical and Biophysical Research Communications vol 401no 3 pp 382ndash384 2010
[52] K-C Chou and C-T Zhang ldquoPrediction of protein structuralclassesrdquo Critical Reviews in Biochemistry andMolecular Biologyvol 30 no 4 pp 275ndash349 1995
[53] W Chen and H Lin ldquoIdentification of voltage-gated potassiumchannel subfamilies from sequence information using supportvectormachinerdquoComputers in Biology andMedicine vol 42 no4 pp 504ndash507 2012
[54] K-C Chou and H-B Shen ldquoRecent progress in proteinsubcellular location predictionrdquo Analytical Biochemistry vol370 no 1 pp 1ndash16 2007
[55] P Feng H Lin W Chen and Y Zuo ldquoPredicting the typesof J-proteins using clustered amino acidsrdquo BioMed ResearchInternational vol 2014 Article ID 935719 8 pages 2014
[56] K-C Chou and H-B Shen ldquoProtIdent a web server foridentifying proteases and their types by fusing functionaldomain and sequential evolution informationrdquoBiochemical andBiophysical Research Communications vol 376 no 2 pp 321ndash325 2008
[57] K-C Chou and Y-D Cai ldquoPrediction of membrane proteintypes by incorporating amphipathic effectsrdquo Journal of ChemicalInformation and Modeling vol 45 no 2 pp 407ndash413 2005
[58] F Tom ldquoROC graphs notes and practical considerations forresearchersrdquo Machine Learning vol 31 pp 1ndash38 2004
[59] JHuang andCX Ling ldquoUsingAUCand accuracy in evaluatinglearning algorithmsrdquo IEEE Transactions on Knowledge andDataEngineering vol 17 no 3 pp 299ndash310 2005
[60] F D Foresee and M T Hagan ldquoGauss-Newton approximationto Bayesian learningrdquo in Proceedings of the International Con-ference on Neural Networks vol 3 pp 1930ndash1935 Houston TexUSA 1997
Computational and Mathematical Methods in Medicine 5
RFRSfeatureselectionsubsystem
Heart disease dataset Data discretization
Classificationsubsystem
An ensemble classifierCross-validation
ReliefF algorithm
Feature extraction
Feature reduction
Heuristic RS reduction algorithm
Optimal performance pi Optimal reduct rij
S = (UA V f) A = C cup D C = C1 D = D1 m ge 1 n ge 1C2 Cm D2 Dn
K = C1 (1 le k le m)C2 Ck
Reducts R = R1 Ri = ri1 i ge 1 1 le j le kR2 Ri ri2 rij
Training set T = T1 Ti = ti1 D i ge 1 1 le j le kti2 tijT2 Ti
Test set V = V1 Vi = i1 D i ge 1 1 le j le kV2 Vi i2 ij
Performance set P = p1 pi = pi1 i ge 1 1 le j le kpi2 pijp2 pi
Trained set T998400 = T9984001 T998400i = t998400i1 D i ge 1 1 le j le kT9984002 T998400i t998400i2 t
998400ij
Figure 1 Structure of RFRS-based classification system
sets The decision tree is a nonparametric learning algorithmthat does not need to search for optimal parameters in thetraining stage and thus is used as a weak learner for ensemblelearning [49] In this paper the ensemble classifier uses theC45 decision tree as the base classifier We use the boostingtechnique to construct ensemble classifiers Jackknife cross-validation is used to increase the amount of data for testingthe results The optimal reduct is the reduct that obtains thebest classification accuracy
4 Experimental Results
41 Dataset TheStatlog (Heart) dataset used in ourworkwasobtained from the UCI machine learning database [3] Thisdataset contains 270 observations and 2 classes the presence
and absence of heart disease The samples include 13 condi-tion features presented in Table 1 We denote the 13 featuresas C1 to C13
42 Performance Evaluation Methods
421 Confusion Matrix Sensitivity Specificity and AccuracyA confusion matrix [50] contains information about actualand predicted classifications performed by a classificationsystem The performance of such systems is commonlyevaluated using the data in the matrix Table 2 shows theconfusion matrix for a two-class classifier
In the confusion matrix TP is the number of true posi-tives representing the cases with heart disease that are cor-rectly classified into the heart disease class FN is the numberof false negatives representing cases with heart disease that
6 Computational and Mathematical Methods in Medicine
Heuristic RS reduction algorithmInput a decision table 119878 = (119880 119862119863) 119862 = 1198881 1198882 119888119898119863 = 1198891 1198892 119889119899Output RedStep 1 Return Core(1) Corelarr (2) For 119894 = 1 to119898(3) Select 119888119894 from 119862(4) Calculate Ind(119862) Ind(119862 minus 119888119894) and Ind(119863)(5) Calculate pos119862(119863) pos119862minus119888119894 (119863) and 119903119862(119863)(6) Calculate Sig(119886119894) Sig(119886119894) = 119903119862(119863) minus 119903119862minus119888119894 (119863)(7) If sig(119888119894) = 0(8) core = core cup 119888119894(9) End if(10) End forStep 2 Return Red(1) Red = Core(2) 1198621015840 = 119862 minus Red(3) While Sig(Red119863) = Sig(119862119863) do
Compute the weight of each feature 119888 in 1198621015840 using the ReliefF algorithmSelect a feature 119888 according to its weight let Red=Red cup 119888Initialize all the necessary parameters for the GA-based search engine according to theresults of the last step and search for satisfactory reducts
End while
Algorithm 2 Pseudocode of heuristic RS reduction algorithm
are classified into the healthy class TN is the number of truenegatives representing healthy cases that are correctly classi-fied into the healthy class Finally FP is the number of falsepositives representing the healthy cases that are incorrectlyclassified into the heart disease class [50]
The performance of the proposed system was evaluatedbased on sensitivity specificity and accuracy tests which usethe true positive (TP) true negative (TN) false negative (FN)and false positive (FP) terms [33]These criteria are calculatedas follows [41]
Sensitivity (Sn) = TPTP + FN
times 100
Specificity (Sp) = TNFP + TN
times 100
Accuracy (Acc) = TP + TNTP + TN + FP + FN
times 100
(8)
422 Cross-Validation Three cross-validation methodsnamely subsampling tests independent dataset tests andjackknife tests are often employed to evaluate the predictivecapability of a predictor [51] Among the three methodsthe jackknife test is deemed the least arbitrary and the mostobjective and rigorous [52 53] because it always yields aunique outcome as demonstrated by a penetrating analysis ina recent comprehensive review [54 55] Therefore thejackknife test has been widely and increasingly adopted inmany areas [56 57]
Accordingly the jackknife test was employed to examinethe performance of the model proposed in this paper Forjackknife cross-validation each sequence in the training
dataset is in turn singled out as an independent test sampleand all the parameter rules are calculated based on theremaining samples without including the one being treatedas the test sample
423 Receiver Operating Characteristics (ROC) The receiveroperating characteristic (ROC) curve is used for analyzing theprediction performance of a predictor [58] It is usually plot-ted using the true positive rate versus the false positive rate asthe discrimination threshold of classification algorithm isvaried The area under the ROC curve (AUC) is widely usedand relatively accepted in classification studies because itprovides a good summary of a classifierrsquos performance [59]
43 Results and Discussion
431 Results and Analysis on the Statlog (Heart) DatasetFirst we used the equal interval binningmethod to discretizethe original data In the feature extraction module the num-ber of k-nearest neighbors in the ReliefF algorithm was set to10 and the threshold 120575 was set to 002 Table 3 summarizesthe results of the ReliefF algorithm Based on these resultsC5and C6 were removed In Module 3 we obtained 15 reductsusing the heuristic RS reduction algorithm implemented inMATLAB 2014a
Trials were conducted using 70ndash30 training-test par-titions using all the reduced feature sets Jackknife cross-validation was performed on the dataset The number ofdesired base classifiers k was set to 50 100 and 150 Thecalculations were run 10 times and the highest classification
Computational and Mathematical Methods in Medicine 7
Table 1 Feature information of Statlog (Heart) dataset
Feature Code Description Domain Data type Mean Standard deviationAge 1198621 mdash 29ndash77 Real 54 9Sex 1198622 Male female 0 1 Binary mdash mdash
Chest pain type 1198623Angina
asymptomaticabnormal
1 2 3 4 Nominal mdash mdash
Resting blood pressure 1198624 mdash 94ndash200 Real 131344 17862Serum cholesterol in mgdl 1198625 mdash 126ndash564 Real 249659 51686Fasting blood sugar gt 120mgdl 1198626 mdash 0 1 Binary mdash mdash
Resting electrocardiographic results 1198627Norm
abnormal hyper 0 1 2 Nominal mdash mdash
Maximum heart rate achieved 1198628 mdash 71ndash202 Real 149678 231666Exercise-induced angina 1198629 mdash 0 1 Binary mdash mdashOld peak = ST depression induced by exercise relative to rest 11986210 mdash 0ndash62 Real 105 1145Slope of the peak exercise ST segment 11986211 Up flat down 1 2 3 Ordered mdash mdashNumber of major vessels (0ndash3) colored by fluoroscopy 11986212 mdash 0 1 2 3 Real mdash mdash
Thal 11986213Normal fixed
defectreversible defect
3 6 7 Nominal mdash mdash
Table 2 The confusion matrix
Predicted patientswith heart disease
Predicted healthypersons
Actual patients withheart disease True positive (TP) False negative (FN)
Actual healthypersons False positive (FP) True negative (TN)
performances for each training-test partition are provided inTable 4
In Table 4 1198772 obtains the best test set classificationaccuracy (9259) using the ensemble classifiers when 119896 =100 The training process is shown in Figure 2 The trainingand test ROC curves are shown in Figure 3
432 Comparison with Other Classifiers In this sectionour ensemble classification method is compared with theindividual C45 decision tree and Naıve Bayes and BayesianNeural Networks (BNN)methodsTheC45 decision tree andNaıve Bayes are common classifiers Bayesian Neural Net-works (BNN) is a classifier that uses Bayesian regularizationto train feed-forward neural networks [60] and has betterperformance than pure neural networks The classificationaccuracy results of the four classifiers are listed in Table 5Theensemble classification method has better performance thanthe individual C45 classifier and the other two classifiers
433 Comparison of the Results with Other Studies Wecompared our results with the results of other studies Table 6shows the classification accuracies of our study and previousmethods
0 10 20 30 40 50 60 70 80 90 100008
01
012
014
016
018
02
022
024
Number of trees
Test
class
ifica
tion
erro
r
Figure 2 Training process of 1198777
The results show that our proposed method obtainssuperior and promising results in classifying heart diseasepatients We believe that the proposed RFRS-based classi-fication system can be exceedingly beneficial in assistingphysicians in making accurate decisions
8 Computational and Mathematical Methods in Medicine
In this paper a novel ReliefF and Rough Set- (RFRS-)based classification system is proposed for heart diseasediagnosisThemain novelty of this paper lies in the proposedapproach the combination of the ReliefF and RS methodsto classify heart disease problems in an efficient and fastmanner The RFRS classification system consists of twosubsystems the RFRS feature selection subsystem and theclassification subsystemThe Statlog (Heart) dataset from theUCI machine learning database [3] was selected to test thesystemThe experimental results show that the reduct1198772 (11986211198623 1198627 1198628 11986211 11986212 11986213) achieves the highest classificationaccuracy (9259) using an ensemble classifier with the C45decision tree as the weak learner The results also show thatthe RFRS method has superior performance compared to
three common classifiers in terms of ACC sensitivity andspecificity In addition the performance of the proposedsystem is superior to that of existingmethods in the literatureBased on empirical analysis the results indicate that theproposed classification system can be used as a promisingalternative tool in medical decision making for heart diseasediagnosis
However the proposed method also has some weak-nesses The number of the nearest neighbors (119896) and theweight threshold (120579) are not stable in the ReliefF algorithm[20] One solution to this problem is to compute estimatesfor all possible numbers and take the highest estimate of eachfeature as the final result [20] We need to perform moreexperiments to find the optimal parameter values for theReliefF algorithm in the future
Computational and Mathematical Methods in Medicine 9
Table 6 Comparison of our results with those of other studies
Author Method Classification accuracy ()Our study RFRS classification system 9259Lee [4] Graphical characteristics of BSWFM combined with Euclidean distance 874Tomar and Agarwal [5] Feature selection-based LSTSVM 8559Buscema et al [6] TWIST algorithm 8414Subbulakshmi et al [7] ELM 875Karegowda et al [8] GA + Naıve Bayes 8587Srinivas et al [9] Naıve Bayes 8370Polat and Gunes [10] RBF kernel 119865-score + LS-SVM 8370Ozsen and Gunes [11] GA-AWAIS 8743Helmy and Rasheed [12] Algebraic Sigmoid 8524Wang et al [13] Linear kernel SVM classifiers 8337Ozsen and Gunes [14] Hybrid similarity measure 8395Kahramanli and Allahverdi [15] Hybrid neural network method 868Yan et al [16] ICA + SVM 8375Sahan et al [17] AWAIS 8259Duch et al [18] KNN classifier 856BSWFM bounded sumof weighted fuzzymembership functions LSTSVM Least Square Twin Support VectorMachine TWIST Training with Input Selectionand Testing ELM Extreme LearningMachine GA genetic algorithm SVM support vectormachine ICA imperialist competitive algorithm AWAIS attributeweighted artificial immune system119870NN 119896-nearest neighbor
Test ROC
0 10
01
02
02
03
04
04
05
06
06
07
08
08
09
1
False positive rate
True
pos
itive
rate
0 10
01
02
02
03
04
04
05
06
06
07
08
08
09
1
False positive rate
True
pos
itive
rate
Training ROC
Figure 3 ROC curves for training and test sets
Competing Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
This study was supported by the National Natural ScienceFoundation of China (Grant no 71432007)
References
[1] K D Kochanek J Xu S L Murphy A M Minino and H-C Kung ldquoDeaths final data for 2009rdquo National Vital StatisticsReports vol 60 no 3 pp 1ndash116 2011
[2] H Temurtas N Yumusak and F Temurtas ldquoA comparativestudy on diabetes disease diagnosis using neural networksrdquoExpert Systems with Applications vol 36 no 4 pp 8610ndash86152009
10 Computational and Mathematical Methods in Medicine
[3] UCI Repository of Machine Learning Databases httparchiveicsuciedumldatasetsStatlog+28Heart29
[4] S-H Lee ldquoFeature selection based on the center of gravity ofBSWFMs using NEWFMrdquo Engineering Applications of ArtificialIntelligence vol 45 pp 482ndash487 2015
[5] D Tomar and S Agarwal ldquoFeature selection based least squaretwin support vector machine for diagnosis of heart diseaserdquoInternational Journal of Bio-Science and Bio-Technology vol 6no 2 pp 69ndash82 2014
[6] M Buscema M Breda and W Lodwick ldquoTraining with InputSelection and Testing (TWIST) algorithm a significant advancein pattern recognition performance of machine learningrdquoJournal of Intelligent Learning Systems and Applications vol 5no 1 pp 29ndash38 2013
[7] C V Subbulakshmi S N Deepa and N Malathi ldquoExtremelearning machine for two category data classificationrdquo inProceedings of the IEEE International Conference on AdvancedCommunication Control and Computing Technologies (ICAC-CCT rsquo12) pp 458ndash461 Ramanathapuram India August 2012
[8] A G Karegowda A SManjunath andM A Jayaram ldquoFeaturesubset selection problem using wrapper approach in supervisedlearningrdquo International Journal of Computer Applications vol 1no 7 pp 13ndash17 2010
[9] K Srinivas B K Rani and A Govrdhan ldquoApplications ofdata mining techniques in healthcare and prediction of heartattacksrdquo International Journal on Computer Science and Engi-neering vol 2 pp 250ndash255 2010
[10] K Polat and S Gunes ldquoA new feature selection methodon classification of medical datasets kernel F-score featureselectionrdquo Expert Systems with Applications vol 36 no 7 pp10367ndash10373 2009
[11] S Ozsen and S Gunes ldquoAttribute weighting via genetic algo-rithms for attributeweighted artificial immune system (AWAIS)and its application to heart disease and liver disorders prob-lemsrdquo Expert Systems with Applications vol 36 no 1 pp 386ndash392 2009
[12] T Helmy and Z Rasheed ldquoMulti-category bioinformaticsdataset classification using extreme learning machinerdquo in Pro-ceedings of the IEEE Congress on Evolutionary Computation(CEC rsquo09) pp 3234ndash3240 Trondheim Norway May 2009
[13] S-J Wang A Mathew Y Chen L-F Xi L Ma and JLee ldquoEmpirical analysis of support vector machine ensembleclassifiersrdquo Expert Systems with Applications vol 36 no 3 pp6466ndash6476 2009
[14] S Ozsen and S Gunes ldquoEffect of feature-type in selectingdistance measure for an artificial immune system as a patternrecognizerrdquoDigital Signal Processing vol 18 no 4 pp 635ndash6452008
[15] H Kahramanli and N Allahverdi ldquoDesign of a hybrid systemfor the diabetes and heart diseasesrdquo Expert Systems withApplications vol 35 no 1-2 pp 82ndash89 2008
[16] G Yan G Ma J Lv and B Song ldquoCombining independentcomponent analysis with support vector machinesrdquo in Pro-ceedings of the in 1st International Symposium on Systems andControl in Aerospace and Astronautics (ISSCAA rsquo06) pp 493ndash496 Harbin China January 2006
[17] S Sahan K Polat H Kodaz and S Gunes ldquoThe medicalapplications of attribute weighted artificial immune system(AWAIS) diagnosis of heart and diabetes diseasesrdquo ArtificialImmune Systems vol 3627 pp 456ndash468 2005
[18] W Duch R Adamczak and K Grabczewski ldquoA new method-ology of extraction optimization and application of crisp and
fuzzy logical rulesrdquo IEEE Transactions on Neural Networks vol12 no 2 pp 277ndash306 2001
[19] M P McRae B Bozkurt C M Ballantyne et al ldquoCardiacScoreCard a diagnostic multivariate index assay system forpredicting a spectrumof cardiovascular diseaserdquoExpert Systemswith Applications vol 54 pp 136ndash147 2016
[20] M Robnik-Sikonja and I Kononenko ldquoTheoretical and empir-ical analysis of ReliefF and RReliefFrdquoMachine Learning vol 53no 1-2 pp 23ndash69 2003
[21] D Wettschereck D W Aha and T Mohri ldquoA review andempirical evaluation of feature weighting methods for a classof lazy learning algorithmsrdquo Artificial Intelligence Review vol11 no 1ndash5 pp 273ndash314 1997
[22] I Kononenko E Simec and M R Sikonja ldquoOvercomingthe myopia of inductive learning algorithms with RELIEFFrdquoApplied Intelligence vol 7 no 1 pp 39ndash55 1997
[23] K Kira and L Rendell ldquoA practical approach to featureselectionrdquo in Proceedings of the International Conference onMachine Learning pp 249ndash256 Morgan Kaufmann AberdeenScotland 1992
[24] I Kononenko ldquoEstimating attributes analysis and extension ofReliefFrdquo in Machine Learning ECML-94 European ConferenceonMachine Learning Catania Italy April 6ndash8 1994 Proceedingsvol 784 of Lecture Notes in Computer Science pp 171ndash182Springer Berlin Germany 1994
[25] R Ruiz J C Riquelme and J S Aguilar-Ruiz ldquoHeuristicsearch over a ranking for feature selectionrdquo in Proceedings of8th InternationalWork-Conference onArtificial NeuralNetworks(IWANN rsquo05) Barcelona Spain June 2005 vol 3512 of LecturesNotes in Computer Science pp 742ndash749 Springer Berlin Ger-many 2005
[26] N Spolaor E A ChermanM CMonard andHD Lee ldquoFilterapproach feature selection methods to support multi-labellearning based on ReliefF and information gainrdquo in Advancesin Artificial IntelligencemdashSBIA 2012 21th Brazilian Symposiumon Artificial Intelligence Curitiba Brazil October 20ndash25 2012Proceedings vol 7589 of Lectures Notes in Computer Science pp72ndash81 Springer Berlin Germany 2012
[27] R Ruiz J C Riquelme and J S Aguilar-Ruiz ldquoFast featureranking algorithmrdquo in Proceedings of the Knowledge-BasedIntelligent Information and Engineering Systems (KES rsquo03) pp325ndash331 Springer Berlin Germany 2003
[28] V Jovanoski and N Lavrac ldquoFeature subset selection inassociation rules learning systemsrdquo in Proceedings of AnalysisWarehousing and Mining the Data pp 74ndash77 1999
[29] J J Liu and J T Y Kwok ldquoAn extended genetic rule inductionalgorithmrdquo in Proceedings of the Congress on EvolutionaryComputation pp 458ndash463 LA Jolla Calif USA July 2000
[30] L-X Zhang J-X Wang Y-N Zhao and Z-H Yang ldquoA novelhybrid feature selection algorithm using ReliefF estimationfor GA-Wrapper searchrdquo in Proceedings of the InternationalConference onMachine Learning andCybernetics vol 1 pp 380ndash384 IEEE Xirsquoan China November 2003
[31] Z Zhao L Wang and H Liu ldquoEfficient spectral featureselectionwithminimum redundancyrdquo in Proceedings of the 24thAAAI Conference on Artificial Intelligence AAAI Atlanta GaUSA 2010
[32] F Nie H Huang X Cai and C H Ding ldquoEfficient androbust feature selection via joint ℓ21-norms minimizationrdquo inAdvances in Neural Information Processing Systems pp 1813ndash1821 MIT Press 2010
Computational and Mathematical Methods in Medicine 11
[33] S-Y Jiang and L-X Wang ldquoEfficient feature selection based oncorrelation measure between continuous and discrete featuresrdquoInformation Processing Letters vol 116 no 2 pp 203ndash215 2016
[34] M J Kearns and U V Vazirani An Introduction to Compu-tational Learning Theory MIT Press Cambridge Mass USA1994
[35] R Bellman Adaptive Control Processes A Guided Tour RANDCorporation Research Studies Princeton University PressPrinceton NJ USA 1961
[36] Z Pawlak ldquoRough set theory and its applications to dataanalysisrdquo Cybernetics and Systems vol 29 no 7 pp 661ndash6881998
[37] A-E Hassanien ldquoRough set approach for attribute reductionand rule generation a case of patients with suspected breastcancerrdquo Journal of the American Society for Information Scienceand Technology vol 55 no 11 pp 954ndash962 2004
[38] A Skowron and C Rauszer ldquoThe discernibility matricesand functions in information systemsrdquo in Intelligent DecisionSupport-Handbook of Applications and Advances of the RoughSetsTheory SystemTheory Knowledge Engineering and ProblemSolving R Słowinski Ed vol 11 pp 331ndash362 KluwerAcademicDordrecht Netherlands 1992
[39] Z Pawlak ldquoRough set approach to knowledge-based decisionsupportrdquo European Journal of Operational Research vol 99 no1 pp 48ndash57 1997
[40] X Wang J Yang X Teng W Xia and R Jensen ldquoFeatureselection based on rough sets and particle swarm optimizationrdquoPattern Recognition Letters vol 28 no 4 pp 459ndash471 2007
[41] C Ma J Ouyang H-L Chen and X-H Zhao ldquoAn efficientdiagnosis system for Parkinsonrsquos disease using kernel-basedextreme learning machine with subtractive clustering featuresweighting approachrdquo Computational and Mathematical Meth-ods in Medicine vol 2014 Article ID 985789 14 pages 2014
[42] A H El-Baz ldquoHybrid intelligent system-based rough set andensemble classifier for breast cancer diagnosisrdquoNeural Comput-ing and Applications vol 26 no 2 pp 437ndash446 2015
[43] J-H Eom S-CKim andB-T Zhang ldquoAptaCDSS-E a classifierensemble-based clinical decision support system for cardiovas-cular disease level predictionrdquo Expert Systems with Applicationsvol 34 no 4 pp 2465ndash2479 2008
[44] Y Ma Ensemble Machine Learning Methods and ApplicationsSpringer New York NY USA 2012
[45] S A Etemad and A Arya ldquoClassification and translation ofstyle and affect in human motion using RBF neural networksrdquoNeurocomputing vol 129 pp 585ndash595 2014
[46] Z Pawlak ldquoRough setsrdquo International Journal of Computer ampInformation Sciences vol 11 no 5 pp 341ndash356 1982
[47] Z Pawlak ldquoRough sets and intelligent data analysisrdquo Informa-tion Sciences vol 147 no 1ndash4 pp 1ndash12 2002
[48] Y Huang P J McCullagh and N D Black ldquoAn optimization ofReliefF for classification in large datasetsrdquoData and KnowledgeEngineering vol 68 no 11 pp 1348ndash1356 2009
[49] J Sun M-Y Jia and H Li ldquoAdaBoost ensemble for financialdistress prediction an empirical comparison with data fromChinese listed companiesrdquo Expert Systems with Applicationsvol 38 no 8 pp 9305ndash9312 2011
[50] R Kohavi and F Provost ldquoGlossary of termsrdquo Machine Learn-ing vol 30 no 2-3 pp 271ndash274 1998
[51] W Chen and H Lin ldquoPrediction of midbody centrosome andkinetochore proteins based on gene ontology informationrdquoBiochemical and Biophysical Research Communications vol 401no 3 pp 382ndash384 2010
[52] K-C Chou and C-T Zhang ldquoPrediction of protein structuralclassesrdquo Critical Reviews in Biochemistry andMolecular Biologyvol 30 no 4 pp 275ndash349 1995
[53] W Chen and H Lin ldquoIdentification of voltage-gated potassiumchannel subfamilies from sequence information using supportvectormachinerdquoComputers in Biology andMedicine vol 42 no4 pp 504ndash507 2012
[54] K-C Chou and H-B Shen ldquoRecent progress in proteinsubcellular location predictionrdquo Analytical Biochemistry vol370 no 1 pp 1ndash16 2007
[55] P Feng H Lin W Chen and Y Zuo ldquoPredicting the typesof J-proteins using clustered amino acidsrdquo BioMed ResearchInternational vol 2014 Article ID 935719 8 pages 2014
[56] K-C Chou and H-B Shen ldquoProtIdent a web server foridentifying proteases and their types by fusing functionaldomain and sequential evolution informationrdquoBiochemical andBiophysical Research Communications vol 376 no 2 pp 321ndash325 2008
[57] K-C Chou and Y-D Cai ldquoPrediction of membrane proteintypes by incorporating amphipathic effectsrdquo Journal of ChemicalInformation and Modeling vol 45 no 2 pp 407ndash413 2005
[58] F Tom ldquoROC graphs notes and practical considerations forresearchersrdquo Machine Learning vol 31 pp 1ndash38 2004
[59] JHuang andCX Ling ldquoUsingAUCand accuracy in evaluatinglearning algorithmsrdquo IEEE Transactions on Knowledge andDataEngineering vol 17 no 3 pp 299ndash310 2005
[60] F D Foresee and M T Hagan ldquoGauss-Newton approximationto Bayesian learningrdquo in Proceedings of the International Con-ference on Neural Networks vol 3 pp 1930ndash1935 Houston TexUSA 1997
6 Computational and Mathematical Methods in Medicine
Heuristic RS reduction algorithmInput a decision table 119878 = (119880 119862119863) 119862 = 1198881 1198882 119888119898119863 = 1198891 1198892 119889119899Output RedStep 1 Return Core(1) Corelarr (2) For 119894 = 1 to119898(3) Select 119888119894 from 119862(4) Calculate Ind(119862) Ind(119862 minus 119888119894) and Ind(119863)(5) Calculate pos119862(119863) pos119862minus119888119894 (119863) and 119903119862(119863)(6) Calculate Sig(119886119894) Sig(119886119894) = 119903119862(119863) minus 119903119862minus119888119894 (119863)(7) If sig(119888119894) = 0(8) core = core cup 119888119894(9) End if(10) End forStep 2 Return Red(1) Red = Core(2) 1198621015840 = 119862 minus Red(3) While Sig(Red119863) = Sig(119862119863) do
Compute the weight of each feature 119888 in 1198621015840 using the ReliefF algorithmSelect a feature 119888 according to its weight let Red=Red cup 119888Initialize all the necessary parameters for the GA-based search engine according to theresults of the last step and search for satisfactory reducts
End while
Algorithm 2 Pseudocode of heuristic RS reduction algorithm
are classified into the healthy class TN is the number of truenegatives representing healthy cases that are correctly classi-fied into the healthy class Finally FP is the number of falsepositives representing the healthy cases that are incorrectlyclassified into the heart disease class [50]
The performance of the proposed system was evaluatedbased on sensitivity specificity and accuracy tests which usethe true positive (TP) true negative (TN) false negative (FN)and false positive (FP) terms [33]These criteria are calculatedas follows [41]
Sensitivity (Sn) = TPTP + FN
times 100
Specificity (Sp) = TNFP + TN
times 100
Accuracy (Acc) = TP + TNTP + TN + FP + FN
times 100
(8)
422 Cross-Validation Three cross-validation methodsnamely subsampling tests independent dataset tests andjackknife tests are often employed to evaluate the predictivecapability of a predictor [51] Among the three methodsthe jackknife test is deemed the least arbitrary and the mostobjective and rigorous [52 53] because it always yields aunique outcome as demonstrated by a penetrating analysis ina recent comprehensive review [54 55] Therefore thejackknife test has been widely and increasingly adopted inmany areas [56 57]
Accordingly the jackknife test was employed to examinethe performance of the model proposed in this paper Forjackknife cross-validation each sequence in the training
dataset is in turn singled out as an independent test sampleand all the parameter rules are calculated based on theremaining samples without including the one being treatedas the test sample
423 Receiver Operating Characteristics (ROC) The receiveroperating characteristic (ROC) curve is used for analyzing theprediction performance of a predictor [58] It is usually plot-ted using the true positive rate versus the false positive rate asthe discrimination threshold of classification algorithm isvaried The area under the ROC curve (AUC) is widely usedand relatively accepted in classification studies because itprovides a good summary of a classifierrsquos performance [59]
43 Results and Discussion
431 Results and Analysis on the Statlog (Heart) DatasetFirst we used the equal interval binningmethod to discretizethe original data In the feature extraction module the num-ber of k-nearest neighbors in the ReliefF algorithm was set to10 and the threshold 120575 was set to 002 Table 3 summarizesthe results of the ReliefF algorithm Based on these resultsC5and C6 were removed In Module 3 we obtained 15 reductsusing the heuristic RS reduction algorithm implemented inMATLAB 2014a
Trials were conducted using 70ndash30 training-test par-titions using all the reduced feature sets Jackknife cross-validation was performed on the dataset The number ofdesired base classifiers k was set to 50 100 and 150 Thecalculations were run 10 times and the highest classification
Computational and Mathematical Methods in Medicine 7
Table 1 Feature information of Statlog (Heart) dataset
Feature Code Description Domain Data type Mean Standard deviationAge 1198621 mdash 29ndash77 Real 54 9Sex 1198622 Male female 0 1 Binary mdash mdash
Chest pain type 1198623Angina
asymptomaticabnormal
1 2 3 4 Nominal mdash mdash
Resting blood pressure 1198624 mdash 94ndash200 Real 131344 17862Serum cholesterol in mgdl 1198625 mdash 126ndash564 Real 249659 51686Fasting blood sugar gt 120mgdl 1198626 mdash 0 1 Binary mdash mdash
Resting electrocardiographic results 1198627Norm
abnormal hyper 0 1 2 Nominal mdash mdash
Maximum heart rate achieved 1198628 mdash 71ndash202 Real 149678 231666Exercise-induced angina 1198629 mdash 0 1 Binary mdash mdashOld peak = ST depression induced by exercise relative to rest 11986210 mdash 0ndash62 Real 105 1145Slope of the peak exercise ST segment 11986211 Up flat down 1 2 3 Ordered mdash mdashNumber of major vessels (0ndash3) colored by fluoroscopy 11986212 mdash 0 1 2 3 Real mdash mdash
Thal 11986213Normal fixed
defectreversible defect
3 6 7 Nominal mdash mdash
Table 2 The confusion matrix
Predicted patientswith heart disease
Predicted healthypersons
Actual patients withheart disease True positive (TP) False negative (FN)
Actual healthypersons False positive (FP) True negative (TN)
performances for each training-test partition are provided inTable 4
In Table 4 1198772 obtains the best test set classificationaccuracy (9259) using the ensemble classifiers when 119896 =100 The training process is shown in Figure 2 The trainingand test ROC curves are shown in Figure 3
432 Comparison with Other Classifiers In this sectionour ensemble classification method is compared with theindividual C45 decision tree and Naıve Bayes and BayesianNeural Networks (BNN)methodsTheC45 decision tree andNaıve Bayes are common classifiers Bayesian Neural Net-works (BNN) is a classifier that uses Bayesian regularizationto train feed-forward neural networks [60] and has betterperformance than pure neural networks The classificationaccuracy results of the four classifiers are listed in Table 5Theensemble classification method has better performance thanthe individual C45 classifier and the other two classifiers
433 Comparison of the Results with Other Studies Wecompared our results with the results of other studies Table 6shows the classification accuracies of our study and previousmethods
0 10 20 30 40 50 60 70 80 90 100008
01
012
014
016
018
02
022
024
Number of trees
Test
class
ifica
tion
erro
r
Figure 2 Training process of 1198777
The results show that our proposed method obtainssuperior and promising results in classifying heart diseasepatients We believe that the proposed RFRS-based classi-fication system can be exceedingly beneficial in assistingphysicians in making accurate decisions
8 Computational and Mathematical Methods in Medicine
In this paper a novel ReliefF and Rough Set- (RFRS-)based classification system is proposed for heart diseasediagnosisThemain novelty of this paper lies in the proposedapproach the combination of the ReliefF and RS methodsto classify heart disease problems in an efficient and fastmanner The RFRS classification system consists of twosubsystems the RFRS feature selection subsystem and theclassification subsystemThe Statlog (Heart) dataset from theUCI machine learning database [3] was selected to test thesystemThe experimental results show that the reduct1198772 (11986211198623 1198627 1198628 11986211 11986212 11986213) achieves the highest classificationaccuracy (9259) using an ensemble classifier with the C45decision tree as the weak learner The results also show thatthe RFRS method has superior performance compared to
three common classifiers in terms of ACC sensitivity andspecificity In addition the performance of the proposedsystem is superior to that of existingmethods in the literatureBased on empirical analysis the results indicate that theproposed classification system can be used as a promisingalternative tool in medical decision making for heart diseasediagnosis
However the proposed method also has some weak-nesses The number of the nearest neighbors (119896) and theweight threshold (120579) are not stable in the ReliefF algorithm[20] One solution to this problem is to compute estimatesfor all possible numbers and take the highest estimate of eachfeature as the final result [20] We need to perform moreexperiments to find the optimal parameter values for theReliefF algorithm in the future
Computational and Mathematical Methods in Medicine 9
Table 6 Comparison of our results with those of other studies
Author Method Classification accuracy ()Our study RFRS classification system 9259Lee [4] Graphical characteristics of BSWFM combined with Euclidean distance 874Tomar and Agarwal [5] Feature selection-based LSTSVM 8559Buscema et al [6] TWIST algorithm 8414Subbulakshmi et al [7] ELM 875Karegowda et al [8] GA + Naıve Bayes 8587Srinivas et al [9] Naıve Bayes 8370Polat and Gunes [10] RBF kernel 119865-score + LS-SVM 8370Ozsen and Gunes [11] GA-AWAIS 8743Helmy and Rasheed [12] Algebraic Sigmoid 8524Wang et al [13] Linear kernel SVM classifiers 8337Ozsen and Gunes [14] Hybrid similarity measure 8395Kahramanli and Allahverdi [15] Hybrid neural network method 868Yan et al [16] ICA + SVM 8375Sahan et al [17] AWAIS 8259Duch et al [18] KNN classifier 856BSWFM bounded sumof weighted fuzzymembership functions LSTSVM Least Square Twin Support VectorMachine TWIST Training with Input Selectionand Testing ELM Extreme LearningMachine GA genetic algorithm SVM support vectormachine ICA imperialist competitive algorithm AWAIS attributeweighted artificial immune system119870NN 119896-nearest neighbor
Test ROC
0 10
01
02
02
03
04
04
05
06
06
07
08
08
09
1
False positive rate
True
pos
itive
rate
0 10
01
02
02
03
04
04
05
06
06
07
08
08
09
1
False positive rate
True
pos
itive
rate
Training ROC
Figure 3 ROC curves for training and test sets
Competing Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
This study was supported by the National Natural ScienceFoundation of China (Grant no 71432007)
References
[1] K D Kochanek J Xu S L Murphy A M Minino and H-C Kung ldquoDeaths final data for 2009rdquo National Vital StatisticsReports vol 60 no 3 pp 1ndash116 2011
[2] H Temurtas N Yumusak and F Temurtas ldquoA comparativestudy on diabetes disease diagnosis using neural networksrdquoExpert Systems with Applications vol 36 no 4 pp 8610ndash86152009
10 Computational and Mathematical Methods in Medicine
[3] UCI Repository of Machine Learning Databases httparchiveicsuciedumldatasetsStatlog+28Heart29
[4] S-H Lee ldquoFeature selection based on the center of gravity ofBSWFMs using NEWFMrdquo Engineering Applications of ArtificialIntelligence vol 45 pp 482ndash487 2015
[5] D Tomar and S Agarwal ldquoFeature selection based least squaretwin support vector machine for diagnosis of heart diseaserdquoInternational Journal of Bio-Science and Bio-Technology vol 6no 2 pp 69ndash82 2014
[6] M Buscema M Breda and W Lodwick ldquoTraining with InputSelection and Testing (TWIST) algorithm a significant advancein pattern recognition performance of machine learningrdquoJournal of Intelligent Learning Systems and Applications vol 5no 1 pp 29ndash38 2013
[7] C V Subbulakshmi S N Deepa and N Malathi ldquoExtremelearning machine for two category data classificationrdquo inProceedings of the IEEE International Conference on AdvancedCommunication Control and Computing Technologies (ICAC-CCT rsquo12) pp 458ndash461 Ramanathapuram India August 2012
[8] A G Karegowda A SManjunath andM A Jayaram ldquoFeaturesubset selection problem using wrapper approach in supervisedlearningrdquo International Journal of Computer Applications vol 1no 7 pp 13ndash17 2010
[9] K Srinivas B K Rani and A Govrdhan ldquoApplications ofdata mining techniques in healthcare and prediction of heartattacksrdquo International Journal on Computer Science and Engi-neering vol 2 pp 250ndash255 2010
[10] K Polat and S Gunes ldquoA new feature selection methodon classification of medical datasets kernel F-score featureselectionrdquo Expert Systems with Applications vol 36 no 7 pp10367ndash10373 2009
[11] S Ozsen and S Gunes ldquoAttribute weighting via genetic algo-rithms for attributeweighted artificial immune system (AWAIS)and its application to heart disease and liver disorders prob-lemsrdquo Expert Systems with Applications vol 36 no 1 pp 386ndash392 2009
[12] T Helmy and Z Rasheed ldquoMulti-category bioinformaticsdataset classification using extreme learning machinerdquo in Pro-ceedings of the IEEE Congress on Evolutionary Computation(CEC rsquo09) pp 3234ndash3240 Trondheim Norway May 2009
[13] S-J Wang A Mathew Y Chen L-F Xi L Ma and JLee ldquoEmpirical analysis of support vector machine ensembleclassifiersrdquo Expert Systems with Applications vol 36 no 3 pp6466ndash6476 2009
[14] S Ozsen and S Gunes ldquoEffect of feature-type in selectingdistance measure for an artificial immune system as a patternrecognizerrdquoDigital Signal Processing vol 18 no 4 pp 635ndash6452008
[15] H Kahramanli and N Allahverdi ldquoDesign of a hybrid systemfor the diabetes and heart diseasesrdquo Expert Systems withApplications vol 35 no 1-2 pp 82ndash89 2008
[16] G Yan G Ma J Lv and B Song ldquoCombining independentcomponent analysis with support vector machinesrdquo in Pro-ceedings of the in 1st International Symposium on Systems andControl in Aerospace and Astronautics (ISSCAA rsquo06) pp 493ndash496 Harbin China January 2006
[17] S Sahan K Polat H Kodaz and S Gunes ldquoThe medicalapplications of attribute weighted artificial immune system(AWAIS) diagnosis of heart and diabetes diseasesrdquo ArtificialImmune Systems vol 3627 pp 456ndash468 2005
[18] W Duch R Adamczak and K Grabczewski ldquoA new method-ology of extraction optimization and application of crisp and
fuzzy logical rulesrdquo IEEE Transactions on Neural Networks vol12 no 2 pp 277ndash306 2001
[19] M P McRae B Bozkurt C M Ballantyne et al ldquoCardiacScoreCard a diagnostic multivariate index assay system forpredicting a spectrumof cardiovascular diseaserdquoExpert Systemswith Applications vol 54 pp 136ndash147 2016
[20] M Robnik-Sikonja and I Kononenko ldquoTheoretical and empir-ical analysis of ReliefF and RReliefFrdquoMachine Learning vol 53no 1-2 pp 23ndash69 2003
[21] D Wettschereck D W Aha and T Mohri ldquoA review andempirical evaluation of feature weighting methods for a classof lazy learning algorithmsrdquo Artificial Intelligence Review vol11 no 1ndash5 pp 273ndash314 1997
[22] I Kononenko E Simec and M R Sikonja ldquoOvercomingthe myopia of inductive learning algorithms with RELIEFFrdquoApplied Intelligence vol 7 no 1 pp 39ndash55 1997
[23] K Kira and L Rendell ldquoA practical approach to featureselectionrdquo in Proceedings of the International Conference onMachine Learning pp 249ndash256 Morgan Kaufmann AberdeenScotland 1992
[24] I Kononenko ldquoEstimating attributes analysis and extension ofReliefFrdquo in Machine Learning ECML-94 European ConferenceonMachine Learning Catania Italy April 6ndash8 1994 Proceedingsvol 784 of Lecture Notes in Computer Science pp 171ndash182Springer Berlin Germany 1994
[25] R Ruiz J C Riquelme and J S Aguilar-Ruiz ldquoHeuristicsearch over a ranking for feature selectionrdquo in Proceedings of8th InternationalWork-Conference onArtificial NeuralNetworks(IWANN rsquo05) Barcelona Spain June 2005 vol 3512 of LecturesNotes in Computer Science pp 742ndash749 Springer Berlin Ger-many 2005
[26] N Spolaor E A ChermanM CMonard andHD Lee ldquoFilterapproach feature selection methods to support multi-labellearning based on ReliefF and information gainrdquo in Advancesin Artificial IntelligencemdashSBIA 2012 21th Brazilian Symposiumon Artificial Intelligence Curitiba Brazil October 20ndash25 2012Proceedings vol 7589 of Lectures Notes in Computer Science pp72ndash81 Springer Berlin Germany 2012
[27] R Ruiz J C Riquelme and J S Aguilar-Ruiz ldquoFast featureranking algorithmrdquo in Proceedings of the Knowledge-BasedIntelligent Information and Engineering Systems (KES rsquo03) pp325ndash331 Springer Berlin Germany 2003
[28] V Jovanoski and N Lavrac ldquoFeature subset selection inassociation rules learning systemsrdquo in Proceedings of AnalysisWarehousing and Mining the Data pp 74ndash77 1999
[29] J J Liu and J T Y Kwok ldquoAn extended genetic rule inductionalgorithmrdquo in Proceedings of the Congress on EvolutionaryComputation pp 458ndash463 LA Jolla Calif USA July 2000
[30] L-X Zhang J-X Wang Y-N Zhao and Z-H Yang ldquoA novelhybrid feature selection algorithm using ReliefF estimationfor GA-Wrapper searchrdquo in Proceedings of the InternationalConference onMachine Learning andCybernetics vol 1 pp 380ndash384 IEEE Xirsquoan China November 2003
[31] Z Zhao L Wang and H Liu ldquoEfficient spectral featureselectionwithminimum redundancyrdquo in Proceedings of the 24thAAAI Conference on Artificial Intelligence AAAI Atlanta GaUSA 2010
[32] F Nie H Huang X Cai and C H Ding ldquoEfficient androbust feature selection via joint ℓ21-norms minimizationrdquo inAdvances in Neural Information Processing Systems pp 1813ndash1821 MIT Press 2010
Computational and Mathematical Methods in Medicine 11
[33] S-Y Jiang and L-X Wang ldquoEfficient feature selection based oncorrelation measure between continuous and discrete featuresrdquoInformation Processing Letters vol 116 no 2 pp 203ndash215 2016
[34] M J Kearns and U V Vazirani An Introduction to Compu-tational Learning Theory MIT Press Cambridge Mass USA1994
[35] R Bellman Adaptive Control Processes A Guided Tour RANDCorporation Research Studies Princeton University PressPrinceton NJ USA 1961
[36] Z Pawlak ldquoRough set theory and its applications to dataanalysisrdquo Cybernetics and Systems vol 29 no 7 pp 661ndash6881998
[37] A-E Hassanien ldquoRough set approach for attribute reductionand rule generation a case of patients with suspected breastcancerrdquo Journal of the American Society for Information Scienceand Technology vol 55 no 11 pp 954ndash962 2004
[38] A Skowron and C Rauszer ldquoThe discernibility matricesand functions in information systemsrdquo in Intelligent DecisionSupport-Handbook of Applications and Advances of the RoughSetsTheory SystemTheory Knowledge Engineering and ProblemSolving R Słowinski Ed vol 11 pp 331ndash362 KluwerAcademicDordrecht Netherlands 1992
[39] Z Pawlak ldquoRough set approach to knowledge-based decisionsupportrdquo European Journal of Operational Research vol 99 no1 pp 48ndash57 1997
[40] X Wang J Yang X Teng W Xia and R Jensen ldquoFeatureselection based on rough sets and particle swarm optimizationrdquoPattern Recognition Letters vol 28 no 4 pp 459ndash471 2007
[41] C Ma J Ouyang H-L Chen and X-H Zhao ldquoAn efficientdiagnosis system for Parkinsonrsquos disease using kernel-basedextreme learning machine with subtractive clustering featuresweighting approachrdquo Computational and Mathematical Meth-ods in Medicine vol 2014 Article ID 985789 14 pages 2014
[42] A H El-Baz ldquoHybrid intelligent system-based rough set andensemble classifier for breast cancer diagnosisrdquoNeural Comput-ing and Applications vol 26 no 2 pp 437ndash446 2015
[43] J-H Eom S-CKim andB-T Zhang ldquoAptaCDSS-E a classifierensemble-based clinical decision support system for cardiovas-cular disease level predictionrdquo Expert Systems with Applicationsvol 34 no 4 pp 2465ndash2479 2008
[44] Y Ma Ensemble Machine Learning Methods and ApplicationsSpringer New York NY USA 2012
[45] S A Etemad and A Arya ldquoClassification and translation ofstyle and affect in human motion using RBF neural networksrdquoNeurocomputing vol 129 pp 585ndash595 2014
[46] Z Pawlak ldquoRough setsrdquo International Journal of Computer ampInformation Sciences vol 11 no 5 pp 341ndash356 1982
[47] Z Pawlak ldquoRough sets and intelligent data analysisrdquo Informa-tion Sciences vol 147 no 1ndash4 pp 1ndash12 2002
[48] Y Huang P J McCullagh and N D Black ldquoAn optimization ofReliefF for classification in large datasetsrdquoData and KnowledgeEngineering vol 68 no 11 pp 1348ndash1356 2009
[49] J Sun M-Y Jia and H Li ldquoAdaBoost ensemble for financialdistress prediction an empirical comparison with data fromChinese listed companiesrdquo Expert Systems with Applicationsvol 38 no 8 pp 9305ndash9312 2011
[50] R Kohavi and F Provost ldquoGlossary of termsrdquo Machine Learn-ing vol 30 no 2-3 pp 271ndash274 1998
[51] W Chen and H Lin ldquoPrediction of midbody centrosome andkinetochore proteins based on gene ontology informationrdquoBiochemical and Biophysical Research Communications vol 401no 3 pp 382ndash384 2010
[52] K-C Chou and C-T Zhang ldquoPrediction of protein structuralclassesrdquo Critical Reviews in Biochemistry andMolecular Biologyvol 30 no 4 pp 275ndash349 1995
[53] W Chen and H Lin ldquoIdentification of voltage-gated potassiumchannel subfamilies from sequence information using supportvectormachinerdquoComputers in Biology andMedicine vol 42 no4 pp 504ndash507 2012
[54] K-C Chou and H-B Shen ldquoRecent progress in proteinsubcellular location predictionrdquo Analytical Biochemistry vol370 no 1 pp 1ndash16 2007
[55] P Feng H Lin W Chen and Y Zuo ldquoPredicting the typesof J-proteins using clustered amino acidsrdquo BioMed ResearchInternational vol 2014 Article ID 935719 8 pages 2014
[56] K-C Chou and H-B Shen ldquoProtIdent a web server foridentifying proteases and their types by fusing functionaldomain and sequential evolution informationrdquoBiochemical andBiophysical Research Communications vol 376 no 2 pp 321ndash325 2008
[57] K-C Chou and Y-D Cai ldquoPrediction of membrane proteintypes by incorporating amphipathic effectsrdquo Journal of ChemicalInformation and Modeling vol 45 no 2 pp 407ndash413 2005
[58] F Tom ldquoROC graphs notes and practical considerations forresearchersrdquo Machine Learning vol 31 pp 1ndash38 2004
[59] JHuang andCX Ling ldquoUsingAUCand accuracy in evaluatinglearning algorithmsrdquo IEEE Transactions on Knowledge andDataEngineering vol 17 no 3 pp 299ndash310 2005
[60] F D Foresee and M T Hagan ldquoGauss-Newton approximationto Bayesian learningrdquo in Proceedings of the International Con-ference on Neural Networks vol 3 pp 1930ndash1935 Houston TexUSA 1997
Computational and Mathematical Methods in Medicine 7
Table 1 Feature information of Statlog (Heart) dataset
Feature Code Description Domain Data type Mean Standard deviationAge 1198621 mdash 29ndash77 Real 54 9Sex 1198622 Male female 0 1 Binary mdash mdash
Chest pain type 1198623Angina
asymptomaticabnormal
1 2 3 4 Nominal mdash mdash
Resting blood pressure 1198624 mdash 94ndash200 Real 131344 17862Serum cholesterol in mgdl 1198625 mdash 126ndash564 Real 249659 51686Fasting blood sugar gt 120mgdl 1198626 mdash 0 1 Binary mdash mdash
Resting electrocardiographic results 1198627Norm
abnormal hyper 0 1 2 Nominal mdash mdash
Maximum heart rate achieved 1198628 mdash 71ndash202 Real 149678 231666Exercise-induced angina 1198629 mdash 0 1 Binary mdash mdashOld peak = ST depression induced by exercise relative to rest 11986210 mdash 0ndash62 Real 105 1145Slope of the peak exercise ST segment 11986211 Up flat down 1 2 3 Ordered mdash mdashNumber of major vessels (0ndash3) colored by fluoroscopy 11986212 mdash 0 1 2 3 Real mdash mdash
Thal 11986213Normal fixed
defectreversible defect
3 6 7 Nominal mdash mdash
Table 2 The confusion matrix
Predicted patientswith heart disease
Predicted healthypersons
Actual patients withheart disease True positive (TP) False negative (FN)
Actual healthypersons False positive (FP) True negative (TN)
performances for each training-test partition are provided inTable 4
In Table 4 1198772 obtains the best test set classificationaccuracy (9259) using the ensemble classifiers when 119896 =100 The training process is shown in Figure 2 The trainingand test ROC curves are shown in Figure 3
432 Comparison with Other Classifiers In this sectionour ensemble classification method is compared with theindividual C45 decision tree and Naıve Bayes and BayesianNeural Networks (BNN)methodsTheC45 decision tree andNaıve Bayes are common classifiers Bayesian Neural Net-works (BNN) is a classifier that uses Bayesian regularizationto train feed-forward neural networks [60] and has betterperformance than pure neural networks The classificationaccuracy results of the four classifiers are listed in Table 5Theensemble classification method has better performance thanthe individual C45 classifier and the other two classifiers
433 Comparison of the Results with Other Studies Wecompared our results with the results of other studies Table 6shows the classification accuracies of our study and previousmethods
0 10 20 30 40 50 60 70 80 90 100008
01
012
014
016
018
02
022
024
Number of trees
Test
class
ifica
tion
erro
r
Figure 2 Training process of 1198777
The results show that our proposed method obtainssuperior and promising results in classifying heart diseasepatients We believe that the proposed RFRS-based classi-fication system can be exceedingly beneficial in assistingphysicians in making accurate decisions
8 Computational and Mathematical Methods in Medicine
In this paper a novel ReliefF and Rough Set- (RFRS-)based classification system is proposed for heart diseasediagnosisThemain novelty of this paper lies in the proposedapproach the combination of the ReliefF and RS methodsto classify heart disease problems in an efficient and fastmanner The RFRS classification system consists of twosubsystems the RFRS feature selection subsystem and theclassification subsystemThe Statlog (Heart) dataset from theUCI machine learning database [3] was selected to test thesystemThe experimental results show that the reduct1198772 (11986211198623 1198627 1198628 11986211 11986212 11986213) achieves the highest classificationaccuracy (9259) using an ensemble classifier with the C45decision tree as the weak learner The results also show thatthe RFRS method has superior performance compared to
three common classifiers in terms of ACC sensitivity andspecificity In addition the performance of the proposedsystem is superior to that of existingmethods in the literatureBased on empirical analysis the results indicate that theproposed classification system can be used as a promisingalternative tool in medical decision making for heart diseasediagnosis
However the proposed method also has some weak-nesses The number of the nearest neighbors (119896) and theweight threshold (120579) are not stable in the ReliefF algorithm[20] One solution to this problem is to compute estimatesfor all possible numbers and take the highest estimate of eachfeature as the final result [20] We need to perform moreexperiments to find the optimal parameter values for theReliefF algorithm in the future
Computational and Mathematical Methods in Medicine 9
Table 6 Comparison of our results with those of other studies
Author Method Classification accuracy ()Our study RFRS classification system 9259Lee [4] Graphical characteristics of BSWFM combined with Euclidean distance 874Tomar and Agarwal [5] Feature selection-based LSTSVM 8559Buscema et al [6] TWIST algorithm 8414Subbulakshmi et al [7] ELM 875Karegowda et al [8] GA + Naıve Bayes 8587Srinivas et al [9] Naıve Bayes 8370Polat and Gunes [10] RBF kernel 119865-score + LS-SVM 8370Ozsen and Gunes [11] GA-AWAIS 8743Helmy and Rasheed [12] Algebraic Sigmoid 8524Wang et al [13] Linear kernel SVM classifiers 8337Ozsen and Gunes [14] Hybrid similarity measure 8395Kahramanli and Allahverdi [15] Hybrid neural network method 868Yan et al [16] ICA + SVM 8375Sahan et al [17] AWAIS 8259Duch et al [18] KNN classifier 856BSWFM bounded sumof weighted fuzzymembership functions LSTSVM Least Square Twin Support VectorMachine TWIST Training with Input Selectionand Testing ELM Extreme LearningMachine GA genetic algorithm SVM support vectormachine ICA imperialist competitive algorithm AWAIS attributeweighted artificial immune system119870NN 119896-nearest neighbor
Test ROC
0 10
01
02
02
03
04
04
05
06
06
07
08
08
09
1
False positive rate
True
pos
itive
rate
0 10
01
02
02
03
04
04
05
06
06
07
08
08
09
1
False positive rate
True
pos
itive
rate
Training ROC
Figure 3 ROC curves for training and test sets
Competing Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
This study was supported by the National Natural ScienceFoundation of China (Grant no 71432007)
References
[1] K D Kochanek J Xu S L Murphy A M Minino and H-C Kung ldquoDeaths final data for 2009rdquo National Vital StatisticsReports vol 60 no 3 pp 1ndash116 2011
[2] H Temurtas N Yumusak and F Temurtas ldquoA comparativestudy on diabetes disease diagnosis using neural networksrdquoExpert Systems with Applications vol 36 no 4 pp 8610ndash86152009
10 Computational and Mathematical Methods in Medicine
[3] UCI Repository of Machine Learning Databases httparchiveicsuciedumldatasetsStatlog+28Heart29
[4] S-H Lee ldquoFeature selection based on the center of gravity ofBSWFMs using NEWFMrdquo Engineering Applications of ArtificialIntelligence vol 45 pp 482ndash487 2015
[5] D Tomar and S Agarwal ldquoFeature selection based least squaretwin support vector machine for diagnosis of heart diseaserdquoInternational Journal of Bio-Science and Bio-Technology vol 6no 2 pp 69ndash82 2014
[6] M Buscema M Breda and W Lodwick ldquoTraining with InputSelection and Testing (TWIST) algorithm a significant advancein pattern recognition performance of machine learningrdquoJournal of Intelligent Learning Systems and Applications vol 5no 1 pp 29ndash38 2013
[7] C V Subbulakshmi S N Deepa and N Malathi ldquoExtremelearning machine for two category data classificationrdquo inProceedings of the IEEE International Conference on AdvancedCommunication Control and Computing Technologies (ICAC-CCT rsquo12) pp 458ndash461 Ramanathapuram India August 2012
[8] A G Karegowda A SManjunath andM A Jayaram ldquoFeaturesubset selection problem using wrapper approach in supervisedlearningrdquo International Journal of Computer Applications vol 1no 7 pp 13ndash17 2010
[9] K Srinivas B K Rani and A Govrdhan ldquoApplications ofdata mining techniques in healthcare and prediction of heartattacksrdquo International Journal on Computer Science and Engi-neering vol 2 pp 250ndash255 2010
[10] K Polat and S Gunes ldquoA new feature selection methodon classification of medical datasets kernel F-score featureselectionrdquo Expert Systems with Applications vol 36 no 7 pp10367ndash10373 2009
[11] S Ozsen and S Gunes ldquoAttribute weighting via genetic algo-rithms for attributeweighted artificial immune system (AWAIS)and its application to heart disease and liver disorders prob-lemsrdquo Expert Systems with Applications vol 36 no 1 pp 386ndash392 2009
[12] T Helmy and Z Rasheed ldquoMulti-category bioinformaticsdataset classification using extreme learning machinerdquo in Pro-ceedings of the IEEE Congress on Evolutionary Computation(CEC rsquo09) pp 3234ndash3240 Trondheim Norway May 2009
[13] S-J Wang A Mathew Y Chen L-F Xi L Ma and JLee ldquoEmpirical analysis of support vector machine ensembleclassifiersrdquo Expert Systems with Applications vol 36 no 3 pp6466ndash6476 2009
[14] S Ozsen and S Gunes ldquoEffect of feature-type in selectingdistance measure for an artificial immune system as a patternrecognizerrdquoDigital Signal Processing vol 18 no 4 pp 635ndash6452008
[15] H Kahramanli and N Allahverdi ldquoDesign of a hybrid systemfor the diabetes and heart diseasesrdquo Expert Systems withApplications vol 35 no 1-2 pp 82ndash89 2008
[16] G Yan G Ma J Lv and B Song ldquoCombining independentcomponent analysis with support vector machinesrdquo in Pro-ceedings of the in 1st International Symposium on Systems andControl in Aerospace and Astronautics (ISSCAA rsquo06) pp 493ndash496 Harbin China January 2006
[17] S Sahan K Polat H Kodaz and S Gunes ldquoThe medicalapplications of attribute weighted artificial immune system(AWAIS) diagnosis of heart and diabetes diseasesrdquo ArtificialImmune Systems vol 3627 pp 456ndash468 2005
[18] W Duch R Adamczak and K Grabczewski ldquoA new method-ology of extraction optimization and application of crisp and
fuzzy logical rulesrdquo IEEE Transactions on Neural Networks vol12 no 2 pp 277ndash306 2001
[19] M P McRae B Bozkurt C M Ballantyne et al ldquoCardiacScoreCard a diagnostic multivariate index assay system forpredicting a spectrumof cardiovascular diseaserdquoExpert Systemswith Applications vol 54 pp 136ndash147 2016
[20] M Robnik-Sikonja and I Kononenko ldquoTheoretical and empir-ical analysis of ReliefF and RReliefFrdquoMachine Learning vol 53no 1-2 pp 23ndash69 2003
[21] D Wettschereck D W Aha and T Mohri ldquoA review andempirical evaluation of feature weighting methods for a classof lazy learning algorithmsrdquo Artificial Intelligence Review vol11 no 1ndash5 pp 273ndash314 1997
[22] I Kononenko E Simec and M R Sikonja ldquoOvercomingthe myopia of inductive learning algorithms with RELIEFFrdquoApplied Intelligence vol 7 no 1 pp 39ndash55 1997
[23] K Kira and L Rendell ldquoA practical approach to featureselectionrdquo in Proceedings of the International Conference onMachine Learning pp 249ndash256 Morgan Kaufmann AberdeenScotland 1992
[24] I Kononenko ldquoEstimating attributes analysis and extension ofReliefFrdquo in Machine Learning ECML-94 European ConferenceonMachine Learning Catania Italy April 6ndash8 1994 Proceedingsvol 784 of Lecture Notes in Computer Science pp 171ndash182Springer Berlin Germany 1994
[25] R Ruiz J C Riquelme and J S Aguilar-Ruiz ldquoHeuristicsearch over a ranking for feature selectionrdquo in Proceedings of8th InternationalWork-Conference onArtificial NeuralNetworks(IWANN rsquo05) Barcelona Spain June 2005 vol 3512 of LecturesNotes in Computer Science pp 742ndash749 Springer Berlin Ger-many 2005
[26] N Spolaor E A ChermanM CMonard andHD Lee ldquoFilterapproach feature selection methods to support multi-labellearning based on ReliefF and information gainrdquo in Advancesin Artificial IntelligencemdashSBIA 2012 21th Brazilian Symposiumon Artificial Intelligence Curitiba Brazil October 20ndash25 2012Proceedings vol 7589 of Lectures Notes in Computer Science pp72ndash81 Springer Berlin Germany 2012
[27] R Ruiz J C Riquelme and J S Aguilar-Ruiz ldquoFast featureranking algorithmrdquo in Proceedings of the Knowledge-BasedIntelligent Information and Engineering Systems (KES rsquo03) pp325ndash331 Springer Berlin Germany 2003
[28] V Jovanoski and N Lavrac ldquoFeature subset selection inassociation rules learning systemsrdquo in Proceedings of AnalysisWarehousing and Mining the Data pp 74ndash77 1999
[29] J J Liu and J T Y Kwok ldquoAn extended genetic rule inductionalgorithmrdquo in Proceedings of the Congress on EvolutionaryComputation pp 458ndash463 LA Jolla Calif USA July 2000
[30] L-X Zhang J-X Wang Y-N Zhao and Z-H Yang ldquoA novelhybrid feature selection algorithm using ReliefF estimationfor GA-Wrapper searchrdquo in Proceedings of the InternationalConference onMachine Learning andCybernetics vol 1 pp 380ndash384 IEEE Xirsquoan China November 2003
[31] Z Zhao L Wang and H Liu ldquoEfficient spectral featureselectionwithminimum redundancyrdquo in Proceedings of the 24thAAAI Conference on Artificial Intelligence AAAI Atlanta GaUSA 2010
[32] F Nie H Huang X Cai and C H Ding ldquoEfficient androbust feature selection via joint ℓ21-norms minimizationrdquo inAdvances in Neural Information Processing Systems pp 1813ndash1821 MIT Press 2010
Computational and Mathematical Methods in Medicine 11
[33] S-Y Jiang and L-X Wang ldquoEfficient feature selection based oncorrelation measure between continuous and discrete featuresrdquoInformation Processing Letters vol 116 no 2 pp 203ndash215 2016
[34] M J Kearns and U V Vazirani An Introduction to Compu-tational Learning Theory MIT Press Cambridge Mass USA1994
[35] R Bellman Adaptive Control Processes A Guided Tour RANDCorporation Research Studies Princeton University PressPrinceton NJ USA 1961
[36] Z Pawlak ldquoRough set theory and its applications to dataanalysisrdquo Cybernetics and Systems vol 29 no 7 pp 661ndash6881998
[37] A-E Hassanien ldquoRough set approach for attribute reductionand rule generation a case of patients with suspected breastcancerrdquo Journal of the American Society for Information Scienceand Technology vol 55 no 11 pp 954ndash962 2004
[38] A Skowron and C Rauszer ldquoThe discernibility matricesand functions in information systemsrdquo in Intelligent DecisionSupport-Handbook of Applications and Advances of the RoughSetsTheory SystemTheory Knowledge Engineering and ProblemSolving R Słowinski Ed vol 11 pp 331ndash362 KluwerAcademicDordrecht Netherlands 1992
[39] Z Pawlak ldquoRough set approach to knowledge-based decisionsupportrdquo European Journal of Operational Research vol 99 no1 pp 48ndash57 1997
[40] X Wang J Yang X Teng W Xia and R Jensen ldquoFeatureselection based on rough sets and particle swarm optimizationrdquoPattern Recognition Letters vol 28 no 4 pp 459ndash471 2007
[41] C Ma J Ouyang H-L Chen and X-H Zhao ldquoAn efficientdiagnosis system for Parkinsonrsquos disease using kernel-basedextreme learning machine with subtractive clustering featuresweighting approachrdquo Computational and Mathematical Meth-ods in Medicine vol 2014 Article ID 985789 14 pages 2014
[42] A H El-Baz ldquoHybrid intelligent system-based rough set andensemble classifier for breast cancer diagnosisrdquoNeural Comput-ing and Applications vol 26 no 2 pp 437ndash446 2015
[43] J-H Eom S-CKim andB-T Zhang ldquoAptaCDSS-E a classifierensemble-based clinical decision support system for cardiovas-cular disease level predictionrdquo Expert Systems with Applicationsvol 34 no 4 pp 2465ndash2479 2008
[44] Y Ma Ensemble Machine Learning Methods and ApplicationsSpringer New York NY USA 2012
[45] S A Etemad and A Arya ldquoClassification and translation ofstyle and affect in human motion using RBF neural networksrdquoNeurocomputing vol 129 pp 585ndash595 2014
[46] Z Pawlak ldquoRough setsrdquo International Journal of Computer ampInformation Sciences vol 11 no 5 pp 341ndash356 1982
[47] Z Pawlak ldquoRough sets and intelligent data analysisrdquo Informa-tion Sciences vol 147 no 1ndash4 pp 1ndash12 2002
[48] Y Huang P J McCullagh and N D Black ldquoAn optimization ofReliefF for classification in large datasetsrdquoData and KnowledgeEngineering vol 68 no 11 pp 1348ndash1356 2009
[49] J Sun M-Y Jia and H Li ldquoAdaBoost ensemble for financialdistress prediction an empirical comparison with data fromChinese listed companiesrdquo Expert Systems with Applicationsvol 38 no 8 pp 9305ndash9312 2011
[50] R Kohavi and F Provost ldquoGlossary of termsrdquo Machine Learn-ing vol 30 no 2-3 pp 271ndash274 1998
[51] W Chen and H Lin ldquoPrediction of midbody centrosome andkinetochore proteins based on gene ontology informationrdquoBiochemical and Biophysical Research Communications vol 401no 3 pp 382ndash384 2010
[52] K-C Chou and C-T Zhang ldquoPrediction of protein structuralclassesrdquo Critical Reviews in Biochemistry andMolecular Biologyvol 30 no 4 pp 275ndash349 1995
[53] W Chen and H Lin ldquoIdentification of voltage-gated potassiumchannel subfamilies from sequence information using supportvectormachinerdquoComputers in Biology andMedicine vol 42 no4 pp 504ndash507 2012
[54] K-C Chou and H-B Shen ldquoRecent progress in proteinsubcellular location predictionrdquo Analytical Biochemistry vol370 no 1 pp 1ndash16 2007
[55] P Feng H Lin W Chen and Y Zuo ldquoPredicting the typesof J-proteins using clustered amino acidsrdquo BioMed ResearchInternational vol 2014 Article ID 935719 8 pages 2014
[56] K-C Chou and H-B Shen ldquoProtIdent a web server foridentifying proteases and their types by fusing functionaldomain and sequential evolution informationrdquoBiochemical andBiophysical Research Communications vol 376 no 2 pp 321ndash325 2008
[57] K-C Chou and Y-D Cai ldquoPrediction of membrane proteintypes by incorporating amphipathic effectsrdquo Journal of ChemicalInformation and Modeling vol 45 no 2 pp 407ndash413 2005
[58] F Tom ldquoROC graphs notes and practical considerations forresearchersrdquo Machine Learning vol 31 pp 1ndash38 2004
[59] JHuang andCX Ling ldquoUsingAUCand accuracy in evaluatinglearning algorithmsrdquo IEEE Transactions on Knowledge andDataEngineering vol 17 no 3 pp 299ndash310 2005
[60] F D Foresee and M T Hagan ldquoGauss-Newton approximationto Bayesian learningrdquo in Proceedings of the International Con-ference on Neural Networks vol 3 pp 1930ndash1935 Houston TexUSA 1997
In this paper a novel ReliefF and Rough Set- (RFRS-)based classification system is proposed for heart diseasediagnosisThemain novelty of this paper lies in the proposedapproach the combination of the ReliefF and RS methodsto classify heart disease problems in an efficient and fastmanner The RFRS classification system consists of twosubsystems the RFRS feature selection subsystem and theclassification subsystemThe Statlog (Heart) dataset from theUCI machine learning database [3] was selected to test thesystemThe experimental results show that the reduct1198772 (11986211198623 1198627 1198628 11986211 11986212 11986213) achieves the highest classificationaccuracy (9259) using an ensemble classifier with the C45decision tree as the weak learner The results also show thatthe RFRS method has superior performance compared to
three common classifiers in terms of ACC sensitivity andspecificity In addition the performance of the proposedsystem is superior to that of existingmethods in the literatureBased on empirical analysis the results indicate that theproposed classification system can be used as a promisingalternative tool in medical decision making for heart diseasediagnosis
However the proposed method also has some weak-nesses The number of the nearest neighbors (119896) and theweight threshold (120579) are not stable in the ReliefF algorithm[20] One solution to this problem is to compute estimatesfor all possible numbers and take the highest estimate of eachfeature as the final result [20] We need to perform moreexperiments to find the optimal parameter values for theReliefF algorithm in the future
Computational and Mathematical Methods in Medicine 9
Table 6 Comparison of our results with those of other studies
Author Method Classification accuracy ()Our study RFRS classification system 9259Lee [4] Graphical characteristics of BSWFM combined with Euclidean distance 874Tomar and Agarwal [5] Feature selection-based LSTSVM 8559Buscema et al [6] TWIST algorithm 8414Subbulakshmi et al [7] ELM 875Karegowda et al [8] GA + Naıve Bayes 8587Srinivas et al [9] Naıve Bayes 8370Polat and Gunes [10] RBF kernel 119865-score + LS-SVM 8370Ozsen and Gunes [11] GA-AWAIS 8743Helmy and Rasheed [12] Algebraic Sigmoid 8524Wang et al [13] Linear kernel SVM classifiers 8337Ozsen and Gunes [14] Hybrid similarity measure 8395Kahramanli and Allahverdi [15] Hybrid neural network method 868Yan et al [16] ICA + SVM 8375Sahan et al [17] AWAIS 8259Duch et al [18] KNN classifier 856BSWFM bounded sumof weighted fuzzymembership functions LSTSVM Least Square Twin Support VectorMachine TWIST Training with Input Selectionand Testing ELM Extreme LearningMachine GA genetic algorithm SVM support vectormachine ICA imperialist competitive algorithm AWAIS attributeweighted artificial immune system119870NN 119896-nearest neighbor
Test ROC
0 10
01
02
02
03
04
04
05
06
06
07
08
08
09
1
False positive rate
True
pos
itive
rate
0 10
01
02
02
03
04
04
05
06
06
07
08
08
09
1
False positive rate
True
pos
itive
rate
Training ROC
Figure 3 ROC curves for training and test sets
Competing Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
This study was supported by the National Natural ScienceFoundation of China (Grant no 71432007)
References
[1] K D Kochanek J Xu S L Murphy A M Minino and H-C Kung ldquoDeaths final data for 2009rdquo National Vital StatisticsReports vol 60 no 3 pp 1ndash116 2011
[2] H Temurtas N Yumusak and F Temurtas ldquoA comparativestudy on diabetes disease diagnosis using neural networksrdquoExpert Systems with Applications vol 36 no 4 pp 8610ndash86152009
10 Computational and Mathematical Methods in Medicine
[3] UCI Repository of Machine Learning Databases httparchiveicsuciedumldatasetsStatlog+28Heart29
[4] S-H Lee ldquoFeature selection based on the center of gravity ofBSWFMs using NEWFMrdquo Engineering Applications of ArtificialIntelligence vol 45 pp 482ndash487 2015
[5] D Tomar and S Agarwal ldquoFeature selection based least squaretwin support vector machine for diagnosis of heart diseaserdquoInternational Journal of Bio-Science and Bio-Technology vol 6no 2 pp 69ndash82 2014
[6] M Buscema M Breda and W Lodwick ldquoTraining with InputSelection and Testing (TWIST) algorithm a significant advancein pattern recognition performance of machine learningrdquoJournal of Intelligent Learning Systems and Applications vol 5no 1 pp 29ndash38 2013
[7] C V Subbulakshmi S N Deepa and N Malathi ldquoExtremelearning machine for two category data classificationrdquo inProceedings of the IEEE International Conference on AdvancedCommunication Control and Computing Technologies (ICAC-CCT rsquo12) pp 458ndash461 Ramanathapuram India August 2012
[8] A G Karegowda A SManjunath andM A Jayaram ldquoFeaturesubset selection problem using wrapper approach in supervisedlearningrdquo International Journal of Computer Applications vol 1no 7 pp 13ndash17 2010
[9] K Srinivas B K Rani and A Govrdhan ldquoApplications ofdata mining techniques in healthcare and prediction of heartattacksrdquo International Journal on Computer Science and Engi-neering vol 2 pp 250ndash255 2010
[10] K Polat and S Gunes ldquoA new feature selection methodon classification of medical datasets kernel F-score featureselectionrdquo Expert Systems with Applications vol 36 no 7 pp10367ndash10373 2009
[11] S Ozsen and S Gunes ldquoAttribute weighting via genetic algo-rithms for attributeweighted artificial immune system (AWAIS)and its application to heart disease and liver disorders prob-lemsrdquo Expert Systems with Applications vol 36 no 1 pp 386ndash392 2009
[12] T Helmy and Z Rasheed ldquoMulti-category bioinformaticsdataset classification using extreme learning machinerdquo in Pro-ceedings of the IEEE Congress on Evolutionary Computation(CEC rsquo09) pp 3234ndash3240 Trondheim Norway May 2009
[13] S-J Wang A Mathew Y Chen L-F Xi L Ma and JLee ldquoEmpirical analysis of support vector machine ensembleclassifiersrdquo Expert Systems with Applications vol 36 no 3 pp6466ndash6476 2009
[14] S Ozsen and S Gunes ldquoEffect of feature-type in selectingdistance measure for an artificial immune system as a patternrecognizerrdquoDigital Signal Processing vol 18 no 4 pp 635ndash6452008
[15] H Kahramanli and N Allahverdi ldquoDesign of a hybrid systemfor the diabetes and heart diseasesrdquo Expert Systems withApplications vol 35 no 1-2 pp 82ndash89 2008
[16] G Yan G Ma J Lv and B Song ldquoCombining independentcomponent analysis with support vector machinesrdquo in Pro-ceedings of the in 1st International Symposium on Systems andControl in Aerospace and Astronautics (ISSCAA rsquo06) pp 493ndash496 Harbin China January 2006
[17] S Sahan K Polat H Kodaz and S Gunes ldquoThe medicalapplications of attribute weighted artificial immune system(AWAIS) diagnosis of heart and diabetes diseasesrdquo ArtificialImmune Systems vol 3627 pp 456ndash468 2005
[18] W Duch R Adamczak and K Grabczewski ldquoA new method-ology of extraction optimization and application of crisp and
fuzzy logical rulesrdquo IEEE Transactions on Neural Networks vol12 no 2 pp 277ndash306 2001
[19] M P McRae B Bozkurt C M Ballantyne et al ldquoCardiacScoreCard a diagnostic multivariate index assay system forpredicting a spectrumof cardiovascular diseaserdquoExpert Systemswith Applications vol 54 pp 136ndash147 2016
[20] M Robnik-Sikonja and I Kononenko ldquoTheoretical and empir-ical analysis of ReliefF and RReliefFrdquoMachine Learning vol 53no 1-2 pp 23ndash69 2003
[21] D Wettschereck D W Aha and T Mohri ldquoA review andempirical evaluation of feature weighting methods for a classof lazy learning algorithmsrdquo Artificial Intelligence Review vol11 no 1ndash5 pp 273ndash314 1997
[22] I Kononenko E Simec and M R Sikonja ldquoOvercomingthe myopia of inductive learning algorithms with RELIEFFrdquoApplied Intelligence vol 7 no 1 pp 39ndash55 1997
[23] K Kira and L Rendell ldquoA practical approach to featureselectionrdquo in Proceedings of the International Conference onMachine Learning pp 249ndash256 Morgan Kaufmann AberdeenScotland 1992
[24] I Kononenko ldquoEstimating attributes analysis and extension ofReliefFrdquo in Machine Learning ECML-94 European ConferenceonMachine Learning Catania Italy April 6ndash8 1994 Proceedingsvol 784 of Lecture Notes in Computer Science pp 171ndash182Springer Berlin Germany 1994
[25] R Ruiz J C Riquelme and J S Aguilar-Ruiz ldquoHeuristicsearch over a ranking for feature selectionrdquo in Proceedings of8th InternationalWork-Conference onArtificial NeuralNetworks(IWANN rsquo05) Barcelona Spain June 2005 vol 3512 of LecturesNotes in Computer Science pp 742ndash749 Springer Berlin Ger-many 2005
[26] N Spolaor E A ChermanM CMonard andHD Lee ldquoFilterapproach feature selection methods to support multi-labellearning based on ReliefF and information gainrdquo in Advancesin Artificial IntelligencemdashSBIA 2012 21th Brazilian Symposiumon Artificial Intelligence Curitiba Brazil October 20ndash25 2012Proceedings vol 7589 of Lectures Notes in Computer Science pp72ndash81 Springer Berlin Germany 2012
[27] R Ruiz J C Riquelme and J S Aguilar-Ruiz ldquoFast featureranking algorithmrdquo in Proceedings of the Knowledge-BasedIntelligent Information and Engineering Systems (KES rsquo03) pp325ndash331 Springer Berlin Germany 2003
[28] V Jovanoski and N Lavrac ldquoFeature subset selection inassociation rules learning systemsrdquo in Proceedings of AnalysisWarehousing and Mining the Data pp 74ndash77 1999
[29] J J Liu and J T Y Kwok ldquoAn extended genetic rule inductionalgorithmrdquo in Proceedings of the Congress on EvolutionaryComputation pp 458ndash463 LA Jolla Calif USA July 2000
[30] L-X Zhang J-X Wang Y-N Zhao and Z-H Yang ldquoA novelhybrid feature selection algorithm using ReliefF estimationfor GA-Wrapper searchrdquo in Proceedings of the InternationalConference onMachine Learning andCybernetics vol 1 pp 380ndash384 IEEE Xirsquoan China November 2003
[31] Z Zhao L Wang and H Liu ldquoEfficient spectral featureselectionwithminimum redundancyrdquo in Proceedings of the 24thAAAI Conference on Artificial Intelligence AAAI Atlanta GaUSA 2010
[32] F Nie H Huang X Cai and C H Ding ldquoEfficient androbust feature selection via joint ℓ21-norms minimizationrdquo inAdvances in Neural Information Processing Systems pp 1813ndash1821 MIT Press 2010
Computational and Mathematical Methods in Medicine 11
[33] S-Y Jiang and L-X Wang ldquoEfficient feature selection based oncorrelation measure between continuous and discrete featuresrdquoInformation Processing Letters vol 116 no 2 pp 203ndash215 2016
[34] M J Kearns and U V Vazirani An Introduction to Compu-tational Learning Theory MIT Press Cambridge Mass USA1994
[35] R Bellman Adaptive Control Processes A Guided Tour RANDCorporation Research Studies Princeton University PressPrinceton NJ USA 1961
[36] Z Pawlak ldquoRough set theory and its applications to dataanalysisrdquo Cybernetics and Systems vol 29 no 7 pp 661ndash6881998
[37] A-E Hassanien ldquoRough set approach for attribute reductionand rule generation a case of patients with suspected breastcancerrdquo Journal of the American Society for Information Scienceand Technology vol 55 no 11 pp 954ndash962 2004
[38] A Skowron and C Rauszer ldquoThe discernibility matricesand functions in information systemsrdquo in Intelligent DecisionSupport-Handbook of Applications and Advances of the RoughSetsTheory SystemTheory Knowledge Engineering and ProblemSolving R Słowinski Ed vol 11 pp 331ndash362 KluwerAcademicDordrecht Netherlands 1992
[39] Z Pawlak ldquoRough set approach to knowledge-based decisionsupportrdquo European Journal of Operational Research vol 99 no1 pp 48ndash57 1997
[40] X Wang J Yang X Teng W Xia and R Jensen ldquoFeatureselection based on rough sets and particle swarm optimizationrdquoPattern Recognition Letters vol 28 no 4 pp 459ndash471 2007
[41] C Ma J Ouyang H-L Chen and X-H Zhao ldquoAn efficientdiagnosis system for Parkinsonrsquos disease using kernel-basedextreme learning machine with subtractive clustering featuresweighting approachrdquo Computational and Mathematical Meth-ods in Medicine vol 2014 Article ID 985789 14 pages 2014
[42] A H El-Baz ldquoHybrid intelligent system-based rough set andensemble classifier for breast cancer diagnosisrdquoNeural Comput-ing and Applications vol 26 no 2 pp 437ndash446 2015
[43] J-H Eom S-CKim andB-T Zhang ldquoAptaCDSS-E a classifierensemble-based clinical decision support system for cardiovas-cular disease level predictionrdquo Expert Systems with Applicationsvol 34 no 4 pp 2465ndash2479 2008
[44] Y Ma Ensemble Machine Learning Methods and ApplicationsSpringer New York NY USA 2012
[45] S A Etemad and A Arya ldquoClassification and translation ofstyle and affect in human motion using RBF neural networksrdquoNeurocomputing vol 129 pp 585ndash595 2014
[46] Z Pawlak ldquoRough setsrdquo International Journal of Computer ampInformation Sciences vol 11 no 5 pp 341ndash356 1982
[47] Z Pawlak ldquoRough sets and intelligent data analysisrdquo Informa-tion Sciences vol 147 no 1ndash4 pp 1ndash12 2002
[48] Y Huang P J McCullagh and N D Black ldquoAn optimization ofReliefF for classification in large datasetsrdquoData and KnowledgeEngineering vol 68 no 11 pp 1348ndash1356 2009
[49] J Sun M-Y Jia and H Li ldquoAdaBoost ensemble for financialdistress prediction an empirical comparison with data fromChinese listed companiesrdquo Expert Systems with Applicationsvol 38 no 8 pp 9305ndash9312 2011
[50] R Kohavi and F Provost ldquoGlossary of termsrdquo Machine Learn-ing vol 30 no 2-3 pp 271ndash274 1998
[51] W Chen and H Lin ldquoPrediction of midbody centrosome andkinetochore proteins based on gene ontology informationrdquoBiochemical and Biophysical Research Communications vol 401no 3 pp 382ndash384 2010
[52] K-C Chou and C-T Zhang ldquoPrediction of protein structuralclassesrdquo Critical Reviews in Biochemistry andMolecular Biologyvol 30 no 4 pp 275ndash349 1995
[53] W Chen and H Lin ldquoIdentification of voltage-gated potassiumchannel subfamilies from sequence information using supportvectormachinerdquoComputers in Biology andMedicine vol 42 no4 pp 504ndash507 2012
[54] K-C Chou and H-B Shen ldquoRecent progress in proteinsubcellular location predictionrdquo Analytical Biochemistry vol370 no 1 pp 1ndash16 2007
[55] P Feng H Lin W Chen and Y Zuo ldquoPredicting the typesof J-proteins using clustered amino acidsrdquo BioMed ResearchInternational vol 2014 Article ID 935719 8 pages 2014
[56] K-C Chou and H-B Shen ldquoProtIdent a web server foridentifying proteases and their types by fusing functionaldomain and sequential evolution informationrdquoBiochemical andBiophysical Research Communications vol 376 no 2 pp 321ndash325 2008
[57] K-C Chou and Y-D Cai ldquoPrediction of membrane proteintypes by incorporating amphipathic effectsrdquo Journal of ChemicalInformation and Modeling vol 45 no 2 pp 407ndash413 2005
[58] F Tom ldquoROC graphs notes and practical considerations forresearchersrdquo Machine Learning vol 31 pp 1ndash38 2004
[59] JHuang andCX Ling ldquoUsingAUCand accuracy in evaluatinglearning algorithmsrdquo IEEE Transactions on Knowledge andDataEngineering vol 17 no 3 pp 299ndash310 2005
[60] F D Foresee and M T Hagan ldquoGauss-Newton approximationto Bayesian learningrdquo in Proceedings of the International Con-ference on Neural Networks vol 3 pp 1930ndash1935 Houston TexUSA 1997
Computational and Mathematical Methods in Medicine 9
Table 6 Comparison of our results with those of other studies
Author Method Classification accuracy ()Our study RFRS classification system 9259Lee [4] Graphical characteristics of BSWFM combined with Euclidean distance 874Tomar and Agarwal [5] Feature selection-based LSTSVM 8559Buscema et al [6] TWIST algorithm 8414Subbulakshmi et al [7] ELM 875Karegowda et al [8] GA + Naıve Bayes 8587Srinivas et al [9] Naıve Bayes 8370Polat and Gunes [10] RBF kernel 119865-score + LS-SVM 8370Ozsen and Gunes [11] GA-AWAIS 8743Helmy and Rasheed [12] Algebraic Sigmoid 8524Wang et al [13] Linear kernel SVM classifiers 8337Ozsen and Gunes [14] Hybrid similarity measure 8395Kahramanli and Allahverdi [15] Hybrid neural network method 868Yan et al [16] ICA + SVM 8375Sahan et al [17] AWAIS 8259Duch et al [18] KNN classifier 856BSWFM bounded sumof weighted fuzzymembership functions LSTSVM Least Square Twin Support VectorMachine TWIST Training with Input Selectionand Testing ELM Extreme LearningMachine GA genetic algorithm SVM support vectormachine ICA imperialist competitive algorithm AWAIS attributeweighted artificial immune system119870NN 119896-nearest neighbor
Test ROC
0 10
01
02
02
03
04
04
05
06
06
07
08
08
09
1
False positive rate
True
pos
itive
rate
0 10
01
02
02
03
04
04
05
06
06
07
08
08
09
1
False positive rate
True
pos
itive
rate
Training ROC
Figure 3 ROC curves for training and test sets
Competing Interests
The authors declare that there is no conflict of interestsregarding the publication of this paper
Acknowledgments
This study was supported by the National Natural ScienceFoundation of China (Grant no 71432007)
References
[1] K D Kochanek J Xu S L Murphy A M Minino and H-C Kung ldquoDeaths final data for 2009rdquo National Vital StatisticsReports vol 60 no 3 pp 1ndash116 2011
[2] H Temurtas N Yumusak and F Temurtas ldquoA comparativestudy on diabetes disease diagnosis using neural networksrdquoExpert Systems with Applications vol 36 no 4 pp 8610ndash86152009
10 Computational and Mathematical Methods in Medicine
[3] UCI Repository of Machine Learning Databases httparchiveicsuciedumldatasetsStatlog+28Heart29
[4] S-H Lee ldquoFeature selection based on the center of gravity ofBSWFMs using NEWFMrdquo Engineering Applications of ArtificialIntelligence vol 45 pp 482ndash487 2015
[5] D Tomar and S Agarwal ldquoFeature selection based least squaretwin support vector machine for diagnosis of heart diseaserdquoInternational Journal of Bio-Science and Bio-Technology vol 6no 2 pp 69ndash82 2014
[6] M Buscema M Breda and W Lodwick ldquoTraining with InputSelection and Testing (TWIST) algorithm a significant advancein pattern recognition performance of machine learningrdquoJournal of Intelligent Learning Systems and Applications vol 5no 1 pp 29ndash38 2013
[7] C V Subbulakshmi S N Deepa and N Malathi ldquoExtremelearning machine for two category data classificationrdquo inProceedings of the IEEE International Conference on AdvancedCommunication Control and Computing Technologies (ICAC-CCT rsquo12) pp 458ndash461 Ramanathapuram India August 2012
[8] A G Karegowda A SManjunath andM A Jayaram ldquoFeaturesubset selection problem using wrapper approach in supervisedlearningrdquo International Journal of Computer Applications vol 1no 7 pp 13ndash17 2010
[9] K Srinivas B K Rani and A Govrdhan ldquoApplications ofdata mining techniques in healthcare and prediction of heartattacksrdquo International Journal on Computer Science and Engi-neering vol 2 pp 250ndash255 2010
[10] K Polat and S Gunes ldquoA new feature selection methodon classification of medical datasets kernel F-score featureselectionrdquo Expert Systems with Applications vol 36 no 7 pp10367ndash10373 2009
[11] S Ozsen and S Gunes ldquoAttribute weighting via genetic algo-rithms for attributeweighted artificial immune system (AWAIS)and its application to heart disease and liver disorders prob-lemsrdquo Expert Systems with Applications vol 36 no 1 pp 386ndash392 2009
[12] T Helmy and Z Rasheed ldquoMulti-category bioinformaticsdataset classification using extreme learning machinerdquo in Pro-ceedings of the IEEE Congress on Evolutionary Computation(CEC rsquo09) pp 3234ndash3240 Trondheim Norway May 2009
[13] S-J Wang A Mathew Y Chen L-F Xi L Ma and JLee ldquoEmpirical analysis of support vector machine ensembleclassifiersrdquo Expert Systems with Applications vol 36 no 3 pp6466ndash6476 2009
[14] S Ozsen and S Gunes ldquoEffect of feature-type in selectingdistance measure for an artificial immune system as a patternrecognizerrdquoDigital Signal Processing vol 18 no 4 pp 635ndash6452008
[15] H Kahramanli and N Allahverdi ldquoDesign of a hybrid systemfor the diabetes and heart diseasesrdquo Expert Systems withApplications vol 35 no 1-2 pp 82ndash89 2008
[16] G Yan G Ma J Lv and B Song ldquoCombining independentcomponent analysis with support vector machinesrdquo in Pro-ceedings of the in 1st International Symposium on Systems andControl in Aerospace and Astronautics (ISSCAA rsquo06) pp 493ndash496 Harbin China January 2006
[17] S Sahan K Polat H Kodaz and S Gunes ldquoThe medicalapplications of attribute weighted artificial immune system(AWAIS) diagnosis of heart and diabetes diseasesrdquo ArtificialImmune Systems vol 3627 pp 456ndash468 2005
[18] W Duch R Adamczak and K Grabczewski ldquoA new method-ology of extraction optimization and application of crisp and
fuzzy logical rulesrdquo IEEE Transactions on Neural Networks vol12 no 2 pp 277ndash306 2001
[19] M P McRae B Bozkurt C M Ballantyne et al ldquoCardiacScoreCard a diagnostic multivariate index assay system forpredicting a spectrumof cardiovascular diseaserdquoExpert Systemswith Applications vol 54 pp 136ndash147 2016
[20] M Robnik-Sikonja and I Kononenko ldquoTheoretical and empir-ical analysis of ReliefF and RReliefFrdquoMachine Learning vol 53no 1-2 pp 23ndash69 2003
[21] D Wettschereck D W Aha and T Mohri ldquoA review andempirical evaluation of feature weighting methods for a classof lazy learning algorithmsrdquo Artificial Intelligence Review vol11 no 1ndash5 pp 273ndash314 1997
[22] I Kononenko E Simec and M R Sikonja ldquoOvercomingthe myopia of inductive learning algorithms with RELIEFFrdquoApplied Intelligence vol 7 no 1 pp 39ndash55 1997
[23] K Kira and L Rendell ldquoA practical approach to featureselectionrdquo in Proceedings of the International Conference onMachine Learning pp 249ndash256 Morgan Kaufmann AberdeenScotland 1992
[24] I Kononenko ldquoEstimating attributes analysis and extension ofReliefFrdquo in Machine Learning ECML-94 European ConferenceonMachine Learning Catania Italy April 6ndash8 1994 Proceedingsvol 784 of Lecture Notes in Computer Science pp 171ndash182Springer Berlin Germany 1994
[25] R Ruiz J C Riquelme and J S Aguilar-Ruiz ldquoHeuristicsearch over a ranking for feature selectionrdquo in Proceedings of8th InternationalWork-Conference onArtificial NeuralNetworks(IWANN rsquo05) Barcelona Spain June 2005 vol 3512 of LecturesNotes in Computer Science pp 742ndash749 Springer Berlin Ger-many 2005
[26] N Spolaor E A ChermanM CMonard andHD Lee ldquoFilterapproach feature selection methods to support multi-labellearning based on ReliefF and information gainrdquo in Advancesin Artificial IntelligencemdashSBIA 2012 21th Brazilian Symposiumon Artificial Intelligence Curitiba Brazil October 20ndash25 2012Proceedings vol 7589 of Lectures Notes in Computer Science pp72ndash81 Springer Berlin Germany 2012
[27] R Ruiz J C Riquelme and J S Aguilar-Ruiz ldquoFast featureranking algorithmrdquo in Proceedings of the Knowledge-BasedIntelligent Information and Engineering Systems (KES rsquo03) pp325ndash331 Springer Berlin Germany 2003
[28] V Jovanoski and N Lavrac ldquoFeature subset selection inassociation rules learning systemsrdquo in Proceedings of AnalysisWarehousing and Mining the Data pp 74ndash77 1999
[29] J J Liu and J T Y Kwok ldquoAn extended genetic rule inductionalgorithmrdquo in Proceedings of the Congress on EvolutionaryComputation pp 458ndash463 LA Jolla Calif USA July 2000
[30] L-X Zhang J-X Wang Y-N Zhao and Z-H Yang ldquoA novelhybrid feature selection algorithm using ReliefF estimationfor GA-Wrapper searchrdquo in Proceedings of the InternationalConference onMachine Learning andCybernetics vol 1 pp 380ndash384 IEEE Xirsquoan China November 2003
[31] Z Zhao L Wang and H Liu ldquoEfficient spectral featureselectionwithminimum redundancyrdquo in Proceedings of the 24thAAAI Conference on Artificial Intelligence AAAI Atlanta GaUSA 2010
[32] F Nie H Huang X Cai and C H Ding ldquoEfficient androbust feature selection via joint ℓ21-norms minimizationrdquo inAdvances in Neural Information Processing Systems pp 1813ndash1821 MIT Press 2010
Computational and Mathematical Methods in Medicine 11
[33] S-Y Jiang and L-X Wang ldquoEfficient feature selection based oncorrelation measure between continuous and discrete featuresrdquoInformation Processing Letters vol 116 no 2 pp 203ndash215 2016
[34] M J Kearns and U V Vazirani An Introduction to Compu-tational Learning Theory MIT Press Cambridge Mass USA1994
[35] R Bellman Adaptive Control Processes A Guided Tour RANDCorporation Research Studies Princeton University PressPrinceton NJ USA 1961
[36] Z Pawlak ldquoRough set theory and its applications to dataanalysisrdquo Cybernetics and Systems vol 29 no 7 pp 661ndash6881998
[37] A-E Hassanien ldquoRough set approach for attribute reductionand rule generation a case of patients with suspected breastcancerrdquo Journal of the American Society for Information Scienceand Technology vol 55 no 11 pp 954ndash962 2004
[38] A Skowron and C Rauszer ldquoThe discernibility matricesand functions in information systemsrdquo in Intelligent DecisionSupport-Handbook of Applications and Advances of the RoughSetsTheory SystemTheory Knowledge Engineering and ProblemSolving R Słowinski Ed vol 11 pp 331ndash362 KluwerAcademicDordrecht Netherlands 1992
[39] Z Pawlak ldquoRough set approach to knowledge-based decisionsupportrdquo European Journal of Operational Research vol 99 no1 pp 48ndash57 1997
[40] X Wang J Yang X Teng W Xia and R Jensen ldquoFeatureselection based on rough sets and particle swarm optimizationrdquoPattern Recognition Letters vol 28 no 4 pp 459ndash471 2007
[41] C Ma J Ouyang H-L Chen and X-H Zhao ldquoAn efficientdiagnosis system for Parkinsonrsquos disease using kernel-basedextreme learning machine with subtractive clustering featuresweighting approachrdquo Computational and Mathematical Meth-ods in Medicine vol 2014 Article ID 985789 14 pages 2014
[42] A H El-Baz ldquoHybrid intelligent system-based rough set andensemble classifier for breast cancer diagnosisrdquoNeural Comput-ing and Applications vol 26 no 2 pp 437ndash446 2015
[43] J-H Eom S-CKim andB-T Zhang ldquoAptaCDSS-E a classifierensemble-based clinical decision support system for cardiovas-cular disease level predictionrdquo Expert Systems with Applicationsvol 34 no 4 pp 2465ndash2479 2008
[44] Y Ma Ensemble Machine Learning Methods and ApplicationsSpringer New York NY USA 2012
[45] S A Etemad and A Arya ldquoClassification and translation ofstyle and affect in human motion using RBF neural networksrdquoNeurocomputing vol 129 pp 585ndash595 2014
[46] Z Pawlak ldquoRough setsrdquo International Journal of Computer ampInformation Sciences vol 11 no 5 pp 341ndash356 1982
[47] Z Pawlak ldquoRough sets and intelligent data analysisrdquo Informa-tion Sciences vol 147 no 1ndash4 pp 1ndash12 2002
[48] Y Huang P J McCullagh and N D Black ldquoAn optimization ofReliefF for classification in large datasetsrdquoData and KnowledgeEngineering vol 68 no 11 pp 1348ndash1356 2009
[49] J Sun M-Y Jia and H Li ldquoAdaBoost ensemble for financialdistress prediction an empirical comparison with data fromChinese listed companiesrdquo Expert Systems with Applicationsvol 38 no 8 pp 9305ndash9312 2011
[50] R Kohavi and F Provost ldquoGlossary of termsrdquo Machine Learn-ing vol 30 no 2-3 pp 271ndash274 1998
[51] W Chen and H Lin ldquoPrediction of midbody centrosome andkinetochore proteins based on gene ontology informationrdquoBiochemical and Biophysical Research Communications vol 401no 3 pp 382ndash384 2010
[52] K-C Chou and C-T Zhang ldquoPrediction of protein structuralclassesrdquo Critical Reviews in Biochemistry andMolecular Biologyvol 30 no 4 pp 275ndash349 1995
[53] W Chen and H Lin ldquoIdentification of voltage-gated potassiumchannel subfamilies from sequence information using supportvectormachinerdquoComputers in Biology andMedicine vol 42 no4 pp 504ndash507 2012
[54] K-C Chou and H-B Shen ldquoRecent progress in proteinsubcellular location predictionrdquo Analytical Biochemistry vol370 no 1 pp 1ndash16 2007
[55] P Feng H Lin W Chen and Y Zuo ldquoPredicting the typesof J-proteins using clustered amino acidsrdquo BioMed ResearchInternational vol 2014 Article ID 935719 8 pages 2014
[56] K-C Chou and H-B Shen ldquoProtIdent a web server foridentifying proteases and their types by fusing functionaldomain and sequential evolution informationrdquoBiochemical andBiophysical Research Communications vol 376 no 2 pp 321ndash325 2008
[57] K-C Chou and Y-D Cai ldquoPrediction of membrane proteintypes by incorporating amphipathic effectsrdquo Journal of ChemicalInformation and Modeling vol 45 no 2 pp 407ndash413 2005
[58] F Tom ldquoROC graphs notes and practical considerations forresearchersrdquo Machine Learning vol 31 pp 1ndash38 2004
[59] JHuang andCX Ling ldquoUsingAUCand accuracy in evaluatinglearning algorithmsrdquo IEEE Transactions on Knowledge andDataEngineering vol 17 no 3 pp 299ndash310 2005
[60] F D Foresee and M T Hagan ldquoGauss-Newton approximationto Bayesian learningrdquo in Proceedings of the International Con-ference on Neural Networks vol 3 pp 1930ndash1935 Houston TexUSA 1997
10 Computational and Mathematical Methods in Medicine
[3] UCI Repository of Machine Learning Databases httparchiveicsuciedumldatasetsStatlog+28Heart29
[4] S-H Lee ldquoFeature selection based on the center of gravity ofBSWFMs using NEWFMrdquo Engineering Applications of ArtificialIntelligence vol 45 pp 482ndash487 2015
[5] D Tomar and S Agarwal ldquoFeature selection based least squaretwin support vector machine for diagnosis of heart diseaserdquoInternational Journal of Bio-Science and Bio-Technology vol 6no 2 pp 69ndash82 2014
[6] M Buscema M Breda and W Lodwick ldquoTraining with InputSelection and Testing (TWIST) algorithm a significant advancein pattern recognition performance of machine learningrdquoJournal of Intelligent Learning Systems and Applications vol 5no 1 pp 29ndash38 2013
[7] C V Subbulakshmi S N Deepa and N Malathi ldquoExtremelearning machine for two category data classificationrdquo inProceedings of the IEEE International Conference on AdvancedCommunication Control and Computing Technologies (ICAC-CCT rsquo12) pp 458ndash461 Ramanathapuram India August 2012
[8] A G Karegowda A SManjunath andM A Jayaram ldquoFeaturesubset selection problem using wrapper approach in supervisedlearningrdquo International Journal of Computer Applications vol 1no 7 pp 13ndash17 2010
[9] K Srinivas B K Rani and A Govrdhan ldquoApplications ofdata mining techniques in healthcare and prediction of heartattacksrdquo International Journal on Computer Science and Engi-neering vol 2 pp 250ndash255 2010
[10] K Polat and S Gunes ldquoA new feature selection methodon classification of medical datasets kernel F-score featureselectionrdquo Expert Systems with Applications vol 36 no 7 pp10367ndash10373 2009
[11] S Ozsen and S Gunes ldquoAttribute weighting via genetic algo-rithms for attributeweighted artificial immune system (AWAIS)and its application to heart disease and liver disorders prob-lemsrdquo Expert Systems with Applications vol 36 no 1 pp 386ndash392 2009
[12] T Helmy and Z Rasheed ldquoMulti-category bioinformaticsdataset classification using extreme learning machinerdquo in Pro-ceedings of the IEEE Congress on Evolutionary Computation(CEC rsquo09) pp 3234ndash3240 Trondheim Norway May 2009
[13] S-J Wang A Mathew Y Chen L-F Xi L Ma and JLee ldquoEmpirical analysis of support vector machine ensembleclassifiersrdquo Expert Systems with Applications vol 36 no 3 pp6466ndash6476 2009
[14] S Ozsen and S Gunes ldquoEffect of feature-type in selectingdistance measure for an artificial immune system as a patternrecognizerrdquoDigital Signal Processing vol 18 no 4 pp 635ndash6452008
[15] H Kahramanli and N Allahverdi ldquoDesign of a hybrid systemfor the diabetes and heart diseasesrdquo Expert Systems withApplications vol 35 no 1-2 pp 82ndash89 2008
[16] G Yan G Ma J Lv and B Song ldquoCombining independentcomponent analysis with support vector machinesrdquo in Pro-ceedings of the in 1st International Symposium on Systems andControl in Aerospace and Astronautics (ISSCAA rsquo06) pp 493ndash496 Harbin China January 2006
[17] S Sahan K Polat H Kodaz and S Gunes ldquoThe medicalapplications of attribute weighted artificial immune system(AWAIS) diagnosis of heart and diabetes diseasesrdquo ArtificialImmune Systems vol 3627 pp 456ndash468 2005
[18] W Duch R Adamczak and K Grabczewski ldquoA new method-ology of extraction optimization and application of crisp and
fuzzy logical rulesrdquo IEEE Transactions on Neural Networks vol12 no 2 pp 277ndash306 2001
[19] M P McRae B Bozkurt C M Ballantyne et al ldquoCardiacScoreCard a diagnostic multivariate index assay system forpredicting a spectrumof cardiovascular diseaserdquoExpert Systemswith Applications vol 54 pp 136ndash147 2016
[20] M Robnik-Sikonja and I Kononenko ldquoTheoretical and empir-ical analysis of ReliefF and RReliefFrdquoMachine Learning vol 53no 1-2 pp 23ndash69 2003
[21] D Wettschereck D W Aha and T Mohri ldquoA review andempirical evaluation of feature weighting methods for a classof lazy learning algorithmsrdquo Artificial Intelligence Review vol11 no 1ndash5 pp 273ndash314 1997
[22] I Kononenko E Simec and M R Sikonja ldquoOvercomingthe myopia of inductive learning algorithms with RELIEFFrdquoApplied Intelligence vol 7 no 1 pp 39ndash55 1997
[23] K Kira and L Rendell ldquoA practical approach to featureselectionrdquo in Proceedings of the International Conference onMachine Learning pp 249ndash256 Morgan Kaufmann AberdeenScotland 1992
[24] I Kononenko ldquoEstimating attributes analysis and extension ofReliefFrdquo in Machine Learning ECML-94 European ConferenceonMachine Learning Catania Italy April 6ndash8 1994 Proceedingsvol 784 of Lecture Notes in Computer Science pp 171ndash182Springer Berlin Germany 1994
[25] R Ruiz J C Riquelme and J S Aguilar-Ruiz ldquoHeuristicsearch over a ranking for feature selectionrdquo in Proceedings of8th InternationalWork-Conference onArtificial NeuralNetworks(IWANN rsquo05) Barcelona Spain June 2005 vol 3512 of LecturesNotes in Computer Science pp 742ndash749 Springer Berlin Ger-many 2005
[26] N Spolaor E A ChermanM CMonard andHD Lee ldquoFilterapproach feature selection methods to support multi-labellearning based on ReliefF and information gainrdquo in Advancesin Artificial IntelligencemdashSBIA 2012 21th Brazilian Symposiumon Artificial Intelligence Curitiba Brazil October 20ndash25 2012Proceedings vol 7589 of Lectures Notes in Computer Science pp72ndash81 Springer Berlin Germany 2012
[27] R Ruiz J C Riquelme and J S Aguilar-Ruiz ldquoFast featureranking algorithmrdquo in Proceedings of the Knowledge-BasedIntelligent Information and Engineering Systems (KES rsquo03) pp325ndash331 Springer Berlin Germany 2003
[28] V Jovanoski and N Lavrac ldquoFeature subset selection inassociation rules learning systemsrdquo in Proceedings of AnalysisWarehousing and Mining the Data pp 74ndash77 1999
[29] J J Liu and J T Y Kwok ldquoAn extended genetic rule inductionalgorithmrdquo in Proceedings of the Congress on EvolutionaryComputation pp 458ndash463 LA Jolla Calif USA July 2000
[30] L-X Zhang J-X Wang Y-N Zhao and Z-H Yang ldquoA novelhybrid feature selection algorithm using ReliefF estimationfor GA-Wrapper searchrdquo in Proceedings of the InternationalConference onMachine Learning andCybernetics vol 1 pp 380ndash384 IEEE Xirsquoan China November 2003
[31] Z Zhao L Wang and H Liu ldquoEfficient spectral featureselectionwithminimum redundancyrdquo in Proceedings of the 24thAAAI Conference on Artificial Intelligence AAAI Atlanta GaUSA 2010
[32] F Nie H Huang X Cai and C H Ding ldquoEfficient androbust feature selection via joint ℓ21-norms minimizationrdquo inAdvances in Neural Information Processing Systems pp 1813ndash1821 MIT Press 2010
Computational and Mathematical Methods in Medicine 11
[33] S-Y Jiang and L-X Wang ldquoEfficient feature selection based oncorrelation measure between continuous and discrete featuresrdquoInformation Processing Letters vol 116 no 2 pp 203ndash215 2016
[34] M J Kearns and U V Vazirani An Introduction to Compu-tational Learning Theory MIT Press Cambridge Mass USA1994
[35] R Bellman Adaptive Control Processes A Guided Tour RANDCorporation Research Studies Princeton University PressPrinceton NJ USA 1961
[36] Z Pawlak ldquoRough set theory and its applications to dataanalysisrdquo Cybernetics and Systems vol 29 no 7 pp 661ndash6881998
[37] A-E Hassanien ldquoRough set approach for attribute reductionand rule generation a case of patients with suspected breastcancerrdquo Journal of the American Society for Information Scienceand Technology vol 55 no 11 pp 954ndash962 2004
[38] A Skowron and C Rauszer ldquoThe discernibility matricesand functions in information systemsrdquo in Intelligent DecisionSupport-Handbook of Applications and Advances of the RoughSetsTheory SystemTheory Knowledge Engineering and ProblemSolving R Słowinski Ed vol 11 pp 331ndash362 KluwerAcademicDordrecht Netherlands 1992
[39] Z Pawlak ldquoRough set approach to knowledge-based decisionsupportrdquo European Journal of Operational Research vol 99 no1 pp 48ndash57 1997
[40] X Wang J Yang X Teng W Xia and R Jensen ldquoFeatureselection based on rough sets and particle swarm optimizationrdquoPattern Recognition Letters vol 28 no 4 pp 459ndash471 2007
[41] C Ma J Ouyang H-L Chen and X-H Zhao ldquoAn efficientdiagnosis system for Parkinsonrsquos disease using kernel-basedextreme learning machine with subtractive clustering featuresweighting approachrdquo Computational and Mathematical Meth-ods in Medicine vol 2014 Article ID 985789 14 pages 2014
[42] A H El-Baz ldquoHybrid intelligent system-based rough set andensemble classifier for breast cancer diagnosisrdquoNeural Comput-ing and Applications vol 26 no 2 pp 437ndash446 2015
[43] J-H Eom S-CKim andB-T Zhang ldquoAptaCDSS-E a classifierensemble-based clinical decision support system for cardiovas-cular disease level predictionrdquo Expert Systems with Applicationsvol 34 no 4 pp 2465ndash2479 2008
[44] Y Ma Ensemble Machine Learning Methods and ApplicationsSpringer New York NY USA 2012
[45] S A Etemad and A Arya ldquoClassification and translation ofstyle and affect in human motion using RBF neural networksrdquoNeurocomputing vol 129 pp 585ndash595 2014
[46] Z Pawlak ldquoRough setsrdquo International Journal of Computer ampInformation Sciences vol 11 no 5 pp 341ndash356 1982
[47] Z Pawlak ldquoRough sets and intelligent data analysisrdquo Informa-tion Sciences vol 147 no 1ndash4 pp 1ndash12 2002
[48] Y Huang P J McCullagh and N D Black ldquoAn optimization ofReliefF for classification in large datasetsrdquoData and KnowledgeEngineering vol 68 no 11 pp 1348ndash1356 2009
[49] J Sun M-Y Jia and H Li ldquoAdaBoost ensemble for financialdistress prediction an empirical comparison with data fromChinese listed companiesrdquo Expert Systems with Applicationsvol 38 no 8 pp 9305ndash9312 2011
[50] R Kohavi and F Provost ldquoGlossary of termsrdquo Machine Learn-ing vol 30 no 2-3 pp 271ndash274 1998
[51] W Chen and H Lin ldquoPrediction of midbody centrosome andkinetochore proteins based on gene ontology informationrdquoBiochemical and Biophysical Research Communications vol 401no 3 pp 382ndash384 2010
[52] K-C Chou and C-T Zhang ldquoPrediction of protein structuralclassesrdquo Critical Reviews in Biochemistry andMolecular Biologyvol 30 no 4 pp 275ndash349 1995
[53] W Chen and H Lin ldquoIdentification of voltage-gated potassiumchannel subfamilies from sequence information using supportvectormachinerdquoComputers in Biology andMedicine vol 42 no4 pp 504ndash507 2012
[54] K-C Chou and H-B Shen ldquoRecent progress in proteinsubcellular location predictionrdquo Analytical Biochemistry vol370 no 1 pp 1ndash16 2007
[55] P Feng H Lin W Chen and Y Zuo ldquoPredicting the typesof J-proteins using clustered amino acidsrdquo BioMed ResearchInternational vol 2014 Article ID 935719 8 pages 2014
[56] K-C Chou and H-B Shen ldquoProtIdent a web server foridentifying proteases and their types by fusing functionaldomain and sequential evolution informationrdquoBiochemical andBiophysical Research Communications vol 376 no 2 pp 321ndash325 2008
[57] K-C Chou and Y-D Cai ldquoPrediction of membrane proteintypes by incorporating amphipathic effectsrdquo Journal of ChemicalInformation and Modeling vol 45 no 2 pp 407ndash413 2005
[58] F Tom ldquoROC graphs notes and practical considerations forresearchersrdquo Machine Learning vol 31 pp 1ndash38 2004
[59] JHuang andCX Ling ldquoUsingAUCand accuracy in evaluatinglearning algorithmsrdquo IEEE Transactions on Knowledge andDataEngineering vol 17 no 3 pp 299ndash310 2005
[60] F D Foresee and M T Hagan ldquoGauss-Newton approximationto Bayesian learningrdquo in Proceedings of the International Con-ference on Neural Networks vol 3 pp 1930ndash1935 Houston TexUSA 1997
Computational and Mathematical Methods in Medicine 11
[33] S-Y Jiang and L-X Wang ldquoEfficient feature selection based oncorrelation measure between continuous and discrete featuresrdquoInformation Processing Letters vol 116 no 2 pp 203ndash215 2016
[34] M J Kearns and U V Vazirani An Introduction to Compu-tational Learning Theory MIT Press Cambridge Mass USA1994
[35] R Bellman Adaptive Control Processes A Guided Tour RANDCorporation Research Studies Princeton University PressPrinceton NJ USA 1961
[36] Z Pawlak ldquoRough set theory and its applications to dataanalysisrdquo Cybernetics and Systems vol 29 no 7 pp 661ndash6881998
[37] A-E Hassanien ldquoRough set approach for attribute reductionand rule generation a case of patients with suspected breastcancerrdquo Journal of the American Society for Information Scienceand Technology vol 55 no 11 pp 954ndash962 2004
[38] A Skowron and C Rauszer ldquoThe discernibility matricesand functions in information systemsrdquo in Intelligent DecisionSupport-Handbook of Applications and Advances of the RoughSetsTheory SystemTheory Knowledge Engineering and ProblemSolving R Słowinski Ed vol 11 pp 331ndash362 KluwerAcademicDordrecht Netherlands 1992
[39] Z Pawlak ldquoRough set approach to knowledge-based decisionsupportrdquo European Journal of Operational Research vol 99 no1 pp 48ndash57 1997
[40] X Wang J Yang X Teng W Xia and R Jensen ldquoFeatureselection based on rough sets and particle swarm optimizationrdquoPattern Recognition Letters vol 28 no 4 pp 459ndash471 2007
[41] C Ma J Ouyang H-L Chen and X-H Zhao ldquoAn efficientdiagnosis system for Parkinsonrsquos disease using kernel-basedextreme learning machine with subtractive clustering featuresweighting approachrdquo Computational and Mathematical Meth-ods in Medicine vol 2014 Article ID 985789 14 pages 2014
[42] A H El-Baz ldquoHybrid intelligent system-based rough set andensemble classifier for breast cancer diagnosisrdquoNeural Comput-ing and Applications vol 26 no 2 pp 437ndash446 2015
[43] J-H Eom S-CKim andB-T Zhang ldquoAptaCDSS-E a classifierensemble-based clinical decision support system for cardiovas-cular disease level predictionrdquo Expert Systems with Applicationsvol 34 no 4 pp 2465ndash2479 2008
[44] Y Ma Ensemble Machine Learning Methods and ApplicationsSpringer New York NY USA 2012
[45] S A Etemad and A Arya ldquoClassification and translation ofstyle and affect in human motion using RBF neural networksrdquoNeurocomputing vol 129 pp 585ndash595 2014
[46] Z Pawlak ldquoRough setsrdquo International Journal of Computer ampInformation Sciences vol 11 no 5 pp 341ndash356 1982
[47] Z Pawlak ldquoRough sets and intelligent data analysisrdquo Informa-tion Sciences vol 147 no 1ndash4 pp 1ndash12 2002
[48] Y Huang P J McCullagh and N D Black ldquoAn optimization ofReliefF for classification in large datasetsrdquoData and KnowledgeEngineering vol 68 no 11 pp 1348ndash1356 2009
[49] J Sun M-Y Jia and H Li ldquoAdaBoost ensemble for financialdistress prediction an empirical comparison with data fromChinese listed companiesrdquo Expert Systems with Applicationsvol 38 no 8 pp 9305ndash9312 2011
[50] R Kohavi and F Provost ldquoGlossary of termsrdquo Machine Learn-ing vol 30 no 2-3 pp 271ndash274 1998
[51] W Chen and H Lin ldquoPrediction of midbody centrosome andkinetochore proteins based on gene ontology informationrdquoBiochemical and Biophysical Research Communications vol 401no 3 pp 382ndash384 2010
[52] K-C Chou and C-T Zhang ldquoPrediction of protein structuralclassesrdquo Critical Reviews in Biochemistry andMolecular Biologyvol 30 no 4 pp 275ndash349 1995
[53] W Chen and H Lin ldquoIdentification of voltage-gated potassiumchannel subfamilies from sequence information using supportvectormachinerdquoComputers in Biology andMedicine vol 42 no4 pp 504ndash507 2012
[54] K-C Chou and H-B Shen ldquoRecent progress in proteinsubcellular location predictionrdquo Analytical Biochemistry vol370 no 1 pp 1ndash16 2007
[55] P Feng H Lin W Chen and Y Zuo ldquoPredicting the typesof J-proteins using clustered amino acidsrdquo BioMed ResearchInternational vol 2014 Article ID 935719 8 pages 2014
[56] K-C Chou and H-B Shen ldquoProtIdent a web server foridentifying proteases and their types by fusing functionaldomain and sequential evolution informationrdquoBiochemical andBiophysical Research Communications vol 376 no 2 pp 321ndash325 2008
[57] K-C Chou and Y-D Cai ldquoPrediction of membrane proteintypes by incorporating amphipathic effectsrdquo Journal of ChemicalInformation and Modeling vol 45 no 2 pp 407ndash413 2005
[58] F Tom ldquoROC graphs notes and practical considerations forresearchersrdquo Machine Learning vol 31 pp 1ndash38 2004
[59] JHuang andCX Ling ldquoUsingAUCand accuracy in evaluatinglearning algorithmsrdquo IEEE Transactions on Knowledge andDataEngineering vol 17 no 3 pp 299ndash310 2005
[60] F D Foresee and M T Hagan ldquoGauss-Newton approximationto Bayesian learningrdquo in Proceedings of the International Con-ference on Neural Networks vol 3 pp 1930ndash1935 Houston TexUSA 1997
