Feature selection based on rough sets and particle swarm optimization Xiangyang Wang a, * , Jie Yang a , Xiaolong Teng a , Weijun Xia b , Richard Jensen c a Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong University, Shanghai 200030, China b Institute of Automation, Shanghai Jiao Tong University, Shanghai 200030, China c Department of Computer Science, The University of Wales, Aberystwyth, Ceredigion, SY23 3DB Wales, UK Received 28 December 2004; received in revised form 24 March 2006 Available online 7 November 2006 Communicated by S.K. Pal Abstract We propose a new feature selection strategy based on rough sets and particle swarm optimization (PSO). Rough sets have been used as a feature selection method with much success, but current hill-climbing rough set approaches to feature selection are inadequate at finding optimal reductions as no perfect heuristic can guarantee optimality. On the other hand, complete searches are not feasible for even medium-sized datasets. So, stochastic approaches provide a promising feature selection mechanism. Like Genetic Algorithms, PSO is a new evolutionary computation technique, in which each potential solution is seen as a particle with a certain velocity flying through the problem space. The Particle Swarms find optimal regions of the complex search space through the interaction of individuals in the population. PSO is attractive for feature selection in that particle swarms will discover best feature combinations as they fly within the subset space. Compared with GAs, PSO does not need complex operators such as crossover and mutation, it requires only primitive and simple mathematical operators, and is computationally inexpensive in terms of both memory and runtime. Experimentation is car- ried out, using UCI data, which compares the proposed algorithm with a GA-based approach and other deterministic rough set reduc- tion algorithms. The results show that PSO is efficient for rough set-based feature selection. Ó 2006 Elsevier B.V. All rights reserved. Keywords: Feature selection; Rough sets; Reduct; Genetic algorithms; Particle swarm optimization; Hill-climbing method; Stochastic method 1. Introduction In many fields such as data mining, machine learning, pattern recognition and signal processing, datasets contain- ing huge numbers of features are often involved. In such cases, feature selection will be necessary (Liu and Motoda, 1998; Guyon and Elisseeff, 2003). Feature selection is the process of choosing a subset of features from the original set of features forming patterns in a given dataset. The sub- set should be necessary and sufficient to describe target concepts, retaining a suitably high accuracy in representing the original features. The importance of feature selection is to reduce the problem size and resulting search space for learning algorithms. In the design of pattern classifiers it can improve the quality and speed of classification (Kudo and Sklansky, 2000). Due to the abundance of noisy, irrelevant or misleading features, the ability to handle imprecise and inconsistent information in real world problems has become one of the most important requirements for feature selection. Rough sets (Pawlak, 1982, 1991, 1997) can handle uncer- tainty and vagueness, discovering patterns in inconsistent data. Rough sets have been a useful feature selection method in pattern recognition (Chouchoulas and Shen, 2001). The rough set approach to feature selection is to select a subset of features (or attributes), which can predict the decision concepts as well as the original feature set. The optimal criterion for rough set feature selection is to find 0167-8655/$ - see front matter Ó 2006 Elsevier B.V. All rights reserved. doi:10.1016/j.patrec.2006.09.003 * Corresponding author. Tel./fax: +86 21 34204033. E-mail address: [email protected](X. Wang). www.elsevier.com/locate/patrec Pattern Recognition Letters 28 (2007) 459–471
Transcript
www.elsevier.com/locate/patrec
Pattern Recognition Letters 28 (2007) 459–471
Feature selection based on rough sets and particle swarm optimization
Xiangyang Wang a,*, Jie Yang a, Xiaolong Teng a, Weijun Xia b, Richard Jensen c
a Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong University, Shanghai 200030, Chinab Institute of Automation, Shanghai Jiao Tong University, Shanghai 200030, China
c Department of Computer Science, The University of Wales, Aberystwyth, Ceredigion, SY23 3DB Wales, UK
Received 28 December 2004; received in revised form 24 March 2006Available online 7 November 2006
Communicated by S.K. Pal
Abstract
We propose a new feature selection strategy based on rough sets and particle swarm optimization (PSO). Rough sets have been usedas a feature selection method with much success, but current hill-climbing rough set approaches to feature selection are inadequate atfinding optimal reductions as no perfect heuristic can guarantee optimality. On the other hand, complete searches are not feasible foreven medium-sized datasets. So, stochastic approaches provide a promising feature selection mechanism. Like Genetic Algorithms,PSO is a new evolutionary computation technique, in which each potential solution is seen as a particle with a certain velocity flyingthrough the problem space. The Particle Swarms find optimal regions of the complex search space through the interaction of individualsin the population. PSO is attractive for feature selection in that particle swarms will discover best feature combinations as they fly withinthe subset space. Compared with GAs, PSO does not need complex operators such as crossover and mutation, it requires only primitiveand simple mathematical operators, and is computationally inexpensive in terms of both memory and runtime. Experimentation is car-ried out, using UCI data, which compares the proposed algorithm with a GA-based approach and other deterministic rough set reduc-tion algorithms. The results show that PSO is efficient for rough set-based feature selection.� 2006 Elsevier B.V. All rights reserved.
In many fields such as data mining, machine learning,pattern recognition and signal processing, datasets contain-ing huge numbers of features are often involved. In suchcases, feature selection will be necessary (Liu and Motoda,1998; Guyon and Elisseeff, 2003). Feature selection is theprocess of choosing a subset of features from the originalset of features forming patterns in a given dataset. The sub-set should be necessary and sufficient to describe targetconcepts, retaining a suitably high accuracy in representingthe original features. The importance of feature selection is
0167-8655/$ - see front matter � 2006 Elsevier B.V. All rights reserved.
to reduce the problem size and resulting search space forlearning algorithms. In the design of pattern classifiers itcan improve the quality and speed of classification (Kudoand Sklansky, 2000).
Due to the abundance of noisy, irrelevant or misleadingfeatures, the ability to handle imprecise and inconsistentinformation in real world problems has become one ofthe most important requirements for feature selection.Rough sets (Pawlak, 1982, 1991, 1997) can handle uncer-tainty and vagueness, discovering patterns in inconsistentdata. Rough sets have been a useful feature selectionmethod in pattern recognition (Chouchoulas and Shen,2001). The rough set approach to feature selection is toselect a subset of features (or attributes), which can predictthe decision concepts as well as the original feature set. Theoptimal criterion for rough set feature selection is to find
460 X. Wang et al. / Pattern Recognition Letters 28 (2007) 459–471
shortest or minimal reducts while obtaining high qualityclassifiers based on the selected features (Swiniarski andSkowron, 2003). The number of rules generated by thereducts can also be considered as one of the criteria (Bazanet al., 2000).
There are many rough set algorithms for feature selec-tion. The most basic solution to finding minimal reductsis to generate all possible reducts and choose any with min-imal cardinality, which can be done by constructing a kindof discernibility function from the dataset and simplifyingit (Bazan et al., 2000; Komorowski et al., 1999). Starzykuses strong equivalence to simplify discernibility functions(Starzyk et al., 1998; Janusz et al., 2000). Obviously, this isan expensive solution to the problem and is only practicalfor very simple datasets. It has been shown that the prob-lem of minimal reduct generation is NP-hard and the prob-lem of generation of all reducts is exponential (Skowronand Rauszer, 1992). Therefore, heuristic approaches haveto be considered.
In general, there are two kinds of rough set methods forfeature selection, hill-climbing (or greedy) methods and sto-chastic methods (Vafaie and Imam, 1994). The hill-climbingapproaches usually employ rough set attribute significanceas heuristic knowledge. They start off with an empty setor attribute core and then adopt forward selection or back-ward elimination. Forward selection adds in turn, one at atime, the most significant attribute from the candidate set,until the selected set is a reduct. Backward elimination isthe reverse, starting with the full attribute set and removingattributes incrementally. Hu gives a reduction algorithmusing the positive region-based attribute significance asthe guiding heuristic (X. Hu and Cereone, 1995; X. Hu,1995). Wang develops a conditional information entropy-based reduction algorithm, using conditional entropy-basedattribute significance (Wang et al., 2002; Wang and Zhao,2004). Approximate reducts and approximate entropyreducts are also studied by researches, which can be usefulin cases when a smaller number of condition attributes ispreferred over accuracy of classification (Slezak, 1996; Sle-zak and Wroblewski, 2003; Pawlak, 1991). Hu computes thesignificance of an attribute making use of heuristic ideasfrom discernibility matrices and proposes a heuristic reduc-tion algorithm (K. Hu et al., 2003). Susmaga considers bothindiscernibility and discernibility relations in attributereduction (Susmaga, 2004a,b).
The positive region and conditional entropy-basedmethods choose a minimal feature subset that fullydescribes all concepts in a given dataset. The discernibilitymatrix-based method is to select a feature subset with highdiscriminatory power, which guarantees the maximalbetween-class separability for the reduced data sets. Thesemethods consider the best candidate attribute, trying tofind a minimal reduct. However, hill-climbing methodsdo not guarantee to find an optimal or minimal reduct.As no perfect heuristic exists, there can be no guaranteeof optimality. Using attribute significance to discriminatebetween candidates may lead the search down a non-mini-
mal path. It is impossible to predict which combinations ofattributes will lead to an optimal reduct with the additionor deletion of single attributes.
Some researchers use stochastic methods for rough setfeature selection (Bazan et al., 2000). Wroblewski usesgenetic algorithms to find minimal reducts (Wroblewski,1995). He combines a genetic algorithm with a greedy algo-rithm to generate short reducts. However, it uses highlytime-consuming operations and cannot assure that theresulting subset is really a reduct. Bjorvand applies geneticalgorithms to compute approximate reducts (Bjorvand andKomorowski, 1997; Bjorvand, 1997). He takes Wroblew-ski’s work as a foundation, but makes several variationsand practical improvements both in speed and the qualityof approximation. To obtain a good initial populationfor the GA, Bjorvand includes the attribute core in all can-didates. In addition to this, he uses domain knowledge toget the average size of actual reducts and lets the numberof features in the candidates be similar to the number inthe reducts. Also, he allows the user to assign a relativeweight to each attribute when creating the initial popula-tion. To avoid wasting much processing power in a wrongsearch direction, he adopts a dynamic mutation rate that isproportional to the redundancy in the population, prevent-ing all individuals from becoming equal. Zhai proposes anintegrated feature extraction approach based on rough settheory and genetic algorithms (Zhai et al., 2002). Roughsets are used to perform consistency checks, concept for-mation and approximation. By calculating the lower andupper approximations, training data is split into certaintraining data and possible training data. Then, a GA dis-covers best rules from the data sets. The fitness functionis defined as the classification quality of the extracted rules.Ultimately, the features or attributes within rules withhighest indices are selected. The reduct generation algo-rithms based on genetic algorithms and special data struc-tures implemented in the rough set system RSES are quiteefficient (Skowron et al., 2005). Jensen finds minimal roughset reducts using another stochastic strategy, ant colonyoptimization (ACO) (Jensen and Shen, 2003).
Hill-climbing methods are more efficient when dealingwith little noise and a small number of interacting features,but are not assured of optimality. Stochastic methods canprovide a more robust solution at the expense of increasedcomputational effort (Vafaie and Imam, 1994). For systemswhere the optimal or minimal subset is required (perhapsdue to the cost of feature measurement), stochastic featureselection must be used.
In this article we propose a new feature selection mecha-nism, where one investigates how particle swarm optimiza-tion (PSO) can be applied to find optimal feature subsets orrough set reducts. PSO is a new evolutionary computationtechnique proposed by Kennedy and Eberhart (Kennedyand Eberhart, 1995a,b). The particle swarm concept wasmotivated from the simulation of social behavior of biolog-ical organisms. The original intent was to simulate graphi-cally the graceful but unpredictable movement of bird
X. Wang et al. / Pattern Recognition Letters 28 (2007) 459–471 461
flocking. The PSO algorithm mimics the behavior of flyingbirds and their means of information exchange to solveoptimization problems. Each potential solution is seen asa particle with a certain velocity, and ‘‘flies’’ through theproblem space. Each particle adjusts its flight according toits own flying experience and its companions’ flying experi-ence. The particle swarms find optimal regions of complexsearch spaces through the interaction of individuals in apopulation of particles. PSO has been successfully appliedto a large number of difficult combinatorial optimizationproblems; studies show that it often outperforms GeneticAlgorithms (Kennedy and Spears, 1998). PSO is particu-larly attractive for feature selection in that particle swarmswill discover the best feature combinations as they fly withinthe problem space. The performance of the proposed algo-rithm is evaluated using several UCI datasets. It can be seenthat PSO has a strong search capability in the problemspace and can discover optimal solutions quickly.
The rest of this paper is structured as follows. Section 2describes the fundamentals of rough set theory. The princi-ples of PSO and PSO for Rough set-based Feature Selec-tion algorithm (PSORSFS) are presented in Section 3.The effectiveness of the method is demonstrated, comparedwith other algorithms on UCI datasets and discussed inSection 4. Finally, Section 5 concludes the article.
The algorithms used in the comparison include the posi-tive region-based attribute reduction algorithm (POSAR)(Jensen and Shen, 2003; X. Hu, 1995; Wang and Zhao,2004), conditional entropy-based attribute reduction(CEAR) (Wang et al., 2002; Wang and Zhao, 2004), discern-ibility matrix-based attribute reduction (DISMAR) (K. Huet al., 2003) and GA-based attribute reduction (GAAR)(Wroblewski, 1995; Bazan, 1998; Bazan et al., 2000). Dueto paper length restrictions, we do not describe such algo-rithms here, more details can be found in the related references.
2. Rough set preliminaries
Rough set theory (Pawlak, 1991, 1997) is a new mathe-matical approach to imprecision, vagueness and uncer-tainty. In an information system, every object of theuniverse is associated with some information. Objects char-acterized by the same information are indiscernible withrespect to the available information about them. Any setof indiscernible objects is called an elementary set. Anyunion of elementary sets is referred to as a crisp set—other-wise a set is rough (imprecise, vague). A rough set is theapproximation of a vague concept by a pair of precise con-cepts, called lower and upper approximations. The lowerapproximation is a description of the domain objects whichare known with certainty to belong to the subset of interest,whereas the upper approximation is a description of theobjects which possibly belong to the subset. Relative to agiven set of attributes, a set is rough if its lower and upperapproximations are not equal.
The main advantage of rough set analysis is that itrequires no additional knowledge except for the supplied
data. Rough sets perform feature selection using only thegranularity structure of the data (Pal and Skowron, 2003;Jensen and Shen, 2003).
Let I = (U,A) be an information system, where U is theuniverse, a non-empty finite set of objects. A is a non-empty finite set of attributes. "a 2 A determines a functionfa : U! Va, where Va is the set of values of a. If P � A,there is an associated equivalence relation
INDðP Þ ¼ fðx; yÞ 2 U � U j8a 2 P ; faðxÞ ¼ faðyÞg ð1Þ
The partition of U, generated by IND(P) is denoted U/P. If(x,y) 2 IND(P), then x and y are indiscernible by attributesfrom P. The equivalence classes of the P-indiscernibilityrelation are denoted [x]P. The indiscernibility relation isthe mathematical basis of rough set theory.
Let X � U, the P-lower approximation PX and P-upperapproximation PX of set X can be defined as
PX ¼ fx 2 U j½x�P � Xg ð2Þ
PX ¼ fx 2 U j½x�P \ X 6¼ /g ð3Þ
Let P,Q � A be equivalence relations over U, then thepositive, negative and boundary regions can be defined as
POSP ðQÞ ¼[
X2U=Q
PX ð4Þ
NEGP ðQÞ ¼ U �[
X2U=Q
PX ð5Þ
BNDP ðQÞ ¼[
X2U=Q
PX �[
X2U=Q
PX ð6Þ
The positive region of the partition U/Q with respect to P,POSP(Q), is the set of all objects of U that can be certainlyclassified to blocks of the partition U/Q by means of P. Aset is rough (imprecise) if it has a non-empty boundaryregion.
An important issue in data analysis is discoveringdependencies between attributes. Dependency can bedefined in the following way. For P,Q � A, P dependstotally on Q, if and only if IND(P) � IND(Q). That meansthat the partition generated by P is finer than the partitiongenerated by Q. We say that Q depends on P in a degree k
(0 6 k 6 1), denoted P) kQ, if
k ¼ cP ðQÞ ¼jPOSP ðQÞjjU j ð7Þ
If k = 1, Q depends totally on P, if 0 < k < 1, Q dependspartially on P, and if k = 0 then Q does not depend onP. In other words, Q depends totally (partially) on P, ifall (some) objects of the universe U can be certainly classi-fied to blocks of the partition U/Q, employing P.
In a decision system the attribute set contains the condi-tion attribute set C and decision attribute set D, i.e.A = C ¨ D. The degree of dependency between conditionand decision attributes, cC(D), is called the quality of
462 X. Wang et al. / Pattern Recognition Letters 28 (2007) 459–471
approximation of classification, induced by the set of deci-sion attributes (Pawlak, 1997).
The goal of attribute reduction is to remove redundantattributes so that the reduced set provides the same qualityof classification as the original. A reduct is defined as asubset R of the conditional attribute set C such thatcR(D) = cC(D). A given decision table may have manyattribute reducts, the set of all reducts is defined as
In rough set attribute reduction, a reduct with minimalcardinality is searched for. An attempt is made to locatea single element of the minimal reduct set Redmin � Red
Redmin ¼ fR 2 Redj _ R0 2 Red; jRj 6 jR0jg ð9ÞThe intersection of all reducts is called the core, the ele-
ments of which are those attributes that cannot be elimi-nated. The core is defined as:
CoreðCÞ ¼\
Red ð10Þ
3. PSO for feature selection
3.1. The principle of PSO
Particle swarm optimization (PSO) is an evolutionarycomputation technique developed by Kennedy and Eber-hart (1995a,b). The original intent was to graphically sim-ulate the graceful but unpredictable movements of a flockof birds. Initial simulations were modified to form the ori-ginal version of PSO. Later, Shi introduced inertia weightinto the particle swarm optimizer to produce the standardPSO (Shi and Eberhart, 1998a; Eberhart and Shi, 2001).
PSO is initialized with a population of random solu-tions, called ‘particles’. Each particle is treated as a pointin an S-dimensional space. The ith particle is representedas Xi = (xi1,xi2, . . . ,xiS). The best previous position (pbest,the position giving the best fitness value) of any particleis recorded and represented as Pi = (pi1,pi2, . . . ,piS). Theindex of the best particle among all the particles in the pop-ulation is represented by the symbol ‘gbest’. The rate of theposition change (velocity) for particle i is represented asVi = (vi1,vi2, . . . ,viS). The particles are manipulated accord-ing to the following equation:
vid ¼w� vid þ c1 � randð Þ � ðpid � xidÞþ c2 �Randð Þ � ðpgd � xidÞð11Þ
xid ¼ xid þ vid ð12Þwhere d = 1,2, . . . ,S, w is the inertia weight, it is a positivelinear function of time changing according to the generationiteration. Suitable selection of the inertia weight provides abalance between global and local exploration, and results infewer iterations on average to find a sufficiently optimalsolution. The acceleration constants c1 and c2 in Eq. (11)represent the weighting of the stochastic acceleration termsthat pull each particle toward pbest and gbest positions.
Low values allow particles to roam far from target regionsbefore being tugged back, while high values result in abruptmovement toward, or past, target regions. rand( ) andRand( ) are two random functions in the range [0, 1].
Particles’ velocities on each dimension are limited to amaximum velocity, Vmax. It determines how large stepsthrough the solution space each particle is allowed to take.If Vmax is too small, particles may not explore sufficientlybeyond locally good regions. They could become trappedin local optima. On the other hand, if Vmax is too highparticles might fly past good solutions.
The first part of Eq. (11) provides the ‘‘flying particles’’with a degree of memory capability allowing the explora-tion of new search space areas. The second part is the ‘‘cog-nition’’ part, which represents the private thinking of theparticle itself. The third part is the ‘‘social’’ part, which rep-resents the collaboration among the particles. Eq. (11) isused to calculate the particle’s new velocity according toits previous velocity and the distances of its current posi-tion from its own best experience (position) and the group’sbest experience. Then the particle flies toward a new posi-tion according to Eq. (12). The performance of each parti-cle is measured according to a pre-defined fitness function.
We give the pseudo-code of the PSO algorithm here,with a graphic demonstration of PSO given in Fig. 1.
Algorithm PSO
Input:
m: the swarm size; c1, c2: positive acceleration con-stants; w: inertia weightMaxV: maximum velocity of particlesMaxGen: maximum generationMaxFit: maximum fitness value
Output:Pgbest: Global best position
BeginSwarms {xid,vid} = Generate(m); /* Initialize a popu-lation of particles with random positions and veloci-ties on S dimensions*/Pbest(i) = 0; i = 1, . . . ,m, d = 1, . . . ,S
Gbest = 0; Iter = 0;While(Iter < MaxGen and Gbest < MaxFit){For(every particle i)
Fig. 1. (a) PSO process. (b) PSO-based Feature Selection. The principle of updating velocity. Individual particles (1 and 2) are accelerated toward thelocation of the best solution, gbest, and the location of their own personal best, pbest, in the two dimension problem space (feature number andclassification quality).
X. Wang et al. / Pattern Recognition Letters 28 (2007) 459–471 463
}}Iter=Iter+1;
}/*rand() and Rand() are two random functions inthe range [0,1]*/Return P_{gbest}
End
3.2. PSO and rough set-based feature selection (PSORSFS)
We can use the idea of PSO for the optimal feature selec-tion problem. Consider a large feature space full of featuresubsets. Each feature subset can be seen as a point or posi-tion in such a space. If there are N total features, then therewill be 2N kinds of subset, different from each other in thelength and features contained in each subset. The optimalposition is the subset with least length and highest classifi-cation quality. Now we put a particle swarm into this fea-ture space, each particle takes one position. The particlesfly in this space, their goal is to fly to the best position.Over time, they change their position, communicate witheach other, and search around the local best and globalbest position. Eventually, they should converge on good,possibly optimal, positions. It is this exploration abilityof particle swarms that should better equip it to performfeature selection and discover optimal subsets.
To apply the PSO idea to feature selection, some mattersmust first be considered.
3.2.1. Representation of position
We represent the particle’s position as binary bit stringsof length N, where N is the total number of attributes.Every bit represents an attribute, the value ‘1’ means thecorresponding attribute is selected while ‘0’ not selected.Each position is an attribute subset.
3.2.2. Representation of velocityThe velocity of each particle is represented as a positive
integer, varying between 1 and Vmax. It implies how manyof the particle’s bits (features) should be changed, at a par-ticular moment in time, to be the same as that of the globalbest position, i.e. the velocity of the particle flying towardthe best position. The number of different bits betweentwo particles relates to the difference between their posi-tions. See Fig. 1(b) for the principle of velocity updating.
For example, Pgbest = [1 0 1 1 1 0 1 0 0 1], Pi = [0 1 0 0 11 0 1 0 1]. The difference between gbest and the particle’scurrent position is Pgbest � Pi = [1 �1 1 1 0 �1 1 �1 0 0].A value of 1 indicates that compared with the best position,this bit (feature) should be selected but is not, which willdecrease classification quality and lead to a lower fitnessvalue. Assume that the number of 1’s is a. On the otherhand, a value of �1 indicates that, compared with the bestposition, this bit should not be selected, but is selected.Redundant features will make the length of the subsetlonger and lead to a lower fitness value. The number of�1’s is b. We use the value of (a � b) to express the distancebetween two positions; (a � b) may be positive or negative.Such variation makes particles exhibit an exploration abil-ity within the solution space. In this example, (a � b) =4 � 3 = 1, so Pg � Pi = 1.
3.2.3. Position update strategies
After updating the velocity, a particle’s position will beupdated by the new velocity. Assume that the new velocityis V; the number of different bits between the current par-ticle and gbest is xg. Two cases exist when updating theposition:
(1) V 6 xg. In this case, the particle’s velocity is less than,or equal to, the position difference between the
464 X. Wang et al. / Pattern Recognition Letters 28 (2007) 459–471
particle and gbest. V bits of the particle are randomlychanged, different from that of gbest. The particlethen moves toward the global best while still explor-ing the search space, instead of simply being same asgbest.
(2) V > xg. In this case, the particle’s velocity overrunsthe position difference between the particle andgbest. In addition to changing all the different bitsto be same as that of gbest, we should further ran-domly (‘random’ implies ‘exploration ability’) change(V � xg) bits outside the different bits between theparticle and gbest. So after the particle reaches theglobal best position, it keeps on moving some dis-tance toward other directions, enabling furthersearch.
The maximum velocity Vmax serves as a constraint tocontrol the global exploration ability of a particle swarm.A larger Vmax facilitates global exploration, while a smallerVmax encourages local exploitation. When Vmax is too low,particles have more difficulty escaping from locally optimalregions. If Vmax is too high, particles might fly past goodsolutions (Kennedy, 1997).
In our experimentation, we initially limited the particles’velocity in the region [1,N]. However, it was noticed thatafter several generations the swarms converged to a goodbut non-optimal solution, and in the following generationsthe gbest remained stationary. This indicates that the veloc-ity is too high and particles often ‘fly past’ the optimalsolution.
We set Vmax = (1/3) * N and limit the velocity within therange [1, (1/3) * N], which prevents an overly-large velocity.A particle can be near to an optimal solution, but a highvelocity may make it move far away. By limiting the max-imum velocity, particles cannot fly too far away from theoptimal solution. Once finding a global best position, otherparticles will adjust their velocities and positions, searchingaround the best position. After many tests, we found thatan appropriate maximum velocity value is (1/3) * N. IfV < 1, then V = 1. If V > (1/3) * N, V = (1/3) * N. PSOcan often find the optimal solution quickly under such alimit.
3.2.5. Fitness function
We define the fitness function in Eq. (13)
Fitness ¼ a�cRðDÞ þ b�jCj � jRjjCj ð13Þ
Table 1PSORSFS and GAAR parameter settings
Population Generation Crossover probability M
GA 100 100 0.6 0PSO 20 100 – –
where cR(D) is the classification quality of condition attri-bute set R relative to decision D, jRj is the ‘1’ number ofa position or the length of selected feature subset. jCj isthe total number of features. a and b are two parameterscorresponding to the importance of classification qualityand subset length, a 2 [0,1] and b = 1 � a.
This formula means that the classification quality andfeature subset length have different significance for featureselection task. In our experiment we assume that classifica-tion quality is more important than subset length and seta = 0.9, b = 0.1. The high a assures that the best positionis at least a real rough set reduct. The goodness of eachposition is evaluated by this fitness function. The criteriaare to maximize fitness values.
4. Experimental results and discussions
We implement the PSORSFS algorithm and other fourfeature selection algorithms in MatLab 6.5. The computeris Intel P4, 2.66 GHz CPU; 512 MB RAM and the systemis Windows XP Professional. The five algorithms are testedand compared on 27 discrete UCI datasets (Blake et al.,1998). The two algorithms, GAAR and PSORSFS requireadditional parameter settings for their operation. These aregiven in Table 1.
In PSORSFS, the inertia weight decreases along withthe iterations, varying from 1.4 to 0.4 according to Eq.(14).
where MAXITER is the maximum iteration (generation)and Iter is the current iteration.
For 14 of 27 datasets, the five algorithms find the samereducts. These are listed in Table 2. The ‘Features’ columnand ‘Instances’ column give out the total number of fea-tures (attributes) and instances in the datasets. The numberof decision rules and the classification accuracy with differ-ent reducts are also shown. We use the LEM2 algorithm(Stefanowski, 1998) to extract rules from the data andthe global strength (Bazan, 1998; Bazan et al., 2000) forrule negotiation in classification. We apply ten-fold crossvalidation to estimate the classification accuracy. We alsoshow the time in seconds for 100 generations necessaryfor the generation of reducts by PSORSFS.
The experimental results of the other 13 datasets arelisted in Table 3. The five rough set reduction algorithmsare compared. We present the best solution (in terms offeature subset length) each algorithm finds. For these data-sets, some may have more than one optimal reduct; some
utation probability c_1 c_2 Weight Velocity
.4 – – – –2.0 2.0 1.4–0.4 1–(1/3) * N
Table 2Experimental results on 14 datasets
Dataset Features Instances Reduct size Number of rules Classification accuracy (%) Time of PSORSFS (s)
X. Wang et al. / Pattern Recognition Letters 28 (2007) 459–471 465
however have only one (exclusive) optimal reduct. Fromthe results, it can be seen that in some situations hill-climb-ing methods can locate the optimal solution. For example,POSAR finds the exclusive optimal solution for datasetExactly2 and Lymphography. DISMAR finds the exclusiveoptimal solution for dataset Exactly2 and Vote. But forother datasets, sub-optimal solutions are found, containingredundant features. CEAR often contains more redundantfeatures than POSAR and DISMAR.
As for the stochastic search algorithms, GAAR andPSORSFS, from the experimental results it can be seenthat PSORSFS performs better than GAAR. PSORSFSsuccessfully finds the optimal reducts on most of the data-sets. For example, PSORSFS finds an optimal reduct forthe Mushroom data, and finds the exclusive optimal reductfor Exactly2, Vote and Led.
During the experiments, the rough ordering of tech-niques with respect to time is: POSAR < CEAR < PSOR-SAR < GAAR < DISMAR. DISMAR takes a significantamount of time for the computation of the discernibility
matrix, the time increasing quickly with increasing numberof instances in the dataset. CEAR may cost time on com-puting equivalence classes. The stochastic algorithms needtime in generation iterations.
Let N be the number of features (conditional attributes)and M the total objects. The time complexity of POSAR isO(NM2) (Nguyen, 1996; X. Hu, 1995), and that of thereduction based on conditional information entropy(CEAR) is O(NM2) + O(M3), which is composed of thecomputation of core and non-core attribute reduct (Wanget al., 2002). DISMAR has total time complexity O((N +logM)M2) (K. Hu et al., 2003). For GAAR and PSORSFS,the complexity of the fitness function is O(NM2) (Wroblew-ski, 1995). The other impact on time is the number of gen-eration iterations. For the GA method, the crossover andmutation operations also take much time. As for PSO-RSFS, time is mainly spent on evaluating the particles’position (fitness function).
To graphically illustrate the progress of the particleswarm as it searches for optimal solutions, we take
466 X. Wang et al. / Pattern Recognition Letters 28 (2007) 459–471
generation as the horizontal coordinate and the fitnessvalue as the vertical coordinate. This should illustrate theprocess of improvement of the global best as the numberof generations increase. To further highlight the searchprocess, we graph subset length of every particle’s currentposition (horizontal coordinate) against classification qual-ity (vertical coordinate). Each point in the figure is aparticle.
The example for Exactly2 is given below and the otherexamples are listed in Appendix A.
Table 4Classification results with different reducts
Example 1. The process of the particle swarms searchingfor optimal solutions for dataset Exactly2 is given in Table6 and Figs. 2 and 3. In Table 6, ‘‘Best Solution’’ lists thebest feature subset encountered at a particular iteration, inwhich each number denotes one feature of the dataset.
The evaluation criteria or fitness function is a criticalissue in the implementation of stochastic algorithms. Inthe fitness function we use, because the classification qual-ity parameter outclasses that of subset length (a = 0.9,
Fig. 2d. Generation 12 of PSO on dataset Exactly2.
Fig. 2e. Generation 13 of PSO on dataset Exactly2.
Fig. 3. Evolution process of the global best on dataset Exactly2(PSORSFS, GAAR, CEAR).
X. Wang et al. / Pattern Recognition Letters 28 (2007) 459–471 467
468 X. Wang et al. / Pattern Recognition Letters 28 (2007) 459–471
b = 0.1), the optimal solution is assured to be a real roughset reduct or super-reduct. So, the fitness function in PSO isefficient.
From the results and figures, we can see that, comparedwith GA, PSO is quicker in locating the optimal solution.In general, it can find the optimal solution within tens ofgenerations. If exhaustive search is used to find the optimalreduct in the dataset DNA, there will be tens of thousandsof candidate subsets, which is impossible to execute. Butwith PSO, at the 32nd generation the optimal solution isfound.
PSO has a powerful exploration ability; it is a gradualsearching process that approaches optimal solutions. Therunning time of PSO is affected less by the problem dimen-sion (feature numbers), but more by the size of data (seeTables 2 and 4). Mushroom is the largest dataset andrequires the largest time for PSORSFS (14,176 s). Mostof the time is spent on the basic computations of rough sets(computing equivalence classes). For some datasets withmore features, for example Lung and DNA, after findinga sub-optimal solution, the GA cannot find a better one:the fitness/generation graph is a line. However, PSO cansearch in the feature space until the optimal solution isfound. The GA is affected greatly by the number offeatures.
PSO comprises a very simple concept, and the ideas canbe implemented in a few lines of computer code. It requiresonly primitive mathematical operators, and is computa-tionally inexpensive in terms of both memory requirementsand speed. This optimization technique does not suffer,however, from some of the difficulties of GAs; interactionin the group enhances rather than detracts from progresstoward the solution. Further, a particle swarm system hasmemory, which the genetic algorithm does not have.Changes in genetic populations result in the destructionof previous knowledge of the problem. In particle swarmoptimization, individuals who fly past optima are tuggedto return towards them; knowledge of good solutions isretained by all particles (Kennedy and Eberhart, 1995b).
As for PSO, every particle flies in the candidate problemspace, adjusts their velocity and position according to thelocal best and global best. So, all the particles have a pow-erful search capability, which can help the swarm avoiddead ends.
The comparison of the number of decision rules and theclassification accuracy with different reducts are shown inTable 4. We show the time in seconds for 100 generationsnecessary for generation of reducts by PSORSFS.
The results are also compared with the rough set systemRSES (Skowron et al., 2005a,b) (see Table 5). RSES is anexcellent toolset for analyzing data with the use of methodsbased on Rough Set Theory, and is developed by research-ers in Warsaw University. RSES have some reducts calcu-lation methods, such as Exhaustive algorithm, Geneticalgorithm, Calculate dynamic reducts etc. We use theExhaustive algorithm to find reducts. It presents two deter-ministic algorithms for the computation of the whole
reduct set, both algorithms compute the discernibilitymatrix for this purpose (Bazan et al., 2000). The complexityis exponential. The time complexity to find all reducts isO(2NT), where T is the computational cost of finding onereduct, and N is the number of attributes (Bell and Guan,1998). For the last three datasets (Soybean-small, Lungand DNA), RSES cannot find all reducts by the exhaustivealgorithm, due to memory limitations. As a result of this,we use a fast genetic algorithm to find 10 reducts. We usethe decision rules classifier, LEM2 algorithm for globalrules generation and 10-fold cross-validation estimationmethod. We set ‘‘Shortening ratio’’ to 0.9, in order to geta short and minimal rule set. Conflicts are resolved by‘‘Standard Voting’’. Most of the reducts found by PSO-RSFS result in smaller rules and exhibit higher classifica-tion accuracy.
5. Conclusion
This paper discusses the shortcomings of conventionalhill-climbing rough set approaches to feature selection.These techniques often fail to find optimal reductions, asno perfect heuristic can guarantee optimality. On the otherhand, complete searches are not feasible for even medium-sized datasets. So, stochastic approaches provide a promis-ing feature selection mechanism.
We propose a new optimal feature selection techniquebased on rough sets and particle swarm optimization(PSO). PSO has the ability to quickly converge (Shiand Eberhart, 1999), it has a strong search capability inthe problem space and can efficiently find minimalreducts. Experimental results demonstrate competitive per-formance. PSO is a promising method for rough setreduction.
More experimentation and further investigation intothis technique may be required. The inertia weight (w)and maximum velocity (Vmax) have an important impacton the performance of PSO. The selection of the parame-ters may be problem-dependent. Vmax serves as a constraintthat controls the maximum global exploration ability PSOcan have. In many practical problems, it’s difficult to selectthe best Vmax without trial-and-error (Shi and Eberhart,1998b). In our feature selection problem, we first limitthe particles’ velocity to N, since [1, N] is the dynamic rangeof the feature space of each particle. But we find that undersuch a limitation, particles have poor local explorationability. So after many tests, we set Vmax to (1/3) * N, whichis suitable for our problem (see Section 3.2). The inertiaweight balances the global and local exploration abilities.In our experiments, we let it decrease from 1.4 to 0.4 alongwith the iterations. The performance of the PSO algorithmwith linearly decreasing inertia weight can be improvedgreatly and have better results. The larger inertia weightsat the beginning help to find good seeds and the later smallinertia weights facilitate fine search (Shi and Eberhart,1998b, 1999).
Table 8PSO searching process on Mushroom
Iter Best solution Fitness value Feature subset length
X. Wang et al. / Pattern Recognition Letters 28 (2007) 459–471 469
In this paper, we apply PSO to find reducts of minimalcardinality and, like classical genetic algorithms, the parti-cle’s position is a binary representation for attribute sub-sets. An extension to the approach would be to selectreducts due to the number of decision rules it generatesrather than its length alone. If a reduct generates fewerrules, it means that the rules are more general and theyshould better recognize new objects (Bazan et al., 2000).We should also extend to hybrid algorithms (Wroblewski,1995, 1996), order-based PSO for searching approximateentropy reducts (Slezak and Wroblewski, 2003), wherethe particle’s position is a permutation of attributes andPSO is used to find the proper order. Such reducts aremuch more applicable in practice. The fitness functionand position-updating strategy are also key factors inPSO for feature selection, which need to be improved fur-ther. For large data sets, to speed up the computations ofreducts, parallel algorithm may be employed (Susmaga,1998, 2004b).
Appendix A. Experimental examples
Example 2. The process of the particle swarms searchingfor optimal solutions for dataset Vote is given in Table 7and Fig. 4.
Table 7PSO searching process on Vote
Iter Best solution Fitness value Feature subset length
470 X. Wang et al. / Pattern Recognition Letters 28 (2007) 459–471
Example 5. The process of the particle swarms searchingfor optimal solutions for dataset Lung is given in Table10 and Fig. 7.
Example 6. The process of the particle swarms searchingfor optimal solutions for dataset DNA is given in Table11 and Fig. 8.
References
Bazan, J., 1998. A comparison of dynamic and non-dynamic rough setmethods for extracting laws from decision table. In: Polkowski, L.,Skowron, A. (Eds.), Rough Sets in Knowledge Discovery. Physica-Verlag, Heidelberg, pp. 321–365.
Bazan, J., Nguyen, H.S., Nguyen, S.H., Synak, P., Wroblewski, J., 2000.Rough set algorithms in classification problem. In: Polkowski, L.,Tsumoto, S., Lin, T.Y. (Eds.), Rough Set Methods and Applications.Physica-Verlag, Heidelberg, New York, pp. 49–88.
X. Wang et al. / Pattern Recognition Letters 28 (2007) 459–471 471
Bell, D., Guan, J., 1998. Computational methods for rough classificationand discovery. J. ASIS 49 (5), 403–414.
Bjorvand, A.T., 1997. ‘Rough Enough’—a system supporting the roughsets approach. In: Sixth Scandinavian Conference on ArtificialIntelligence SCAI’97.
Bjorvand, A.T., Komorowski, J., 1997. Practical Applications of GeneticAlgorithms for Efficient Reduct Computation, Vol. 4. Wissenschaft &Technik Verlag, pp. 601–606.
Blake, C., Keogh, E., Merz, C.J., 1998. UCI repository of machinelearning databases. Technical Report, Department of Informationand Computer Science, University of California, Irvine, CA. <http://www.ics.uci.edu/mlearn/MLRepository.htm>.
Eberhart R.C., Shi, Y., 2001. Particle swarm optimization: developments,applications and resources. In: Proceedings of IEEE InternationalConference on Evolutionary Computation. Seoul, pp. 81–86.
Guyon, I., Elisseeff, A., 2003. An introduction to variable and featureselection. J. Mach. Learning Res. 3, 1157–1182.
Hu, X., 1995. Knowledge discovery in databases: an attribute-orientedrough set approach, Ph.D. thesis, Regina University.
Hu, X., Cereone, N., 1995. Learning in relational databases: a rough setapproach. Comput. Intell. 11 (2), 323–337.
Hu, K., Lu, Y.C., Shi, C.Y., 2003. Feature ranking in rough sets. AICommun. 16 (1), 41–50.
Janusz, A., Starzyk, J., Nelson, D.E., Sturtz, K., 2000. A mathematicalfoundation for improved reduct generation in information systems.Knowledge Informat. Syst. 2, 131–146.
Jensen, R., Shen, Q., 2003. Finding rough set reducts with ant colonyoptimization. In: Proceedings of the 2003 UK Workshop on Compu-tational Intelligence, pp. 15–22.
Kennedy, J., 1997. The particle swarm: social adaptation of knowledge.In: IEEE International Conference on Evolutionary Computation,April 13–16, pp. 303–308.
Kennedy, J., Eberhart, R.C., 1995a. Particle swarm optimization. In:Proceedings of IEEE International Conference on Neural Networks,Perth, pp. 1942–1948.
Kennedy, J., Eberhart, R.C., 1995b. A new optimizer using particle swarmtheory. In: Sixth International Symposium on Micro Machine andHuman Science. Nagoya, pp. 39–43.
Kennedy, J., Spears, W.M., 1998. Matching algorithms to problems: anexperimental test of the particle swarm and some genetic algorithms onthe multimodal problem generator. In: Proceedings of the IEEEInternational Conference on Evolutionary Computation. pp. 39–43.
Komorowski, J., Pawlak, Z., Polkowski, L., Skowron, A., 1999. Roughsets: a tutorial. In: Pal, S.K., Skowron, A. (Eds.), Rough FuzzyHybridization. A New Trend in Decision-Making. Springer-VerlagSingapore Pte., Ltd., Singapore, pp. 3–98.
Kudo, M., Sklansky, J., 2000. Comparison of algorithms that selectfeatures for pattern classifiers. Pattern Recognition 33 (1), 25–41.
Liu, H., Motoda, H., 1998. Feature Selection for Knowledge Discoveryand Data Mining. Kluwer Academic Publishers.
Nguyen, H.S., 1996. Some efficient algorithms for rough set methods.In: Proceedings of the Sixth International Conference, InformationProcessing and Management of Uncertainty in Knowledge-BasedSystems (IPMU’96) 2, July 1–5, 1996, Granada, Spain pp. 1451–1456.
Pal, S.K., Skowron, A. (Eds.), 2003. Pattern Recognition Lett. 24 (6), 829–933 (special issue on Rough Sets).
Pawlak, Z., 1997. Rough set approach to knowledge-based decisionsupport. Eur. J. Operat. Res. 99, 48–57.
Shi, Y., Eberhart, R.C., 1998a. A modified particle swarm optimizer.In: Proc. IEEE Int. Conf. on Evolutionary Computation. Anchorage,AK, USA, pp. 69–73.
Shi, Y., Eberhart, R.C., 1998b. Parameter selection in particle swarmoptimization. In: Evolutionary Programming VII: Proc. EP98.Springer-Verlag, New York, pp. 591–600.
Shi, Y., Eberhart, R.C., 1999. Empirical study of particle swarmoptimization. In: Proc. 1999 Congress on Evolutionary Computation.IEEE Service Center, Piscataway, NJ, pp. 1945–1950.
Skowron, A., Rauszer, C., 1992. The discernibility matrices and functionsin information systems. In: Slowinski, R. (Ed.), Intelligent DecisionSupport—Handbook of Applications and Advances of the Rough SetsTheory. Kluwer Academic Publishers, Dordrecht, pp. 311–362.
Skowron, A., Bazan, J., Son, N.H., Wroblewski, J., et al., 2005a. RSES 2.2User’s Guide. Institute of Mathematics, Warsaw University, Warsaw,Poland, January 19, 2005. <http://logic.mimuw.edu.pl/~rses>.
Skowron, A., Wang, H., Wojna, A., Bazan J., 2005b. A hierarchicalapproach to multimodal classification. In: Slezak, D., Wang, G.,Szczuka, M., Duentsch, I., Yao, Y.Y., (Eds.), Rough sets, fuzzy sets,data mining, and granular computing. In: Proc. 10th Int. Conf.RSFDGrC 2005, Regina, Canada, September 1–3, 2005, Part 2,Lecture Notes in Artificial Intelligence 3642, Springer, Heidelberg,2005, pp. 119–127.
Slezak, D., 1996. Approximate reducts in decision tables. In: Proc. ofIPMU’96, 1996.
Slezak, D., Wroblewski, J., 2003. Order based genetic algorithms for thesearch of approximate entropy reducts. In: Wang, G.Y., et al. (Eds.),RSFDGrC. LNAI, Vol. 2639. Chongqing, China, 2003, pp. 308–311.
Starzyk, J., Nelson, D.E., Sturtz, K., 1998. Reduct generation ininformation systems. Bull. Int. Rough Set Society 3, 19–22.
Stefanowski, J., 1998. On rough set based approaches to induction ofdecision rules. In: Skowron, A., Polkowski, L. (Eds.), Rough Sets inKnowledge Discovery, Vol. 1. Physica Verlag, Heidelberg, pp. 500–529.
Susmaga, R., 1998. Parallel computation of reducts. In: Polkowski, L.,Skowron, A. (Eds.), RSCTC’98, LNAI 1424, 1998, pp. 450–458.
Susmaga, R., 2004a. Reducts and constructs in attribute reduction.Fundamenta Informaticae, 61(2). IOS Press, 159–181.
Susmaga, R., 2004b. Tree-Like Parallelization of Reduct and ConstructComputation. In: Tsumoto, S., et al. (Eds.), RSCTC 2004, LNAI 3066,2004, pp. 455–464.
Swiniarski, R.W., Skowron, A., 2003. Rough set methods in featureselection and recognition. Pattern Recognition Lett. 24 (6), 833–849.
Vafaie, H., Imam, I.F., 1994. Feature selection methods: geneticalgorithms vs. greedy-like search. In: Proc. Int. Conf. on Fuzzy andIntelligent Control Systems.
Wang, G.Y., Yu, H., Yang, D.C., 2002. Decision table reduction based onconditional information entropy. Chin. J. Comput. 25 (7), 759–766.
Wang, G.Y., Zhao, J., 2004. Theoretical study on attribute reduction ofrough set theory: comparison of algebra and information views. In:Proc. Third IEEE Int. Conf. on Cognitive Informatics.
Wroblewski, J., 1995. Finding minimal reducts using genetic algorithms.In: Proc. Second Annual Join Conf. on Information Sciences,Wrightsville Beach, NC. September 28–October 1, pp. 186–189.
Zhai, L.Y. et al., 2002. Feature extraction using rough set theory andgenetic algorithms—an application for the simplification of productquality evaluation. Comput. Industrial Eng. 43, 661–676.
