A distributed PSO–SVM hybrid system with feature selection and parameter
optimization
Cheng-Lung Huang & Jian-Fan Dun
Soft Computing 2008
Introduction
Hybridizing the particle swarm optimization (PSO) and support vector machines (SVM) to improve the classification accuracy with a small and appropriate feature subset.
Combining the discrete PSO with the continuous-valued PSO
Implementing via a distributed architecture using the web service technology to reduce the computational time.
Introduction
The continuous-valued version is used to optimize the best SVM model parameters.
The discrete version is used to search the optimal feature subset.
PSO can be easily adopted for parallel processing by distributed system.
Support Vector Machine
Kernel Function: RBF (C and Gamma ) Multi-class strategies:
one-against-one (adapt in this study)
one-against-all
Particle swarm optimization
Rnd( ) is a random function in the range[0, 1] Positive constant c1 and c2 are personal and social
learning factors. w is the inertia weight and Inertia weight balances the
global exploration and local exploitation. Pi,d denote the best previous position encountered by the
ith particle. Pg,d denotes the global best position thus far. t denotes the iteration counter.
Particle swarm optimization
The new position of a particle is calculated using the following formula:
Binary PSO
The function S(v) is a sigmoid limiting transformation and rnd( ) is a random number selected from a uniform distribution in [0, 1].
Particle representation
Features mask (discrete-valued) C (continuous-valued) Gamma (continuous-valued)
Fitness definition
WA: SVM classification accuracy weight acci: SVM classification accuracy WF: weight of the features f j :the value of feature mask-‘‘1’’represents that
feature j is selected and ‘‘0’’ represents that feature j is not selected.
nF : the total number of features.
Strategies for setting the inertia weight
Data descriptions
There are eight target classes that need to be classified in this data set.
The data set has 30 features that only five of them (f5, f10, f15, f20, and f25) are relevant to the eight classes.
Experimental procedures
Randomly split the data into ten groups using stratified 10-fold cross validation.
Each group contains training, validation and test sets.
The training set is used to build the SVM model. The validation set is used to determine the proper
training iteration to avoid overtraining The test set is used to evaluate the model’s
classification accuracy.
Setting of the system parameters
Experimental procedures
Experimental procedures
Experimental procedures
Experimental results
Experimental results
HITF : the number of hits on correct features. COVERF : the number of times the selected
feature subset covered the correct features. RATIOF : the ratio of correct features for the ten
experiments (10-fold CV).
Experimental results
f : denote the selected feature subset by the PSO. F : denote correct discriminating features (f5, f10, f15,
f20,and f25 in this experiment),
Experimental results
Fitness
Distributed architectures
CPU Time
Conclusions
Input feature subset selection and the kernel parameters setting are crucial problems.
This study proposed a new hybrid PSO–SVM system to solve these two problems.
To overcome the long training time when dealing with a large-scale dataset, the PSO–SVM can be implemented with a distributed parallel architecture.
Thank You