MetahAtt: Metaheuristicbased optimization extension

Sandro Radovanović, Milan Vukićević, Elena Milovanović, Milena Popović

University of Belgrade, Faculty of Organizational Sciences

Introduction

• Data mining models can be improved by adequate feature selection and/or parameter optimization

• Search for optimal feature/parameter selection is computationally and time consuming.

• Metaheuristics that are implemented are: random search, simulated annealing, hill climbing, variable neighbourhood search, harmony search and iterated local search.

The Problem• Attribute Selection as an optimization problem

• Maximize: Performance of Classification, Wi

• Classification: Classify (W, ATT)

• Performance: PerformanceMeasure(classify (W, ATT))

• Solution

• Search, list all possible values?

• Iterative Methods, e.g. Newton’s Method

• Complexity

• O(n)

• P, NP, NP-Hard

Value Combination of Weights

Performance of Classification

Metaheuristic based optimization

• A metaheuristic algorithms are used to control the execution of a simpler heuristic method, and unlike a one-pass heuristic, does not automatically terminate once a locally optimal solution is found.

Metaheuristic Example

Initial evaluation

• Several meta-heuristic wrapper attribute weighting techniques for Naïve Bayes were evaluated and compared.

• Experiments are conducted on 25 datasets obtained on UCI Machine Learning repository.

• A method for generalization of the results based on the analysis of the dependence between attribute weighting performances and dataset descriptions (meta-features).

• General idea of this research is to identify good performing wrapper optimization techniques for new datasets without the need for evaluation of all attribute optimization techniques.

Evaluation

• First part: For clustering of datasets we used K-means algorithm.

• In order to create better clustering several models with different number of clusters (from 2 to 5) and cluster quality is measured by Davies-Bouldinindex .

• Second part: In order to prevent over training by feature optimization, each wrapper was trained and evaluated on 70% of dataset with 10-fold cross validation. Final evaluation of weighted Naïve Bayes performance is done on 30% of the data (unseen cases).

Cluster centroids

RapidMiner process

Results

Dataset Algorithm ClusterDefault SA VNS ES PSO

anneal 0.614(±0.042) 0.620(±0.059) 0.574(±0.030) 0.797(±0.117) 0.934(±0.152) Cl. 1anneal-orig 0.604(±0.046) 0.636(±0.038) 0.657(±0.033) 0.818(±0.167) 0.931(±0.177) Cl. 1audiology 0.801(±0.052) 0.783(±0.052) 0.676(±0.077) 0.801(±0.071) 0.801(±0.134) Cl. 1kr-vs-kp 0.877(±0.020) 0.837(±0.020) 0.790(±0.018) 0.878(±0.041) 0.913(±0.063) Cl. 1mushroom 0.994(±0.003) 0.978(±0.002) 0.941(±0.002) 0.997(±0.031) 0.999(±0.177) Cl. 1postoperative-patient-data 0.389(±0.090) 0.422(±0.083) 0.344(±0.211) 0.422(±0.216) 0.556(±0.202) Cl. 1splice 0.955(±0.006) 0.934(±0.005) 0.892(±0.006) 0.955(±0.024) 0.956(±0.054) Cl. 1breast cancer 0.714(±0.104) 0.707(±0.082) 0.717(±0.086) 0.725(±0.102) 0.718(±0.094) Cl. 2dermatology 0.877(±0.039) 0.874(±0.043) 0.885(±0.046) 0.918(±0.043) 0.902(±0.051) Cl. 2mfeat-pixel 0.924(±0.016) 0.924(±0.015) 0.916(±0.017) 0.926(±0.018) 0.922(±0.021) Cl. 2tic-tac-toe 0.700(±0.038) 0.702(±0.033) 0.689(±0.028) 0.730(±0.048) 0.725(±0.033) Cl. 2zoo 0.950(±0.050) 0.970(±0.049) 0.921(±0.048) 0.951(±0.045) 0.941(±0.087) Cl. 2bridges 0.653(±0.072) 0.634(±0.080) 0.496(±0.099) 0.626(±0.061) 0.635(±0.103) Cl. 3Car 0.864(±0.022) 0.774(±0.020) 0.764(±0.062) 0.718(±0.077) 0.770(±0.059) Cl. 3Cmc 0.497(±0.046) 0.494(±0.046) 0.498(±0.042) 0.473(±0.050) 0.491(±0.045) Cl. 3credit german 0.754(±0.030) 0.754(±0.025) 0.738(±0.021) 0.742(±0.029) 0.742(±0.021) Cl. 3haberman 0.759(±0.046) 0.739(±0.045) 0.748(±0.045) 0.742(±0.037) 0.742(±0.043) Cl. 3kropt 0.360(±0.007) 0.306(±0.006) 0.282(±0.006) 0.349(±0.050) 0.349(±0.052) Cl. 3nursery 0.903(±0.071) 0.766(±0.056) 0.629(±0.071) 0.895(±0.177) 0.896(±0.172) Cl. 3primary-tumor 0.461(±0.076) 0.434(±0.054) 0.422(±0.051) 0.460(±0.046) 0.443(±0.056) Cl. 3soybean 0.937(±0.029) 0.924(±0.030) 0.905(±0.024) 0.934(±0.028) 0.933(±0.037) Cl. 3tae 0.537(±0.152) 0.497(±0.130) 0.451(±0.142) 0.537(±0.130) 0.530(±0.167) Cl. 3trains 0.400(±0.490) 0.400(±0.490) 0.500(±0.490) 0.400(±0.490) 0.400(±0.500) Cl. 3vote 0.903(±0.044) 0.912(±0.043) 0.894(±0.042) 0.873(±0.038) 0.892(±0.043) Cl. 3vowel 0.625(±0.047) 0.594(±0.066) 0.472(±0.057) 0.584(±0.080) 0.577(±0.136) Cl. 3First 8 3 2 5 7

Second 7 5 1 9 6

Conclusion

• In this paper new extension for optimization of attribute selection and weighting and parameter optimization is presented.

• Optimization is done using metaheuristic based wrapper algorithms.

• We plan to extend MetahAtt with more metaheuristics and and to use described methodology on other algorithms (like SVMs and Neural Networks)

Thank you for your attention

Question?