Rafael Alcalá
Dpto. Computer Science and A.I.University of Granada
18071 – SPAIN
Multi-Objective Evolutionary Fuzzy Systems: An Overview by Problem objectives
nature and optimized components
FUZZ-IEEE 2013 Tutorial, Hyderabad, IndiaAfternoon Session: 14:00-17:00, July 7, 2013
1. Basics on MOEFSs- Introduction to Genetic Fuzzy Systems (GFSs) and its main types- Evolutionary Multiobjective Optimization: Basic concepts and framework
2. Types of MOEFSs by multiobjective nature and optimized components
3. MOEFSs designed for the Interpretability-Accuracy Tradeoff of Fuzzy Systems: Two contradictory objectives- Interpretability issues in fuzzy systems design- Some example approaches
4. Other types of MOEFSs- MOEFSs designed for multi-objective control problems- MOEFSs designed for fuzzy association rule mining
5. New Research Directions in MOEFSs
Contents
1. Basics on MOEFSs- Introduction to Genetic Fuzzy Systems (GFSs) and its main types- Evolutionary Multiobjective Optimization: Basic concepts and framework
2. Types of MOEFSs by multiobjective nature and optimized components
3. MOEFSs designed for the Interpretability-Accuracy Tradeoff of Fuzzy Systems: Two contradictory objectives- Interpretability issues in fuzzy systems design- Some example approaches
4. Other types of MOEFSs- MOEFSs designed for multi-objective control problems- MOEFSs designed for fuzzy association rule mining
5. New Research Directions in MOEFSs
Contents
4
An Revision on Genetic Fuzzy Systems (GFSs)
Brief Introduction
Taxonomy of Genetic Fuzzy Systems
Why do we use GAs? Considering multiple Objectives
The birth, GFSs roadmap, current state and most cited
papers
Introduction to genetic fuzzy systems
Multi-Objective Evolutionary Fuzzy Systems (MOEFSs) are a particular type of Genetic Fuzzy System using Multi-Objective Evolutionary Algorithms (MOEAs)
5
The use of genetic/evolutionary algorithms (GAs) to designfuzzy systems constitutes one of the branches of the SoftComputing paradigm: genetic fuzzy systems (GFSs)
The most known approach is that of genetic fuzzy rule-based systems, where some components of a fuzzy rule-based system (FRBS) are derived (adapted or learnt) usinga GA
Some other approaches include genetic fuzzy neuralnetworks and genetic fuzzy clustering, among others
Introduction to genetic fuzzy systemsBrief Introduction
6
Evolutionary algorithms and machine learning:
Evolutionary algorithms were not specifically designed asmachine learning techniques, as other approaches likeneural networks
However, it is well known that a learning task can bemodelled as an optimization problem, and thus solvedthrough evolution
Their powerful search in complex, ill-defined problem spaceshas permitted applying evolutionary algorithms successfullyto a huge variety of machine learning and knowledgediscovery tasks
Their flexibility and capability to incorporate existingknowledge are also very interesting characteristics for theproblem solving.
Introduction to genetic fuzzy systemsBrief Introduction
7
Genetic Fuzzy Rule-Based Systems:
Genetic Algorithm BasedLearning Process
Knowledge BaseData Base + Rule Base
Fuzzy Rule-Based System
Output InterfaceInput Interface
DESIGN PROCESS
Computation with Fuzzy Rule-Based Systems EnvironmentEnvironment
Introduction to genetic fuzzy systemsBrief Introduction
8
Design of fuzzy rule-based systems:
An FRBS (regardless it is a fuzzy model, a fuzzy logiccontroller or a fuzzy classifier), is comprised by two maincomponents: The Knowledge Base (KB), storing the available problem
knowledge in the form of fuzzy rules The Inference System, applying a fuzzy reasoning method on
the inputs and the KB rules to give a system output
Both must be designed to build an FRBS for a specificapplication: The KB is obtained from expert knowledge or by machine
learning methods The Inference System is set up by choosing the fuzzy operator
for each component (conjunction, implication, defuzzifier, etc.)Sometimes, the latter operators are also parametric andcan be tuned using automatic methods
Introduction to genetic fuzzy systemsBrief Introduction
9
An Example of Fuzzy rule-based system
input FuzzificationInterface
DefuzzificationInterface
RuleBase
DataBase
Knowledge Base
InferenceMechanism
output
R1: IF X1 is High AND X2 is Low THEN Y is Medium
R2: IF X1 is Low AND X2 is Low THEN Y is High
…
ML
X1S M L
X2S M L
Y
S
Introduction to genetic fuzzy systemsBrief Introduction
10
The KB design involves two subproblems, related to its two subcomponents:
– Definition of the Data Base (DB):• Variable universes of discourse• Scaling factors or functions• Granularity (number of linguistic terms/labels) per
variable• Membership functions associated to the labels
– Derivation of the Rule Base (RB): fuzzy rule composition
Introduction to genetic fuzzy systemsBrief Introduction
11
As said, there are two different ways to design the KB:
– From human expert information
– By means of machine learning methods guided by the existing numerical information (fuzzy modeling and classification) or by a model of the system being controlled
Introduction to genetic fuzzy systemsBrief Introduction
12
Genetic fuzzy systems
Genetic tuning
Genetic KB learning
Genetic adaptivedefuzzification methods
Genetic tuning of KB parameters
Genetic adaptiveinference system
Genetic adaptiveinference engine
Genetic learningof KB componentsand inference engine parameters
Genetic learning ofFRBS components
Introduction to genetic fuzzy systemsTaxonomy of Genetic Fuzzy Systems
F. Herrera, Genetic Fuzzy Systems: Taxonomy, Current Research Trends and Prospects. Evolutionary Intelligence 1 (2008) 27-46 doi: 10.1007/s12065-007-0001-5
Associated Website: http://sci2s.ugr.es/gfs/
Genetic KBlearning
Genetic ruleselection(A priori ruleextraction)
Genetic DB learning
Simultaneous genetic learning ofKB components
Geneticlearningof linguisticmodels RB and DB
Genetic fuzzy ruleslearning (ApproximateModels, TS-rules ..)
Genetic RB learningfor prediction
Geneticdescriptiverules extraction
Genetic rulelearning(A priori DB)
EmbeddedgeneticDB learning
A prioirigeneticDB learning
13
Introduction to genetic fuzzy systemsTaxonomy of Genetic Fuzzy Systems
14
Classically:– performed on a predefined DB definition– tuning of the membership function shapes by a
GA
– tuning of the inference parameters
VS S M VLL
Introduction to genetic fuzzy systems1. Genetic Tuning
15
Introduction to genetic fuzzy systems1. Genetic Tuning
16
A predefined Data Base definition is assumed– The fuzzy rules (usually Mamdani-type) are
derived by a GA
Introduction to genetic fuzzy systems2. Genetic Rule Learning
17
Introduction to genetic fuzzy systems2. Genetic Rule Learning
18
– A predefined set of candidate rules is assumed
– The fuzzy rules are selected by a GA for getting a compact rule base (more interpretable, more precise)
Introduction to genetic fuzzy systems3. Genetic Rule Selection
19
Introduction to genetic fuzzy systems3. Genetic Rule Selection
20
Example of genetic rule selection
Learning
1e = ( ),x y1 1
Ne = ( ),x yN N
...
Data Set
Ruleselection
0 2
YXS M L1 1 1 S M L2 2 2
0 2
Initial Data Base
X is THEN Y esIFR1= 2L SX is THEN Y esIFR2= 2S MX is THEN Y esIFR3= 2M L
1
1
1
DerivedRuleBase
Selected Rule BaseX is THEN Y esIFR1= 2L SX is THEN Y esIFR2= 2S M
1
1
Introduction to genetic fuzzy systems3. Genetic Rule Selection
21
– Learning of the membership function shapes by a GA
Introduction to genetic fuzzy systems4. Genetic DB Learning
22
Introduction to genetic fuzzy systems4. Genetic DB Learning
23
The simultaneous derivation properly addresses the strong dependency existing between the RB and the DB
VS S M VLL
Introduction to genetic fuzzy systems5. Simultaneous Genetic Learning of KB Components
24
Introduction to genetic fuzzy systems5. Simultaneous Genetic Learning of KB Components
25
R 1 R 2 R N R 1 R 2 R N R 1 R 2 R N
{ES3,..,EL3} {ES3,..,EL3} {ES3,..,EL3} 1 2 1 … ……… … …
CSCOR CSCCSD
Rule Base Connectives
ConjunctionDefuzWEIGTH
W
Example of the coding scheme for learning an RB and the inference connective parameters
Introduction to genetic fuzzy systems6. Genetic Learning of KB Components and Inference Engine
Parameters
26
EvolutionaryAlgorithm
ScalingFunctions
FuzzyRules
MembershipFunctions
Knowledge Base
ScaledInput Fuzzification
InferenceEngine Defuzzification Scaled
Output
Fuzzy Processing
Evo
lutio
nary
Des
ign
Introduction to genetic fuzzy systems
27
Particular Characteristics of the Genetic Fuzzy Systems We can code different FS components in a chromosome:
Identify relevant inputs
Scaling factors Membership functions, shape functions, optimal shape of
membership funct., granularity (number of labels per variable)
Fuzzy rules, Any inference parameter, ....
We can define different mechanism for managing them(combining genetic operators, coevolution,...)
Introduction to genetic fuzzy systemsWhy do we use GAs?
28
Particular Characteristics of the Genetic Fuzzy Systems
We can consider multiple objectives in the learningmodel (interpretability, precision, ....)
Interpretability
Acc
urac
yParetoSolutions
Introduction to genetic fuzzy systemsConsidering Multiple Objectives
Thrift’s ICGA91 paper (Mamdani-type Rule Base Learning. Pittsburgh approach)Thrift P (1991) Fuzzy logic synthesis with genetic algorithms. In: Proc. of 4thInternational Conference on Genetic Algorithms (ICGA'91), pp 509-513
Valenzuela-Rendón’s PPSN-I paper (Scatter Mamdani-type KB Learning. Michiganapproach)
Valenzuela-Rendon M (1991) The fuzzy classifier system: A classifier system forcontinuously varying variables. In: Proc. of 4th International Conference on GeneticAlgorithms (ICGA'91), pp 346-353
Pham and Karaboga’s Journal of Systems Engineering paper (Relational matrix-based FRBS learning. Pittsburgh approach)
Pham DT, Karaboga D (1991) Optimum design of fuzzy logic controllers usinggenetic algorithms. Journal of Systems Engineering 1:114-118).
Karr’s AI Expert paper (Mamdani-type Data Base Tuning)
Karr C (1991) Genetic algorithms for fuzzy controllers. AI Expert 6(2):26-33.
Almost the whole basis of the area were established in the first year!
The birth of GFSs: 1991
Introduction to genetic fuzzy systemsThe birth, GFSs roadmap, current status and most cited papers
Thrift’s GFS:P. Thrift, Fuzzy logic synthesis with genetic algorithms, Proc. Fourth Intl. Conf. on Genetic Algorithms (ICGA’91), San Diego, USA, 1991, pp. 509–513
– Classical approach: Pittsburgh – the decision table is encoded in a rule consequent array
– The output variable linguistic terms are numbered from 1 to n and comprise the array values. The value 0 represents the rule absence, thus making the GA able to learn the optimal number of rules
– The ordered structure allows the GA to use simple genetic operatorsS M L
S
M
L
X 1X2
R5
R1 R2 R3
R4 R6
R7 R8 R9 1 0 2 0 2 0 2 0 3
Y {B, M, A}1 2 3
MB
AM
M
__
____
__
Introduction to genetic fuzzy systemsThe birth, GFSs roadmap, current status and most cited papers
1991-1996/7: INITIAL GFS SETTING: KB LEARNING:
Establishment of the three classical learning approaches in the GFS field: Michigan,Pittsburgh, and IRL
Different FRBS types: Mamdani, Mamdani DNF, Scatter Mamdani, TSK
Generic applications: Classification, Modeling, and Control
1995-…: FUZZY SYSTEM TUNING:
First: Membership function parameter tuning
Later: other DB components adaptation: scaling factors, context adaptation (scalingfunctions), linguistic hedges, …
Recently: interpretability consideration
GFSs roadmap
Introduction to genetic fuzzy systemsThe birth, GFSs roadmap, current status and most cited papers
1998-…: APPROACHING TO MATURITY?NEW GFS LEARNING APPROACHES:
New EAs: Bacterial genetics, DNA coding, Virus-EA, genetic local search (memeticalgorithms), …
Hybrid learning approaches: a priori DB learning, GFNNs, Michigan-Pitt hybrids, …
Multiobjective evolutionary algorithms
Interpretability-accuracy trade-off consideration
Course of dimensionality (handling large data sets and complex problems): Rule selection (1995-…) Feature selection at global level and fuzzy rule level Hierarchical fuzzy modeling
“Incremental” learning
GFSs roadmap
Introduction to genetic fuzzy systemsThe birth, GFSs roadmap, current status and most cited papers
33
Number of papers on GFSs published in JCR journals
Source: The Thomson Corporation ISI Web of KnowledgeQuery: (TS = (("GA-" OR "GA based" OR evolutionary OR "genetic algorithm*" OR "genetic
programming" OR "evolution strate*" OR "genetic learning" OR "particle swarm" OR "differential evolutio*" OR "ant system*" OR "ant colony" OR "genetic optimi*" OR "estimation of distribution algorithm*") AND ("fuzzy rule*" OR "fuzzy system*" OR "fuzzy neural" OR "neuro-fuzzy" OR "fuzzy control*" OR "fuzzy logic cont*" OR "fuzzy class*" OR "fuzzy if" OR "fuzzy model*" OR "fuzzy association rule*" OR "fuzzy regression"))
Date of analysis: January 3th, 2013 Number of papers: 5079 Number of citations: 30738 Average citations per paper: 6.05
Introduction to genetic fuzzy systemsCurrent state of the GFS area
34
Highly cited papers on GFSs (classic approaches - papers until 2000)
1. Homaifar, A., McCormick, E., Simultaneous Design of Membership Functions and rule sets for fuzzy controllers using genetic algorithms, IEEE TFS 3 (2) (1995) 129-139. Citations: 302
2. Ishibuchi, H., Nozaki, K., Yamamoto, N., Tanaka, H., Selecting fuzzy if-then rules for classification problems using genetic algorithms, IEEE TFS 3 (3) (1995) 260-270. Citations: 284
3. Setnes, M., Roubos, H., GA-fuzzy modeling and classification: complexity and performance, IEEE TFS 8 (5) (2000) 509-522 . Citations: 215
4. Ishibuchi, H., Nakashima, T., Murata, T., Performance evaluation of fuzzy classifier systems for multidimensional pattern classification problems, IEEE TSMC B 29 (5) (1999) 601-618. Citations: 177
5. Shi, Y.H., Eberhart, R., Chen, Y.B., Implementation of evolutionary fuzzy systems, IEEE TFS 7 (2) (1999) 109-119. Citations: 126
6. Park, D., Kandel, A., Langholz, G., Genetic-based new fuzzy reasoning models with application to fuzzy control, IEEE TSMC B 24 (1) (1994) 39-47. Citations: 125
7. Jin, YC (2000) Fuzzy modeling of high-dimensional systems: Complexity reduction and interpretability improvement. IEEE Transactions on Fuzzy Systems 8(2):212-221. Citations: 121
8. Ishibuchi H, Murata T, Turksen IB (1997) Single-objective and two-objective genetic algorithms forselecting linguistic rules for pattern classification problems. Fuzzy Sets and Systems 89(2):135-150 Citations: 116
9. Juang, C.F., Lin, J.Y., Lin, C.T., Genetic reinforcement learning through symbiotic evolution for fuzzy controller design, IEEE TSMC B 30 (2) (2000) 290-302. Citations: 109
10. Herrera, F., Lozano, M., Verdegay, J.L., Tuning fuzzy-logic controllers by genetic algorithms, IJAR 12 (3-4) (1995) 299-315. Citations: 108
Introduction to genetic fuzzy systemsCurrent state of the GFS area
35
Highly cited papers on GFSs (recent approaches – 2001 to 2010)
1. Juang, CF (2002) A TSK-type recurrent fuzzy network for dynamic systems processing by neural network and genetic algorithms. IEEE Transactions on Fuzzy Systems 10(2):155-170. Citations: 144
2. Cordon O, Gomide F, Herrera F, Hoffmann F, Magdalena L (2004) Ten years of genetic fuzzy systems: current framework and new trends. Fuzzy Sets and Systems 141(1):5-31. Citations: 142
3. Roubos H, Setnes M (2001) Compact and transparent fuzzy models and classifiers through iterative complexity reduction. IEEE Transactions on Fuzzy Systems 9(4):516-524. Citations: 105
4. Ishibuchi H, Nakashima T, Murata T (2001) Three-objective genetics-based machine learning for linguistic rule extraction. Information Sciences 136(1-4):109-133. Citations: 97
5. Ishibuchi H, Yamamoto T (2004) Fuzzy rule selection by multi-objective genetic local search algorithms and rule evaluation measures in data mining. Fuzzy Sets and Systems 141(1):59-88. Citations: 96
6. Cordon O, Herrera F, Villar P (2001) Generating the knowledge base of a fuzzy rule-based system by the genetic learning of the data base. IEEE Transactions on Fuzzy Systems 9(4):667-674. Citations: 66
7. Gonzalez J, Rojas I, Ortega J, Pomares H, Fernandez J, Diaz AF (2003) Multiobjective evolutionary optimization of the size, shape, and position parameters of radial basis function networks for function approximation. IEEE Transactions on Neural Networks 14(6):1478-1495. Citations: 61
8. Liu BD, Chen CY, Tsao JY (2001) Design of adaptive fuzzy logic controller based on linguistic-hedge conceptsand genetic algorithms. IEEE Transactions on Systems Man and Cybernetics Part B-Cybernetics 31(1):32-53 Citations: 53
9. Wang HL, Kwong S, Jin YC, et al. (2005) Multi-objective hierarchical genetic algorithm for interpretable fuzzy rule-based knowledge extraction. Fuzzy Sets And Systems 149(1):149-186. Citations: 52
10. Kuo RJ, Chen CH, Hwang YC (2001) An intelligent stock trading decision support system through integration of genetic algorithm based fuzzy neural network and artificial neural network. Fuzzy Sets and Systems 118(1):21-45. Citations: 50
Introduction to genetic fuzzy systemsCurrent state of the GFS area
36
– M. Fazzolari, R. Alcalá, Y. Nojima, H. Ishibuchi, F. Herrera. A review of the application of Multi-Objective Evolutionary Systems: Current status and further directions. IEEE Transactions on Fuzzy Systems 21(1) (2013) 45-65, doi: 10.1109/TFUZZ.2012.2201338
– F. Herrera, Genetic Fuzzy Systems: Taxonomy, Current Research Trends and Prospects. Evolutionary Intelligence 1 (2008) 27-46 doi: 10.1007/s12065-007-0001-5,
– F. Herrera, Genetic Fuzzy Systems: Status, Critical Considerations and Future Directions, International Journal of Computational Intelligence Research 1 (1) (2005) 59-67
– O. Cordón, F. Gomide, F. Herrera, F. Hoffmann, L. Magdalena, Ten Years of Genetic Fuzzy Systems: Current Framework and New Trends, FSS 141 (1) (2004) 5-31
– F. Hoffmann, Evolutionary Algorithms for Fuzzy Control System Design, Proceedings of the IEEE 89 (9) (2001) 1318-1333
GENETIC FUZZY SYSTEMSEvolutionary Tuning and Learning of Fuzzy
Knowledge Bases.O. Cordón, F. Herrera, F. Hoffmann, L. Magdalena
World Scientific, July 2001
H. Ishibuchi, T. Nakashima, M. Nii, Classification and Modeling with Linguistic Information Granules. Advanced Approaches to Linguistic Data Mining. Springer (2005)
Introduction to genetic fuzzy systemsSome References
37
http://sci2s.ugr.es/gfs/biblio.php
Introduction to genetic fuzzy systemsGFSs and MOEFSs Website
http://sci2s.ugr.es/gfs/
1. Basics on MOEFSs- Introduction to Genetic Fuzzy Systems (GFSs) and its main types- Evolutionary Multiobjective Optimization: Basic concepts and framework
2. Types of MOEFSs by multiobjective nature and optimized components
3. MOEFSs designed for the Interpretability-Accuracy Tradeoff of Fuzzy Systems: Two contradictory objectives- Interpretability issues in fuzzy systems design- Some example approaches
4. Other types of MOEFSs- MOEFSs designed for multi-objective control problems- MOEFSs designed for fuzzy association rule mining
5. New Research Directions in MOEFSs
Contents
Multiobjective OptimizationTwo-Objective Maximization Problem:
))(),(()( 21 xxxf ffMaximizeM
axim
ize
Maximize)(1 xf
)(2 xf
))(),(()( 21 xxxf ffMaximizeM
axim
ize
Maximize)(1 xf
)(2 xf
B
AA dominates B
B is dominated by A
(A is better than B )
Comparison between Two Solutions
))(),(()( 21 xxxf ffMaximizeM
axim
ize
Maximize)(1 xf
)(2 xf
B
AA and C are non-dominatedwith each other.C
Comparison between Two Solutions
Pareto-Optimal SolutionsM
axim
ize
Maximize)(1 xf
A Pareto-optimal solution is a solution thatis not dominated by any other solutions.
)(2 xf
Pareto-Optimal Solutions
Pareto FrontM
axim
ize
Maximize)(1 xf
The set of all Pareto-optimal solutions iscalled the Pareto front of the problem.
)(2 xf
Pareto Front
Pareto-Optimal Solutions(Pareto front)
Max
imiz
e
Maximize)(1 xf
Evolutionary multiobjective optimization (EMO)algorithms have been designed to search forPareto-optimal solutions in their single run.
)(2 xf
EMO Algorithms
Maximize g(x) = w1 f1(x) + w2 f2(x)
Maximize
Max
imiz
e
)(1 xf
)(2 xf
Comparison: Weighted Sum Approach
Only a single solution is obtainedby the weighted sum approach.
Feasible Region
w = (w1, w2)
Maximize f1(x), f2(x)
Maximize
Max
imiz
e
)(1 xf
)(2 xf
Comparison: EMO Approach
Only a single solution is obtainedby the weighted sum approach.
Multiple solutions are obtainedby an EMO algorithm.
Feasible Region
Difficulties in Weighted Sum Approach
- This approach is sensitive to the weight vector specification.- This approach can not find any Pareto-optimal solutions in a
non-convex region of the Pareto front in the objective space.
Maximize
Max
imiz
e
)(1 xf
)(2 xf
Feasible Region
w = (w1, w2)
Difficulties in Weighted Sum Approach
- This approach is sensitive to the weight vector specification.- This approach can not find any Pareto-optimal solutions in a
non-convex region of the Pareto front in the objective space.
Maximize
Max
imiz
e
)(1 xf
)(2 xf
Feasible Region
w = (w1, w2)
EMO Approach- EMO approach can find Pareto-optimal solutions even in a non-
convex region of the Pareto front in the objective space.
Maximize
Max
imiz
e
)(1 xf
)(2 xf
Feasible Region
f1: Total profit from knapsack 1
f 2: T
otal
pro
fit fr
om k
naps
ack
2 2000th generation50th generation20th generation
Pareto front
16000 17000 18000 19000 2000016000
17000
18000
19000
20000
21000
Comparison of the Two ApproachesTwo-objective maximization problem
EMO Approach Weighted Sum ApproachExperimental results of a single run of each approach
f1: Total profit from knapsack 1f 2:
Tot
al p
rofit
from
kna
psac
k 2 2000th generation
50th generation20th generation
Pareto front
16000 17000 18000 19000 2000016000
17000
18000
19000
20000
21000
f1: Total profit from knapsack 1
f 2: T
otal
pro
fit fr
om k
naps
ack
2 2000th generation50th generation20th generation
Pareto front
16000 17000 18000 19000 2000016000
17000
18000
19000
20000
21000
Search Direction in Each ApproachTwo-objective maximization problem
EMO Approach Weighted Sum Approachf1: Total profit from knapsack 1
f 2: T
otal
pro
fit fr
om k
naps
ack
2 2000th generation50th generation20th generation
Pareto front
16000 17000 18000 19000 2000016000
17000
18000
19000
20000
21000
Both the diversity and the convergence should be improved in EMO.
Highly Cited EMO PapersTwo Dominant Algorithms: NSGA-II and SPEA1. Deb K et al. (2002) A fast and elitist multiobjective genetic
algorithm: NSGA-II. IEEE TEC. NSGA-II2. Zitzler E, Thiele L (1999) Multiobjective evolutionary
algorithms: A comparative case study and the StrengthPareto approach. IEEE TEC. SPEA (=> SPEA2 in TIK-Report)
3. Fonseca CM, Fleming PJ (1998) Multiobjective optimization andmultiple constraint handling with evolutionary algorithms (Part I):A unified formulation, IEEE SMC Part A.
4. Zitzler E, Thiele L, Laumanns M (2003) Performance assessment ofmultiobjective optimizers: An analysis and review. IEEE TEC.
5. Ishibuchi H, Murata T (1998) A multi-objective genetic local searchalgorithm and its application to flowshop scheduling, IEEE SMCPart C.
Goal of EMO AlgorithmsAn EMO algorithm is designed to search for- all Pareto-optimal solutions- uniformly distributed Pareto optimal solutions- a solution set which approximates the Pareto front
in their single run.
18000 19000 2000017000
18000
19000
20000
Pareto frontObtained solution
Basic Ideas in EMO Algorithm Design
f1: Total profit from knapsack 1
f 2: T
otal
pro
fit fr
om k
naps
ack
2 2000th generation50th generation20th generation
Pareto front
16000 17000 18000 19000 2000016000
17000
18000
19000
20000
21000
Desired search behavior of EMO algorithms
Recently developed well-known EMO algorithms such as NSGA-II and SPEA2 have some common features.
(1) Pareto DominanceConverge to the Pareto front
Max
imiz
e
Maximize
high fitness
low fitness
high fitness
highfitness
low fitness
Max
imiz
e
Maximize
Max
imiz
e
Maximize
Basic Ideas in EMO Algorithm Design
low fitness
high fitness
Recently developed well-known EMO algorithms such as NSGA-II and SPEA2 have some common features:
Basic Ideas in Recent EMO Algorithms
1. Pareto Dominance2. Crowding3. Elite Strategy
f1: Total profit from knapsack 1
f 2: T
otal
pro
fit fr
om k
naps
ack
2 2000th generation50th generation20th generation
Pareto front
16000 17000 18000 19000 2000016000
17000
18000
19000
20000
21000
(2) CrowdingDiversity maintenance
Max
imiz
eMaximize
high fitness
low fitness
high fitness
highfitness
low fitness
Max
imiz
eMaximize
Max
imiz
eMaximize
Basic Ideas in EMO Algorithm Design
high fitness
high fitness
low fitness
(1) Pareto DominanceConverge to the Pareto front
Recently developed well-known EMO algorithms such as NSGA-II and SPEA2 have some common features:
Basic Ideas in EMO Algorithm Design
Example: Crowding Distance in NSGA-IIDistance between adjacent individuals
0 Maximize f1
Max
imiz
e f 2 C
B
Aa
b a+b
Infinitelylarge value
Crowding distance of C is (a + b)
(3) Elitist StrategyNon-dominated solutions are handled as elite solutions.
Max
imiz
eMaximize
high fitness
low fitness
high fitness
highfitness
low fitness
Max
imiz
eMaximize
Max
imiz
eMaximize
Non-dominatedsolutions
(Elite solutions)
Basic Ideas in EMO Algorithm Design
(2) CrowdingDiversity maintenance
(1) Pareto DominanceConverge to the Pareto front
Recently developed well-known EMO algorithms such as NSGA-II and SPEA2 have some common features:
Basic Ideas in Recent EMO Algorithms
(1) Pareto Dominance (Convergence to the Pareto front)(2) Crowding (Diversity Maintenance)(3) Elite Strategy (Non-Dominated Solutions)
f1: Total profit from knapsack 1
f 2: T
otal
pro
fit fr
om k
naps
ack
2 2000th generation50th generation20th generation
Pareto front
16000 17000 18000 19000 2000016000
17000
18000
19000
20000
21000
(1)(2)
(2)
Hot Issues in EMO Research
Utilization of Decision Maker’s Preference- Preference is incorporated into EMO algorithms.- Interactive EMO approaches seem to be promising.
Handling of Many Objectives by EMO Algorithms- Pareto dominance-based algorithms do not work well.- More selection pressure is needed.
Hybridization with Local Search- Hybridization often improves the performance of EMO.- Balance between local and genetic search is important.
Design of New EMO Algorithms (some alternatives to NSGA-II and SPEA2)
- Indicator-based EMO algorithms - Scalarizing function-based EMO algorithms- Use of other search methods such as PSO, ACO and DE.
New Trend in EMO Algorithm Design IBEA: Indicator-Based Evolutionary Algorithm
Maximize f1
Max
imiz
e f 2
x1
x3
x2
Basic IdeaTo maximize a performance indicator of a solution set (not a solution): Hypervolume is often used.
Maximization of this area
}{ where subject to)( Maximize
SNSSI
New Trend in EMO Algorithm Design IBEA: Indicator-Based Evolutionary Algorithm
(Maximization of an Indicator Function)
S : A set of solutionsN: A pre-specified number
of required solutionsX: A feasible region
x x X
Maximize f1
Max
imiz
e f 2
Maximization of this area
New Trend in EMO Algorithm Design MOEA/D: Use of Scalarizing Functions
Its Basic Idea (Decomposition): A multi-objective problemis handled as a set of scalarizing function optimizationproblems with different weight vectors.
(a) Two-objective case (b) Three-objective case
Weight vector
MOEA/D: Multi-objective evolutionary algorithm based on decomposition by Zhang and Li (IEEE TEC 2007)
New Trend in EMO Algorithm Design Hybrid Method: Use of Scalarizing Functions
Probability for scalarizing fitness functions:Parent selection: PPS Generation update: PGU
Initialization
Parent selection
Genetic operation
Generation update
PPS
PGU
1PGU
1PPS
End
Scalarizing fitness function
NSGA-II fitness evaluation mechanism
Ishibuchi et al. (PPSN 2006)
New Trend in EMO Algorithm Design Use of Other Meta-Heuristics (PSO, ACO, etc.) Highly Cited Papers[1] Coello CAC, Pulido GT, Lechuga MS (2004) Handling Multiple
Objectives with Particle Swarm Optimization, IEEE TEC[2] McMullen PR (2001) An Ant Colony Optimization Approach to
Addressing a JIT Sequencing Problem with MultipleObjectives, Artificial Intelligence in Engineering
[3] Ray T, Liew KM (2002) A Swarm Metaphor for MultiobjectiveDesign Optimization, Engineering Optimization
[4] Li XD (2003) A Non-Dominated Sorting Particle SwarmOptimizer for Multiobjective Optimization, GECCO 2003.
[5] Ho SL et al. (2005) A Particle Swarm Optimization-BasedMethod for Multiobjective Design Optimizations, IEEE Trans.on Magnetics
For More InformationWebpage for EMO Papers: EMOO
http://www.lania.mx/~ccoello/EMOO/
For More InformationWebpage for EMO Algorithms and Problems: PISA
http://www.tik.ee.ethz.ch/sop/pisa/
1. Basics on MOEFSs- Introduction to Genetic Fuzzy Systems (GFSs) and its main types- Evolutionary Multiobjective Optimization: Basic concepts and framework
2. Types of MOEFSs by multiobjective nature and optimized components
3. MOEFSs designed for the Interpretability-Accuracy Tradeoff of Fuzzy Systems: Two contradictory objectives- Interpretability issues in fuzzy systems design- Some example approaches
4. Other types of MOEFSs- MOEFSs designed for multi-objective control problems- MOEFSs designed for fuzzy association rule mining
5. New Research Directions in MOEFSs
Contents
Types of MOEFSs by Multiobjective Nature and Optimized Components
The flexibility of FRBSs makes them applicable to a wide range of problems.
From among them, problems with multiple conflicting objectives are of particular interest to researchers, as they are very common and arise wherever optimal decisions need to be taken.
These problems can be tackled using MOEAs for the design of FRBSs, giving way to the so called MOEFSs.
MOEFSs are a type of GFS exploiting MOEAs to design sets of FRBSs with different trade-offs among objectives instead of a single one.
Erro
rSm
all
Larg
e
ComplexitySimple Complicated
Interpretablefuzzy system
Accuratefuzzy system
10x
1R
2R
ConsequentClass 1(0.26)
Class 2(1.00)
11x
DC
DC
Motivations for MOEFSs at their Origin1) Preventing a Deterioration of Interpretability
Motivations for MOEFSs at their Origin Multiobjective Fuzzy System Design (Late 1990s - )
Erro
rSm
all
Larg
e
ComplexitySimple Complicated
Interpretablefuzzy system
Accuratefuzzy systemIdeal
fuzzy system
Goals - Accuracy Maximization- Interpretability Maximization
Erro
rSm
all
Larg
e
ComplexitySimple Complicated
Interpretablefuzzy system
Accuratefuzzy system
Use of EMO algorithms to search for a number of non-dominated fuzzy systems with different Accuracy-Interpretability (A-I) Trade-Offs
Motivations for MOEFSs at their Origin Multiobjective Fuzzy System Design (Late 1990s - )
Motivations for MOEFSs at their Origin2) Avoiding too Complex Models helps to Control Overfitting
Test dataaccuracy
Training dataaccuracy
Accuracy maximization Overfitting
Complexity
Err
or
S*0
Complexity
Test data
S*0
Training data
Err
orMany non-dominated fuzzy systems can be obtained alongthe tradeoff surface by a single run of an EMO algorithm.
Motivations for MOEFSs at their Origin2) Avoiding too Complex Models helps to Control Overfitting
Michela Fazzolari, Rafael Alcalá, Yusuke Nojima, Hisao Ishibuchi, Francisco Herrera. A review on the application of Multi-Objective Genetic Fuzzy Systems: current status and further directions. IEEE Transactions on Fuzzy Systems 21(1) (2013) 45-65, doi: 10.1109/TFUZZ.2012.2201338
Types of MOEFSs by Multiobjective Nature and Optimized Components
However, MOEFSs have been also applied to solve multiobjectivecontrol problems and for fuzzy association rule mining (where different metrics are considered to describe the quality of the obtained rules).
The type of objectives used in these three main categories (A-I trade-off, control and mining fuzzy association rules) represent a different multi-objective nature
Due to this fact, both, the multi-objective nature of the problem faced and type of FRBS components optimized, have been considered recently to propose a two-level taxonomy in,
Michela Fazzolari, Rafael Alcalá, Yusuke Nojima, Hisao Ishibuchi, Francisco Herrera. A review on the application of Multi-Objective Genetic Fuzzy Systems: current status and further directions. IEEE Transactions on Fuzzy Systems 21(1) (2013) 45-65, doi: 10.1109/TFUZZ.2012.2201338
Types of MOEFSs by Multiobjective Nature and Optimized Components: A Two-level Taxonomy
12 papers 24 papers 12 papers 5 papers
6 papers17 papers36 papers
We willmainly focusson this type.
Jin, Yaochu (Ed.) Multi-Objective Machine LearningSpringer-Verlag, 2006
Associated Webpage (http://ssci2s.ugr.es/gfs)
Multiobjective Evolutionary Fuzzy SystemsBibliography
H. Ishibuchi, T. Nakashima, M. Hii. Classification and Modelling with Linguistic Information Granules. Advanced Approaches to Linguistic Data Mining. Springer-Verlag, 2004.
– M. Fazzolari, R. Alcalá, Y. Nojima, H. Ishibuchi, F. Herrera. A review of the application of Multi-Objective Evolutionary Systems: Current status and further directions. IEEE Transactions on Fuzzy Systems 21(1) (2013) 45-65, doi: 10.1109/TFUZZ.2012.2201338
Highly Cited MOEFS Papers[1] Ishibuchi et al. (1997) Single-objective and two-objective
genetic algorithms for selecting linguistic rules for patternclassification problems. Fuzzy Sets & Systems.
[2] Ishibuchi et al. (2001) Three-objective genetics-based machinelearning for linguistic rule extraction. Information Sciences.
[3] Ishibuchi & Yamamoto (2004) Fuzzy rule selection by multi-objective genetic local search algorithms and rule evaluationmeasures in data mining. Fuzzy Sets & Systems.
[4] Wang et al. (2005) Multi-objective hierarchical genetic algorithmfor interpretable fuzzy rule-based knowledge extraction. FuzzySets & Systems.
[5] Johansen & Babuska (2003) Multiobjective identification ofTakagi-Sugeno fuzzy models. IEEE TFS.
1. Basics on MOEFSs- Introduction to Genetic Fuzzy Systems (GFSs) and its main types- Evolutionary Multiobjective Optimization: Basic concepts and framework
2. Types of MOEFSs by multiobjective nature and optimized components
3. MOEFSs designed for the Interpretability-Accuracy Tradeoff of Fuzzy Systems: Two contradictory objectives- Interpretability issues in fuzzy systems design- Some example approaches
4. Other types of MOEFSs- MOEFSs designed for multi-objective control problems- MOEFSs designed for fuzzy association rule mining
5. New Research Directions in MOEFSs
Contents
81
Highly used criteria: Complexity criteria in the learning of FRBSs.
Number of variables, labels, rules, conditions …
Interpretability Issues in Fuzzy System DesignComplexity Criteria
82
Interpretability quality: associated to the meaning of the labels and the size of the rule base
Interpretability Issues in Fuzzy System DesignSemantic Criteria
83
Interpretability quality: associated to the meaning of the labels and the size of the rule base
Interpretability Issues in Fuzzy System DesignSyntactic Criteria
84
Interpretability quality: associated to the meaning of the labels and the size of the rule base
Interpretability Issues in Fuzzy System DesignStrategies to Satisfy Interpretability
85
Interpretability quality:
What is the most interpretable rule base?
Interpretability Issues in Fuzzy System DesignStill not Clear Concepts
Rule Base Level Fuzzy Partition Level
Complexity-based
Interpretability
C1Number of rulesNumber of conditions
C2Number of membership functionsNumber of features
Semantic-basedInterpretability
C3Consistency of rulesRules fired at the same timeTransparency of rule structure (rule weights, etc.)Cointension
C4Absolute Measures:
Completeness or coverage,normalization, distinguishability,complementarity
Relative Measures
A Taxonomy on the Existent Interpretability Measures for Linguistic FRBSs
Most works in C1 and C2 are applied to classificationproblems. They are the classic measures.
There are few works in C3 Still an open problem
86Most works in C4 impose absolute measures orrestrictions. Relativity could be a new possibility.
Still an open problem.
Interpretability of FRBSs is still an open problem since there is no single (or global) comprehensive measure to quantify the interpretability of linguistic models
To get a good global measure it would be necessary to consider appropriate measures from all of the quadrants, in order to take into account the different interpretability properties required for these kinds of systems together.
87
M.J. Gacto, R. Alcalá, F. Herrera
Interpretability of Linguistic Fuzzy Rule-Based Systems: An
Overview on Interpretability Measures, Information Sciences
181:20 (2011) 4340–4360, doi: 10.1016/j.ins.2011.02.021
A thematic website has been developed to maintain this study at:
http://sci2s.ugr.es/fuzzy-interpretability/
A Taxonomy on the Existent Interpretability Measures for Linguistic FRBSs (2)
The different measures from each quadrant could be optimized as different objectives within a multi-objective framework.
They are contradictory to some degree. Not only accuracy is contradictory to inter-pretability. The different measures represent different properties and requirements.
Together with accuracy, many interpretability objectives should be optimized at the same. Two different solutions:
Development of new EMO algorithms for many objective problems (incoming for future)
By grouping complexity measures and semantic measures into two respective indexes. (it would represent the present)
With respect to the objectives nature, while accuracy is hard to improve, interpre-tability is easy to obtain, since interpretable models can even be provided by hand.
These differences between both types of objectives influence the optimization process, by which the applied MOEAs are usually modified or extended.
Applicability of MOEFSs to the I-A problem
1. Basics on MOEFSs- Introduction to Genetic Fuzzy Systems (GFSs) and its main types- Evolutionary Multiobjective Optimization: Basic concepts and framework
2. Types of MOEFSs by multiobjective nature and optimized components
3. MOEFSs designed for the Interpretability-Accuracy Tradeoff of Fuzzy Systems: Two contradictory objectives- Interpretability issues in fuzzy systems design- Some example approaches
4. Other types of MOEFSs- MOEFSs designed for multi-objective control problems- MOEFSs designed for fuzzy association rule mining
5. New Research Directions in MOEFSs
Contents
A-I Trade-Off: Some Example ApproachesBibliography on this category
Michela Fazzolari, Rafael Alcalá, Yusuke Nojima, Hisao Ishibuchi, Francisco Herrera. A review on the application of Multi-Objective Genetic Fuzzy Systems: current status and further directions. IEEE Transactions on Fuzzy Systems 21(1) (2013) 45-65.
• Most of them are basedon 2nd gen. MOEAs
• Usually no more than 3 objectives
• Complexity at thebeginning; Semanticaspects in the last years
• Most of them are Linguistic and Mamdanitype based approaches
• KB learning in the lastyears (granularity as im-portant factor)
• Most of them are improved versions of themost known MOEAs(particularly in the case of KB learning)
In the following we will see a representative example for each type:
o FIRST TYPE: RB Learning
o SECOND TYPE: DB Tuning + Rule Sel.
o THIRD TYPE: KB Learning
Michela Fazzolari, Rafael Alcalá, Yusuke Nojima, Hisao Ishibuchi, Francisco Herrera. A review on the application of Multi-Objective Genetic Fuzzy Systems: current status and further directions. IEEE Transactions on Fuzzy Systems 21(1) (2013) 45-65.
A-I Trade-Off: Some Example ApproachesSome Example Cases
FIRST TYPE: RULE BASE LEARNING - CLASSIFICATION
H. Ishibuchi, T. Yamamoto, Fuzzy rule selection by multi-objective geneticlocal search algorithms and rule evaluation measures in data mining, FuzzySets and Systems, Vol. 141, pp. 59-88 (2004)
A-I Trade-Off: Some Example ApproachesMODEL 1: Multiobjective Rule Selection
1. Heuristic Rule ExtractionA pre-specified number of candidate fuzzy rules are extracted fromnumerical data using a heuristic rule evaluation criterion (data mining).
2. Multiobjective Genetic Fuzzy Rule SelectionA small number of fuzzy rules are selected from the extractedcandidate rules using a multi-objective genetic algorithm (evolutionaryoptimization).
Two-Stage Approach for Rule Base Learning
H. Ishibuchi and T. Yamamoto, “Fuzzy rule selection by multi-objective genetic local search algorithms and rule evaluationmeasures in data mining,” Fuzzy Sets and Systems, Vol. 141,pp. 59-88 (2004).
A-I Trade-Off: Some Example ApproachesMODEL 1: Multiobjective Rule Selection
Fuzzy Rules for n-dimensional ProblemsIf x1 is A1 and … and xn is An then Class C with CF
Ai : Antecedent fuzzy setClass C : Consequent classCF : Rule weight (Certainty factor)
A-I Trade-Off: Some Example ApproachesMODEL 1: Multiobjective Rule Selection
Antecedent Fuzzy Sets (Multiple Partitions)
Usually we do not know an appropriate fuzzy partition for each input variable.
1.0
0.00.0 1.0
S2 L2
1.0
0.00.0 1.0
S2 L2
1.0
0.00.0 1.0
S3 M3 L3
1.0
0.00.0 1.0
S3 M3 L3
1.0
0.00.0 1.0
S4 MS4 ML4 L4
1.0
0.00.0 1.0
S4 MS4 ML4 L4
1.0
0.00.0 1.0
S5 MS5 M5 ML5 L5
1.0
0.00.0 1.0
S5 MS5 M5 ML5 L5
1.0
0.01.00.0
DC
Possible Fuzzy RulesTotal number of possible fuzzy rules
n15114114
Don’t care Don’t care
…x1 xn
A-I Trade-Off: Some Example ApproachesMODEL 1: Multiobjective Rule Selection
Examined Fuzzy RulesThey only examine short fuzzy rules with only a few antecedentconditions.
If x1 is small and x48 is large then Class 1 with 0.58
Rule Weight (Certainty Factor)The rule weight CF of each fuzzy rule is calculated from compatible training patterns.
LS
M
S LM S LM
LS
MCF=1.0(Maximum)
CF=0.37
Class 1Class 2
A-I Trade-Off: Some Example ApproachesMODEL 1: Multiobjective Rule SelectionConsequent ClassThe consequent class of each fuzzy rule is determined by compatible training patterns(i.e., the dominant class in the corresponding fuzzy subspace).
If x1 is small and x2 is large then Class 1 with 1.0
If x1 is small and x2 is small then Class 1 with 0.37
x1
x2
1. Heuristic Rule Extraction
Possible fuzzy rules: (15)n rules
Restriction on the rule length :Only short fuzzy rules
Rule evaluation criterion:The best rules for each class300 fuzzy rules for each class
A-I Trade-Off: Some Example ApproachesMODEL 1: Multiobjective Rule Selection
They extract a pre-specified number of the best fuzzy rules withrespect to a pre-specified heuristic rule evaluation criterion.
Coding:N: Total number of candidate rulessj={0, 1}: Inclusion or exclusion of the j-th rule
Objectives: f1(S), f2(S), f3(S)f1(S) : Number of correctly classified patterns by Sf2(S) : Number of selected rules in Sf3(S) : Total number of antecedent conditions in S
NsssS 21
A-I Trade-Off: Some Example ApproachesMODEL 1: Multiobjective Rule Selection2. Multiobjective Genetic Fuzzy Rule SelectionAlgorithm: Multi-objective Genetic Local Search (MOGLS)
• Selection based on a weighted fitness function (Number of correctly classified training patterns and number of rules)
• Tentative set of non-dominated solutions preserved externally• Elitist strategy: Nelite individuals of the population are randomly replaced with Nelite
individuals randomly extracted from the tentative set of non-dominated solutions
Maximize f1(S) and minimize f2(S)
Maximize
Maximize f1(S) and minimize f2(S), f3(S)
Maximize
Comparison of Four Approaches
)()( 2211 SfwSfw
)()()( 332211 SfwSfwSfw
A-I Trade-Off: Some Example ApproachesMODEL 1: Multiobjective Rule Selection
(1) Two-objective approach
(2) Weighted sum of the two objectives
(3) Three-objective approach
(4) Weighted sum of the three objectives
Data Sets Data set Attributes Patterns Classes LengthBreast W 9 683* 2 3Diabetes 8 768 2 3
Glass 9 214 6 3Heart C 13 297* 5 3
Iris 4 150 3 3Sonar 60 208 2 2Wine 13 178 3 3
Experimental Results (Cleveland Heart)
We can observe the overfitting due to the increase in the number of fuzzy rules.
(a) Error rates on training data (b) Error rates on test data
Erro
r rat
e on
trai
ning
pat
tern
s (%
)
Number of fuzzy rules
Two-objective rule selection Three-objective rule selection
6 8 10 12 14 16 18 20 22
28
30
32
34
36
38
40
Erro
r rat
e on
test
pat
tern
s (%
)
Number of fuzzy rules
Two-objective rule selection Three-objective rule selection
6 8 10 12 14 16 18 20 22
45
46
47
48
49
50
A-I Trade-Off: Some Example ApproachesMODEL 1: Multiobjective Rule Selection
Experimental Results (Sonar)
The generalization ability is increased by increasing the number of fuzzy rules (i.e., the overfitting is not observed).
Erro
r rat
e on
trai
ning
pat
tern
s (%
)
Number of fuzzy rules
Two-objective rule selection Three-objective rule selection
2 3 4 5 6 7 8
10
12
14
16
18
20
22
Erro
r rat
e on
test
pat
tern
s (%
)
Number of fuzzy rules
Two-objective rule selection Three-objective rule selection
2 3 4 5 6 7 820
22
24
26
28
(a) Error rates on training data (b) Error rates on test data
Observation(1) Experimental results showed that each test problem has a different
tradeoff structure.(2) Knowledge on the tradeoff structure is useful in the design of fuzzy
rule-based classification systems.
Error
Complexity
Test Data
Training Data0
Error
Complexity
Test Data
Training Data0
A-I Trade-Off: Some Example ApproachesMODEL 1: Multiobjective Rule Selection
SECOND TYPE: DATA BASE TUNING (+ RULE SELECT.) - REGRESSION
R. Alcalá, J. Alcalá-Fdez, M.J. Gacto, F. Herrera, A multi-objective geneticalgorithm for tuning and rule selection to obtain accurate and compactlinguistic fuzzy rule-based systems, International Journal of Uncertainty,Fuzziness and Knowledge-Based Systems, 15:5 (2007) 539–557
A-I Trade-Off: Some Example ApproachesMODEL 2: Multiobjective Tuning and Rule Selection
M.J. Gacto, R. Alcalá, F. Herrera, Adaptation and Application of Multi-Objective Evolutionary Algorithms for Rule Reduction and ParameterTuning of Fuzzy Rule-Based Systems. Soft Computing 13:5 (2009) 419-436
103
R. Alcalá, J. Alcalá-Fdez, M.J. Gacto, F. Herrera, A multi-objective genetic algorithm for tuning and rule selection to obtain accurate and compact linguistic fuzzy rule-based systems, International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems, 15:5 (2007) 539–557,
Multi-objective EAs are powerful tools to generate GFSs but they are based on getting a large, well distributed and spread off, Pareto set of solutions
– The two criteria to optimize in GFSs are accuracy and interpretability. The former is more important than the latter, so many solutions in the Pareto set are not useful
– Solution: Inject knowledge through the MOEA run to bias the algorithm to generate the desired Pareto front part
A-I Trade-Off: Some Example ApproachesMODEL 2: Multiobjective Tuning and Rule Selection
104
Pareto front classification in an interpretability-accuracy GFSs: Bad rules zone: solutions with bad
performance rules. Removing them improves the accuracy, so no Pareto solutions are located here
Redundant rules zone: solutions with irrelevant rules. Removing them does not affect the accuracy and improves the interpretability
Complementary rules zone: solutions with neither bad nor irrelevant rules. Removing them slightly decreases the accuracy
Important rules zone: solutions with essential rules. Removing them significantly decreases the accuracy
A-I Trade-Off: Some Example ApproachesMODEL 2: Multiobjective Tuning and Rule Selection
105
Accuracy-oriented modifications performed:
– Restart the genetic population at the middle of the run time, keeping the individual with the highest accuracy as the only one in the external population and generating all the new individuals with the same number of rules it has
– In each MOGA step, the number of chromosomes in the external population considered for the binary tournament is decreased, focusing the selection on the higher accuracy individuals
A-I Trade-Off: Some Example ApproachesMODEL 2: Multiobjective Tuning and Rule Selection
106
Obtained results for the medium voltage line problem:Multi-objective genetic tuning + rule selection method:
• 5-fold cross validation 6 runs = 30 runs per algorithm• T-student test with 95% confidence
A-I Trade-Off: Some Example ApproachesMODEL 2: Multiobjective Tuning and Rule Selection
M.J. Gacto, R. Alcalá, F. Herrera,Adaptation and Application of Multi-Objective Evolutionary Algorithms for Rule Reduction andParameter Tuning of Fuzzy Rule-Based Systems, Soft Computing 13:5 (2009) 419-436,
To perform the study we have applied six different approachesbased on the two most known and successful MOEAs:
Application of SPEA2 and NSGA-II Two versions of NSGA-II for finding knees,
NSGA-IIA and NSGA-IIU Two extensions for specific application,
SPEA2Acc and SPEA2Acc2
Two objectives are considered:MSE and Number of Rules
Proper operators have to be selected.
A-I Trade-Off: Some Example ApproachesMODEL 2: Multiobjective Tuning and Rule Selection
STUDY ON SEVERAL ALTERNATIVE APPROACHES AND IMPROVEMENTS: ADAPTATION AND APPLICATION OF MOEAs
NSGA-II FOR FINDING KNEES
A variation of NSGAII in order to find knees in the Pareto front by replacing the crowding measure by either an angle-based measure or an utility-based measure
In our case, a knee could represent the best compromise between accuracy and number of rules.
J. Branke, K. Deb, H. Dierolf, and M. Osswald, “Finding Knees in Multi-objective Optimization,” Proc. Parallel Problem Solving from Nature Conf. - PPSN VIII, LNCS 3242, (Birmingham, UK, 2004) 722–731.
Angle Based Approach
Utility Based Approach
Two different approaches
A-I Trade-Off: Some Example ApproachesMODEL 2: Multiobjective Tuning and Rule Selection
Objective: to improve the search with a more intelligentoperator replacing the HUX crossover in SPEA2ACC
Once BLX is applied a normalized euclidean distance iscalculated between the centric point of the MFs used by eachrule of the offpring and each parent
The closer parent determines if this rule is selected or not forthis offpring
Whit this crossover operator, mutation can be particularly usedto remove rules
Extension of SPEA2Acc (SPEA2Acc2)A New Crossover Operator for the Rule Part
A-I Trade-Off: Some Example ApproachesMODEL 2: Multiobjective Tuning and Rule Selection
Obtained results for the medium voltage line problem:
• 5-fold cross validation 6 runs = 30 runs per algorithm• T-student test with 95% confidence
Method #R MSEtra tra t MSEtst tst t
100,000 evaluations
WM 65.0 57605 2841 + 57934 4733 +
T 65.0 17020 1893 + 21027 4225 +
S 40.9 41158 1167 + 42988 4441 +
TS 41.3 13387 1153 + 17784 3344 +
TS-SPEA2 28.9 11630 1283 + 15387 3108 +
TS-NSGA-II 31.4 11826 1354 + 16047 4070 +
TS-NSGA-IIA 29.7 11798 1615 + 16156 4091 +
TS-NSGA-IIU 30.7 11954 1768 + 15879 4866 +
TS-SPEA2Acc 32.3 10714 1392 = 14252 3181 =
TS-SPEA2Acc2 29.8 10325 1121 * 13935 2759 *
A-I Trade-Off: Some Example ApproachesMODEL 2: Multiobjective Tuning and Rule Selection
111
Comparison of the SPEA2acc2 and classical GA for for the medium voltage line problem:
A-I Trade-Off: Some Example ApproachesMODEL 2: Multiobjective Tuning and Rule Selection
Convergence and an example model
A-I Trade-Off: Some Example ApproachesMODEL 2: Multiobjective Tuning and Rule Selection
THIRD TYPE: KNOWLEDGE BASE LEARNING - REGRESSION
R. Alcalá, P. Ducange, F. Herrera, B. Lazzerini, F. Marcelloni, A Multi-Objective Evolutionary Approach to Concurrently Learn Rule and DataBases of Linguistic Fuzzy Rule-Based Systems, IEEE Transactions on FuzzySystems 17:5 (2009) 1106-1122, doi:10.1109/TFUZZ.2009.2023113
A-I Trade-Off: Some Example ApproachesMODEL 3: Multiobjective Learning of DB and RB
R. Alcalá, P. Ducange, F. Herrera, B. Lazzerini, F. Marcelloni, A Multi-Objective Evolutionary Approach to Concurrently Learn Rule and Data Bases of Linguistic Fuzzy Rule-Based Systems 17:5 (2009) 1106-1122, IEEE Transactions on Fuzzy Systems, doi:10.1109/TFUZZ.2009.2023113,
Rule bases and parameters of the membership functions of theassociated linguistic labels are learnt concurrently.
Accuracy and interpretability are measured in terms ofapproximation error (MSE) and rule base complexity(#Conditions), respectively.
To manage the size of the search space, the linguistic 2-tuplerepresentation model, which allows the symbolic translation of alabel by only considering one parameter, has been exploited
A-I Trade-Off: Some Example ApproachesMODEL 3: Multiobjective Learning of DB and RB
This proposal decreases the tuning complexity, since the 3 parametersper label of the classical tuning are reduced to only 1 translationparameter (the tuning is applied to the level of linguistic partitions)
A-I Trade-Off: Some Example ApproachesMODEL 3: Multiobjective Learning of DB and RB
Coding Scheme and Operators
A double coding scheme (C = CRB+ CDB)
Crossover operator: one point + BLX- crossovers (2 offsprings)
Mutation operators:
Rule Adding: It adds random rules to the RB, where is randomly chosen in [1, max]
A-I Trade-Off: Some Example ApproachesMODEL 3: Multiobjective Learning of DB and RB
Modify RB: It randomly changes elements of the RB part. The number is randomly generated in [1, max]
Modify DB: It changes a gene value at random in the DB part
PAES, NSGA-II and SOGA were applied using this representation and crossover
Operators and Selection Schemes
A-I Trade-Off: Some Example ApproachesMODEL 3: Multiobjective Learning of DB and RB
Analysed Methods
Different population sizes were probed for these MOEAs showingbetter results when the population used for parent selection has similar sizes than those considered by single objective oriented algorithms.
300,000 evaluations to allow complete convergence in all thealgorithms
Method Description Pop. size
SOGARB Rule Base learning with SOGA 64
NSGA-IIRB Rule Base learning with NSGA-II 64
PAESRB Rule Base learning with PAES 64
SOGAKB (Rule Base + Data Base) learning with SOGA 64
NSGA-IIKB (Rule Base + Data Base) learning with NSGA-II 64
PAESKB (Rule Base + Data Base) learning with PAES 64
A-I Trade-Off: Some Example ApproachesMODEL 3: Multiobjective Learning of DB and RB
Average Pareto Fronts and average solution by SOGA (medium voltage lines problem)
1. Most accurate solution is selected from each Pareto
2. Average values are computed and represented
3. These solutions are no more used
4. Repeat to extract the desired avarage Pareto
Only the first 20 solutions are considered
5 Data partitions 80% - 20%6 Runs per partitionA total of 30 RunsTest t-student α = 0.05
A-I Trade-Off: Some Example ApproachesMODEL 3: Multiobjective Learning of DB and RB
REMINDER5 Data partitions 80% - 20%6 Runs per partitionA total of 30 RunsTest t-student α = 0.05
Statistical Analysis
Statistical comparison among MOEAs
Statistical comparison of the best MOEA with SOGA
A-I Trade-Off: Some Example ApproachesMODEL 3: Multiobjective Learning of DB and RB
Convergence
A-I Trade-Off: Some Example ApproachesMODEL 3: Multiobjective Learning of DB and RB
The models obtained by these new approachespresented a better trade-off than those obtained byonly considering performance measures.
Between both multi-objective experimented, namely amodified (2+2)PAES and the classical NSGA-II, themodified (2+2)PAES has shown a better behavior thanNSGA-II.
Finally, the linguistic 2-tuples representationpresented has shown a good positive synergy.
A-I Trade-Off: Some Example ApproachesMODEL 3: Multiobjective Learning of DB and RB
1. Basics on MOEFSs- Introduction to Genetic Fuzzy Systems (GFSs) and its main types- Evolutionary Multiobjective Optimization: Basic concepts and framework
2. Types of MOEFSs by multiobjective nature and optimized components
3. MOEFSs designed for the Interpretability-Accuracy Tradeoff of Fuzzy Systems: Two contradictory objectives- Interpretability issues in fuzzy systems design- Some example approaches
4. Other types of MOEFSs- MOEFSs designed for multi-objective control problems- MOEFSs designed for fuzzy association rule mining
5. New Research Directions in MOEFSs
Contents
MOEFSs for Multiobjective Control ProblemsBibliography on this category
Michela Fazzolari, Rafael Alcalá, Yusuke Nojima, Hisao Ishibuchi, Francisco Herrera. A review on the application of Multi-Objective Genetic Fuzzy Systems: current status and further directions. IEEE Transactions on Fuzzy Systems 21(1) (2013) 45-65.
• Most of them deal with the post-processing of FLC parameters (simplest with reduced search space)
• Earlier works consider 1st-gen. algorithms and only recently the 2nd-gen. have been applied (2006)
•Almost all of them are Linguistic and Mamdani-type based approaches
The multiobjective nature isspecific to each problem
In the following we will see a representatibeexample for the control HVAC Systems
Michela Fazzolari, Rafael Alcalá, Yusuke Nojima, Hisao Ishibuchi, Francisco Herrera. A review on the application of Multi-Objective Genetic Fuzzy Systems: current status and further directions. IEEE Transactions on Fuzzy Systems 21(1) (2013) 45-65.
MOEFSs for Multiobjective Control ProblemsAn example for the control of HVAC Systems
126
MOEFSs: APPLICATION TO A HVAC CONTROL PROBLEM
JOULE-THERMIE JOE-CT98-0090
Aire de Retorno
Aire del Exterior
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Aire de Salida
Aire Suministrado
A B C D E F G
H
IJ
Sala Sala1 n
+/-
h h
-/+
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
- Calentamiento
- Enfriamiento
Heating Ventilating and Air Conditioning Systems
127
Single Objective Previous Approaches
R. Alcalá, J.M. Benítez, J. Casillas, O. Cordón, R. Pérez, Fuzzy Control of HVAC Systems Optimized by Genetic Algorithms. Applied Intelligence 18:2 (2003) 155-177.
R. Alcalá, J. Casillas, O. Cordón, A. González, F. Herrera, A Genetic Rule Weightingand Selection Process for Fuzzy Control of Heating, Ventilating and Air Conditioning Systems. Engineering Applications of Artificial Intelligence 18:3 (2005) 279-296.
R. Alcalá, J. Alcalá-Fdez, M.J. Gacto, F. Herrera, Improving Fuzzy Logic ControllersObtained by Experts: A Case Study in HVAC Systems. Applied Intelligence 31:1 (2009) 10-35.
A new MOEFS to Solve the Problem
M.J. Gacto, R. Alcalá, F. Herrera, A Multi-Objective Evolutionary Algorithm for an Effective Tuning of Fuzzy Logic Controllers in Heating, Ventilating and Air Conditioning Systems. Applied Intelligence 36:2 (2012) 330-347
Models for Fuzzy Control of HVAC Systems
128
Fuzzy Logic Controllers for Energy Efficiency Consumption in Buildings
Energy consumption in buildings is the 40% of the total and more than a half is for indoor climate conditions
The use of appropriate automatic control strategies could result in energy savings ranging 15-85 %
Moreover, in current systems, several criteria are considered and optimized independently without a global strategy
Generic Structure of an Office Building HVAC System
It maintain a good thermal quality in summer and winter It dilutes and removes emissions from people, equipment and activities and supplies clean air
129
Initial Data Base17 Variables
Fuzzy Logic Controllers for Energy Efficiency Consumption in Buildings
Initial Rule Base and FLC Structure
130
Representation of the Test Cells
Two adjacent twin cells were available
A calibrated and validated model of this site was developed to evaluate each FLC
Fuzzy Logic Controllers for Energy Efficiency Consumption in Buildings
131
Goal: multi-criteria optimization of an expert FLC for an HVAC system: reduction ofthe energy consumption but maintaining the required indoor comfort levels
INITIAL RESULTS
MODELS #R PMV>0.501
PMV<-0.502
C02
03
ENERGY04 %
STABILITY05 %
ON-OFF - 0,0 0 0 3206400 - 1136 -
FLC 172 0,0 0 0 2901686 9,50 1505 -32,48
Fuzzy Logic Controllers for Energy Efficiency Consumption in Buildings
132
The controller accuracy is assessed by means of simulations which approximately take 3-4 minutes
The Lateral Tuning is combined with a Rule Selection
Necessity of efficient tuning methodologies:
Efficient adjustment of the MF parameters
Steady-State Genetic Algorithms were applied in the previous aproaches: quick convergence
2000 evaluations 1 run took approximately 4 days
Considering a small population (31 individuals)
A doble coding scheme is considered with the joining
of the selection binary values and the lateral parameters
C = CS CT
MOEFSs for Fuzzy Control of HVAC Systems: Problem Restrictions and Tuning Approach
133
Example of genetic lateral tuning and rule selection
MOEFSs for Fuzzy Control of HVAC Systems: Lateral Tuning + Rule Selection
134
Since the experts were able to provide trusted weights, performance criteria have been combined into a single function F. Thus the objectives are:
• Minimization of F (to improve the performance)
• Minimization of the number of rules (to favour the tuning efficiency)
The following mechanisms or operators have been integrated into the well-known SPEA2 algorithm to improve the Exploration/Exploitation trade-off
• An incest prevention mechanism as the well-known CHC algorithm
• Automatic restarting aplication to avoid local optima
• Progressive concentration on the most accurate solutions for parentselection
• An intelligent crossover operator
MOEFSs for Fuzzy Control of HVAC Systems:An Improved MOEA: SPEA2E/E
135
MOEFSs for Fuzzy Control of HVAC Systems: RESULTS
136
MOEFSs for Fuzzy Control of HVAC Systems: Pareto Fronts Obtained
The obtained fronts are not so wide but they dominatethe remaining wider ones
1. Basics on MOEFSs- Introduction to Genetic Fuzzy Systems (GFSs) and its main types- Evolutionary Multiobjective Optimization: Basic concepts and framework
2. Types of MOEFSs by multiobjective nature and optimized components
3. MOEFSs designed for the Interpretability-Accuracy Tradeoff of Fuzzy Systems: Two contradictory objectives- Interpretability issues in fuzzy systems design- Some example approaches
4. Other types of MOEFSs- MOEFSs designed for multi-objective control problems- MOEFSs designed for fuzzy association rule mining
5. New Research Directions in MOEFSs
Contents
• Predictive induction: Induces rule sets acting as classifiers for solving classification and prediction tasks
• Descriptive induction: Discovers individual rules describing interesting regularities in the data
Therefore: Different goals, different heuristics, different evaluation criteria
• One way to represent knowledge extracted with data mining techniques is by means of association rules, whose basic concept is to represent associations (simultaneity and not causality) between different pairs of sets of attribute values
MOEFSs for Fuzzy Association Rule MiningFuzzy Association Rule Mining
The use of fuzzy sets to describe associations between data: • extends the types of relationships that may be represented, • facilitates the interpretation of rules in linguistic terms, and • avoids unnatural boundaries in the partitioning of the attribute domains
MOEFSs for Fuzzy Association Rule MiningBibliography on this category
Michela Fazzolari, Rafael Alcalá, Yusuke Nojima, Hisao Ishibuchi, Francisco Herrera. A review on the application of Multi-Objective Genetic Fuzzy Systems: current status and further directions. IEEE Transactions on Fuzzy Systems 21(1) (2013) 45-65.
• In most cases, the classical measures of data mining, support and confidence, are used as objectives
• The application of MOEAs to extract fuzzy association rules is quite recent, beginning in 2006
• Therefore, the majority of works exploit a 2nd-generation MOEA
With respect to the multiobjective nature in this category, the aim of the optimization process is not only to improve the general trade-off between the usual metrics of the data mining for the whole set of rules, but also to obtain a large number of different rules, each of them satisfying the objectives to different degrees.
Michela Fazzolari, Rafael Alcalá, Yusuke Nojima, Hisao Ishibuchi, Francisco Herrera. A review on the application of Multi-Objective Genetic Fuzzy Systems: current status and further directions. IEEE Transactions on Fuzzy Systems 21(1) (2013) 45-65.
MOEFSs for Fuzzy Association Rule MiningAn example on Subgroup Discovery
In the following we will see a representatibeexample for Subgroup Discovery on Databases
Subgroup discovery is a process to identify relations between a dependent variable (target variable) and usually many explaining, independent variables.
For example, consider the subgroup described by
”smoker=true AND family history=positive”
for the target variable coronary heart disease=true.
Subgroup discovery does not necessarily focus on finding complete relations; instead partial relations, i.e., (small) subgroups with ”interesting” characteristics can be sufficient.
MOEFSs for Subgroup DiscoveryHow does subgroup discovery work?
• Non-dominated Multi-objective Evolutionary algorithm based on Fuzzy rules extraction for Subgroup Discovery (NMEEF-SD)
MOEFSs for Subgroup DiscoveryNMEEF-SD
C. J. Carmona, P. González, M. J. del Jesus, and F. Herrera,
“NMEEF-SD: Non-dominated Multiobjective Evolutionary Algorithm for Extracting Fuzzy Rules in Subgroup Discovery”,
IEEE Transactions on Fuzzy Systems, vol. 18, no. 5, pp. 958–970, 2010
• Each candidate solution is codified according to the “Chromosome = Rule“ approach, where only the antecedent is represented
• NMEF-SD is able to work with crisp or fuzzy rules
• The fuzzy logic:– Is used in continuous variables– Linguistic labels are defined by means of the corresponding membership
functions– Defines uniform partitions with triangular membership functions
MOEFSs for Subgroup DiscoveryNMEEF-SD
• NMEEF-SD can extract canonical or DNF rules.
– For the canonical rules, only the antecedent is represented through a conjunction of value-variable pairs.
IF X1 = Value3 AND X4 = LL42 THEN Class 2
– For the DNF rules extension, a fixed-length binary representation is usedIF X1 = (Value1 OR Value3) AND X3 = LL31 THEN Class 2
X1 X2 X3 X4
1 0 1 0 0 0 0 0 1 0 0 0 0 0 0 0 0
MOEFSs for Subgroup DiscoveryNMEEF-SD
X1 X1 X1 X1
3 0 0 2
MOEFSs for Subgroup DiscoveryNMEEF-SD
Operation diagram of NMEEF-SD
• This algorithm is based on NSGA-II approach.
• The quality measures selected as objectives:– Support (SupcN)– Unusualness (WRAcc)
• To create an initial population whose size is prefixed by an external parameter.
• A part of the population (75%) using only a maximum percentage of the variables (25% of the rule) which form part of the rule.
• The rest of variables and rules of the population are randomly generated.
• This operator obtains a set of rules with a high generality.
Biased initialisation
MOEFSs for Subgroup DiscoveryNMEEF-SD
• The algorithm uses different operators:– Tournament Selection– Multi-point Crossover– Biased Mutation
Genetic operators
3 2 4 2Two types of mutation (50%)
0 2 4 2Variable eliminated
1 2 4 2Value modified
MOEFSs for Subgroup DiscoveryNMEEF-SD
• The algorithm joins two populations in only one:– Initial population– Offspring population
• The algorithm applies the fast non-dominated sort over the population obtained previously.
• The individuals of the population are classified in fronts of dominance.
• The first front is the Pareto front.
• The algorithm obtains diversity with the operator of crowding distance.
Fast non-dominated sort
MOEFSs for Subgroup DiscoveryNMEEF-SD
• When the algorithm obtains the fronts of dominance checks the evolution of the Pareto front.
If the Pareto front evolves during more than five percent of the evolutive process
Re-Initialisation based on coverage
EVOLVES
1. Introduce the fronts in the next population.
2. If the front has more individuals than can enter in the population, the inviduals are introduced by greater crowding distance.
DOES NOT EVOLVE
1. Eliminates the individualsrepeated in the Pareto front.
2. Replaces these individuals withnew individuals generatedbased on coverage.
MOEFSs for Subgroup DiscoveryNMEEF-SD
• The evolutionary process ends when the number of evaluations is reached.
• The algorithm returns the rules in the Pareto front which reach a predefined fuzzy confidence value threshold.
• The fuzzy confidence is defined in:
Stop condition
MOEFSs for Subgroup DiscoveryNMEEF-SD
M.J. Del Jesus, P. González, F. Herrera, M. Mesonero
Evolutionary fuzzy rule induction process for subgroup discovery: a case study in Marketing
IEEE Transactions on Fuzzy Systems, Vol. 15 (4), 2007, pp. 578-592.
• Different data sets available in UCI repository has been carried out: Australian, Balance, Echo and Vote.
http://www.ics.uci.edu/~mlearn/MLRepository.html
• Ten fold cross validation
• Algorithms compared:– Evolutionary algorithms SDIGA and MESDIF.– Classical methods CN2-SD and Apriori SD.
• Parameters for NMEF-SD:– Population size: 25– Maximum number of evaluations: 5000– Crossover probability 0.60 and mutation probability 0.01
Experimentation
MOEFSs for Subgroup DiscoveryNMEEF-SD
Experimentation
MOEFSs for Subgroup DiscoveryNMEEF-SD
Database Algorithm Rul Var COV SIGN WRAcc SUPcN FCNF
Australian
NMEF-SD 3.58 2.92 0.454 23.178 0.171 0.783 0.930
MESDIF 10.00 3.52 0.311 7.594 0.060 0.577 0.807
SDIGA 2.68 3.28 0.310 16.348 0.120 0.803 0.591
CN2-SD 30.50 4.58 0.400 15.350 0.055 0.649 0.830
AprioriSD 10.00 2.02 0.377 16.998 0.074 0.654 0.863
Balance
NMEF-SD 2.30 2.00 0.362 5.326 0.070 0.530 0.698
MESDIF 28.10 3.08 0.163 3.516 0.022 0.318 0.557
SDIGA 7.40 2.39 0.291 5.331 0.049 0.487 0.664
CN2-SD 15.60 2.23 0.336 8.397 0.063 0.512 0.583
AprioriSD 10.00 1.20 0.333 5.444 0.058 0.480 0.649
Echo
NMEF-SD 3.62 2.35 0.428 1.293 0.043 0.628 0.757
MESDIF 19.74 3.30 0.164 0.877 0.017 0.355 0.591
SDIGA 2.32 2.27 0.394 1.165 0.013 0.566 0.590
CN2-SD 17.30 3.23 0.400 1.181 0.019 0.490 0.667
AprioriSD 9.80 1.70 0.194 0.901 0.034 0.226 0.510
Vote
NMEF-SD 1.10 2.05 0.577 21.974 0.217 0.946 0.979
MESDIF 7.86 3.44 0.429 19.937 0.187 0.827 0.957
SDIGA 3.06 3.19 0.422 18.243 0.180 0.802 0.891
CN2-SD 8.00 1.79 0.438 18.830 0.176 0.858 0.932
AprioriSD 10.00 1.44 0.428 17.060 0.147 0.800 0.930
• When analysing the results is important to take into account:– The relation between Support and Confidence.– Good results in the quality measures of Subgroup Discovery:
Unusualness and Significance.– A good interpretability of the results.
• NMEF-SD obtains:– The best results for the quality measures in the data sets selected.– Better results in generality and precision than others.– The subgroups are good, useful and representative.
Experimentation
MOEFSs for Subgroup DiscoveryNMEEF-SD
1. Basics on MOEFSs- Introduction to Genetic Fuzzy Systems (GFSs) and its main types- Evolutionary Multiobjective Optimization: Basic concepts and framework
2. Types of MOEFSs by multiobjective nature and optimized components
3. MOEFSs designed for the Interpretability-Accuracy Tradeoff of Fuzzy Systems: Two contradictory objectives- Interpretability issues in fuzzy systems design- Some example approaches
4. Other types of MOEFSs- MOEFSs designed for multi-objective control problems- MOEFSs designed for fuzzy association rule mining
5. New Research Directions in MOEFSs
Contents
Current and Future Research Directionsin MOEFSs1) Development of New MGFS Methods with Improved Algorithms- Particular algorithms for multiobjective input selection- Particular algorithms for multiobjective fuzzy partition learning- . . .
An example for learning granularities and selecting conditions can be found in:
M. Antonelli, P. Ducange, B. Lazzerini, and F. Marcelloni, “Learning concurrently partitiongranularities and rule bases of Mamdani fuzzy systems in a multi-objective evolutionary framework,” Int. J. Approx. Reason., vol. 50, n. 7, pp. 1066–1080, 2009.
M. Antonelli, P. Ducange, B. Lazzerini, and F. Marcelloni, “Multi-objective evolutionary learning of granularity, membership function parameters and rules of Mamdani fuzzy systems,” Evolutionary Intelligence, vol. 2, n. 1-2, pp. 21–37, 2009.
Exploiting the concept of virtual partitions with modified PAES
Current and Future Research Directionsin MOEFSs (2)1) Development of New MGFS Methods with Improved Algorithms (2)An example for learning granularities and for selecting variables can be found in:
2) Performance evaluation of MOGFSs• Visualization of Pareto-Optimal Fuzzy Systems• How to compare MGFSs
- A statistical Analysis is needed- Use of non-parametric statistical tests
R. Alcala, M. J. Gacto, and F. Herrera, “A Fast and Scalable Multi-Objective Genetic Fuzzy System forLinguistic Fuzzy Modeling in High-Dimensional RegressionProblems,” IEEE Transactions on Fuzzy Systems 19:4 (2011) 666-681, doi: 10.1109/TFUZZ.2011.2131657
Exploiting the embedded learningof the DB with improved SPEA2
Evaluation indexes in the EMO frameworkevaluate the exploration and exploitationcapabilities of the MOEA. But we are alsointerested in generalization capabilities ofthe FRBSs
Current and Future Research Directionsin MOEFSs (3)2) Performance evaluation of MOGFSs• How to compare MGFSsA recent possibility to apply non-parametric statistical tests:
An extension for the case of more than twoobjectives:
R. Alcalá, P. Ducange, F. Herrera, B. Lazzerini, and F. Marcelloni, “A Multi-objective evolutionary approachto concurrently learn rule and data bases of linguisticfuzzy rule-based systems,” IEEE Trans. Fuzzy. Syst., vol. 17, n. 5, pp. 1106–1122, 2009.
Analyzing the averages onthree representative points bynon-parametric statistical testsfor bi-objective problems(FIRST, MEDIAN, LAST)
M. J. Gacto, R. Alcala, and F. Herrera, “Integration of an index to preserve the semantic interpretability in the multi-objective evolutionary rule selection and tuning of linguistic fuzzy systems,” IEEE Trans. Fuzzy. Syst. vol. 18, n.3, pp. 515-531, 2010.
Projections on bi-objective planes.Then, representativepoints can be obtained in the new non-dominated solutions
Current and Future Research Directionsin MOEFSs (4)3) Reliable Interpretability Measures (Formulations of the Interpretability)- We need well established and accepted measures- Use of new ones for C3 (semantic-RB) as cointension or number of fired rules
The use of relative measures for C4 (semantic-DB) could be promising. First proposal in:
Some recent approaches are also using this kind of measures:
M. Antonelli, P. Ducange, B. Lazzerini, and F. Marcelloni, “Learning knowledge bases of multi-objective evolutionary fuzzy systems by simultaneously optimizing accuracy, complexity and partition integrity, Soft Computing vol. 15, n.12, pp. 2335–2354, 2011.
Measuring the differences to a givenlinguistic partition (obtained fromexperts or automatically by usingabsolute measures): GM3M indexbased on three metrics
M. J. Gacto, R. Alcala, and F. Herrera, “Integration of an index to preserve the semantic interpretability in the multi-objective evolutionary rule selection and tuning of linguistic fuzzy systems,” IEEE Trans. Fuzzy. Syst. vol. 18, n.3, pp. 515-531, 2010.
displacement aspect area
Current and Future Research Directionsin MOEFSs (5)4) Objective dimensionality- New EMO algorithms- Aggregation or selection of a reasonable set of significant measures
5) Scalability issues- High Dimensinality (handling the length of the rules)- Large scale problems (using a reduced subset of examples)
Some approaches dealing with large scale problems:
• M.A. de Vega, J.M Bardallo, F.A. Marquez, A. Peregrin, “Parallel distributed two-levelevolutionary multiobjective methodology for granularity learning and membership functionstuning in linguistic fuzzy systems,” in Proc. of ISDA 2009, pp. 134–139.
• M. Cococcioni, B. Lazzerini, F. Marcelloni, “On reducing computational overhead in multi-objective genetic Takagi–Sugeno fuzzy systems,” Appl. Soft Computing 11:1 (2011), 675-688.
• M. Antonelli, P. Ducange, F. Marcelloni, “Exploiting a coevolutionary approach to concurrently select training instances and learn rule bases of Mamdani fuzzy dystems,” in Proc. of WCCI 2010, 1366–1372.
Parallelization
Fitnessestimation
InstanceSelection
5) Scalability issues (2)
Some approaches dealing with high dimensional problems:
An approach dealing with both high dimensional and large scale problems:
Current and Future Research Directionsin MOEFSs (6)
• H. Ishibuchi, and T. Yamamoto, “Fuzzy rule selection by multi-objective genetic local search algorithms and rule evaluation measures in data mining,” Fuzzy Sets and Systems, vol. 141, pp. 59–88, 2004.
• M. Antonelli, P. Ducange, B. Lazzerini, F. Marcelloni, “Multi-objective EvolutionaryGeneration of Mamdani Fuzzy Rule-Based Systems based on Rule and Condition Selection,” in Proc. of GEFS 2011.
Imposing a maximum rule lenght
Conditionselectionby specificapproach
• R. Alcala, M. J. Gacto, F. Herrera, “A Fast and Scalable Multi-Objective Genetic Fuzzy System for Linguistic Fuzzy Modeling in High-Dimensional Regression Problems,” IEEE Trans. on Fuzzy Systems 19:4 (2011) 666-681.
Using a specific approach for variable selection and fitnessstimation by using a short subset of the examples
6) Automatic selection of the most suitable solution- Determining those solutions with the best generalization ability- Only training data can be took into account
A recent approach on this topic:
Current and Future Research Directionsin MOEFSs (7)
• Ishibuchi H, Nakashima Y, Nojima Y, Double cross-validation for performance evaluation of multi-objective genetic fuzzy systems. In GEFS 2011, pp 31-38.
Using a double cross-validation with two cross-validationloops. The inner loop uses the training data to determine the complexity of the systems with the best validationmeasure, which is used to select the solutions used for theouter loop.
Rafael Alcalá
Dpto. Computer Science and A.I.University of Granada
18071 – SPAIN
Multi-Objective Evolutionary Fuzzy Systems: An Overview by Problem objectives
nature and optimized components
FUZZ-IEEE 2013 Tutorial, Hyderabad, IndiaAfternoon Session: 14:00-17:00, July 7, 2013
Thank you very much for your attention !!!Questions?