Robustness and sensitivity analysis in multiple criteria decisionproblems using rule learner techniques

Claudio M. Rocco S. n, Elvis HernandezUniversidad Central de Venezuela, Caracas, Venezuela

a r t i c l e i n f o

Keywords:Monte Carlo simulationMultiple criteria decisionRule learnersUncertainty

a b s t r a c t

In many situations, a decision-maker is interested in assessing a set of alternatives characterizedsimultaneously by multiple criteria (attributes), and defining a ranking able to synthesize the globalcharacteristics of each alternative, for example, from the best to the worst. This is the case of theassessment of several projects through attributes such as cost, profitability, among others. The behavior ofeach object, for every criterion, is quantified via numerical or categorical “performance values”. Severalmultiple criteria decision techniques could be used to this aim. However the base rank could be influencedby uncertain factors associated to specific criteria (e.g., the “ratio Benefit/Cost of a project” could beaffected by variations in the interest rate) or by decision-maker preferences. In this situation, the decision-maker could be interested knowing what sets of factors are responsible of specific ranking conditions.

This paper describes the input space of a set of factors responsible of a given model behaviorspecification, based on the use of rule learners able to provide a description through a set of “If-Then”rules derived from model samples. These techniques also allow determining the most important factors.An example related to a real decision problem illustrates the proposed approach.

& 2014 Elsevier Ltd. All rights reserved.

1. Introduction

In many situations, a decision-maker is interested in assessing aset of m objects or alternatives ai characterized simultaneously byn criteria or attributes, and defining a ranking able to synthesizethe global characteristics of each object, e.g., from the best to theworst. This is the case, for example, in the assessment of severalengineering projects through attributes such as cost, availability,environmental impact, among others. The behavior of each object,for every criterion, is quantified via performance values PVij (foreach alternative i¼1, .., m and for each criterion j¼1, … ,n) whichcan either be numerical or categorical.

The idea of ranking alternatives is based on one of the fourdiscrete decision-making problems defined as “Problematique γ” in[1], that is, ranking the alternatives from the best to the worstones. Several multicriteria decision techniques (MC) or rankingtechniques could be used to this aim [2]. Ranking techniques togenerate the desired rank are classified as parametric and non-parametric. The first group, like ELECTRE [1], PROMETHEE [3],TOPSIS [4] to name a few, requires information about decision-maker preferences (e.g., criterion weights), while non-parametrictechniques (partial order ranking [5], Hasse diagram technique [6]and Copeland Scores [7]) do not use such information.

In general, the ranking assessment is performed as follows:

1. Define m alternatives and n criteria.2. Define the multi-indicator matrix Q, based on each PVij (for

each alternative i¼1, .., m and for each criterion j¼1, .., n).3. Select a ranking technique.4. Produce a rank of objects according to the selected technique.

However, no matter which MC technique is selected, theranking derived using crisp PV (defined as the Base Rank (BR)),could be influenced by uncertain factors associated to specificcriteria (for example, the criterion “Cost/Benefit ratio of a project”could be affected by variations in the interest rate) or by decision-maker preferences (e.g., criterion weights).

If these uncertain factors are modeled as a probability distribu-tion function then the rank of each alternative could be consideredas a random variable. Several authors [8–11] have analyzed thisproblem: how the uncertainty in the PV (the input) is propagatedor affects the object ranks (the output)?

Recently, Rocco and Tarantola [12] presented two approachesthat extend previous works in two directions:

1. Ranking assessment: based on Monte Carlo simulation, theapproach allows answering several questions regarding rankingrobustness. For example, under uncertainty: What is the prob-ability that the base rank position is maintained? Which is the

Contents lists available at ScienceDirect

journal homepage: www.elsevier.com/locate/ress

Reliability Engineering and System Safety

http://dx.doi.org/10.1016/j.ress.2014.04.0220951-8320/& 2014 Elsevier Ltd. All rights reserved.

n Corresponding author at: Apartado Postal 47937, Los Chaguaramos 1041 A,Caracas, Venezuela. Tel.: +58 412 2528346.

E-mail address: croccoucv@gmail.com (C.M. Rocco S.).

Please cite this article as: Rocco S. CM, Hernandez E. Robustness and sensitivity analysis in multiple criteria decision problems usingrule learner techniques. Reliability Engineering and System Safety (2014), http://dx.doi.org/10.1016/j.ress.2014.04.022i

Reliability Engineering and System Safety ∎ (∎∎∎∎) ∎∎∎–∎∎∎

rank position with the highest probability? What are the possiblerank positions and their corresponding probability?

2. Sensitivity analysis: this approach, based on global sensitivityanalysis techniques [13], allows evaluating the importance ofuncertain factors.

This type of analysis, from input to output, can provide to thedecision-maker a sharper picture of the effects of the uncertaintyin the final ranking that MC techniques provide. Therefore, thedecision-maker can have a better perspective of how stable his/herfinal decision is and often needs to know which factors determinespecific output behavior (output specifications). For example, whatare the values associated to each criterion that make a particularproject be ranked as the best project?

Procedures to cope with such problems are termed as FactorMapping setting, “in which specific points/portions of the modeloutput realizations, or even the entire domain, are mapped back-wards onto the space of the input factors” [14]. Note that thesolution space could be a non-convex and/or sparse space [15–17].

Several approaches have been proposed in the literature toproduce such mapping like Monte Carlo Filtering [13,14], RegionalSensitivity Analysis [18], Generalized Likelihood Uncertainty Esti-mation [19] and Tree-Structured Density Estimation [20].

Other approaches, based on optimization instead of mappingfrom the output into the input space, have been suggested in [15] orrecently in [16,17]. These approaches are able to extract the max-imumvolume hyperbox of the solution space, where factor variationsare assigned independently. The solution space is representedthrough intervals [x1_inf, x1_sup], [x2_inf, x2_sup], …, and [xl_inf,xl_sup], where xj is the jth factor and l is the number of uncertainfactors considered. Fig. 1 illustrates the approach in the case of twofactors x1 and x2. The area delimited by dashed lines defines thefeasible zone. The rectangle (solid lines) represents the box withmaximum area.

The approach proposed in [15] requires an analytical model f(x1, x2, .., xl), while in [16,17] the model is considered as a black-box. In both approaches the hyperbox could be centered at apredefined feasible point or freely centered across the feasiblezone. The widths of the final intervals that define the solutionspace could be considered as a sensitivity index.

This paper proposes an approach based on the use of machinelearning classification techniques [21] able to provide a descriptionof the solution space, based on a set of “If (premise) then(consequence)” rules derived from model samples (i.e., could beused for analytical or black-box models) where (premise) is acondition (or the logical product of several conditions) relatedto a specific factor or variable whereas (consequence) gives a classassignment. For example, for a given project B, the structure of thehypothetical rule

If (Cost/Benefit_Project_B48 AND Employment_ProjectB4120) then

(Rank_project_B¼1)explains when project B is ranked as the first project.Each rule extracted represents a specific hyperbox of the

solution space. This allows to model non-convex solution spaces.Additionally, some rule generation techniques are able to extractthe most important factor, can detect non important factors or canprovide a numerical sensitivity index.

The rest of the paper is organized as follows: Section 2 describesthe problem to be analyzed and proposes a solution based onmachine learning classification techniques. Section 3 presents anoverview of rule generations concepts and examples of solutionsthrough several algorithms. Section 4 describes a case study. Finally,Section 5 shows the conclusions and future work.

2. The problem

2.1. Problem statement

Let A¼{a1, a2, …, am} be a set of m projects, G¼{g1, g2, …, gn}the set of n criteria considered in the evaluation with theirrespective direction of improvement (for example: a high valueis better). Each aiAA is defined by a set of n values that representsthe evaluation of each criterion for the project ai. In general, eachcriterion is considered as a defined mathematical function (e.g.,availability). In this paper PVij means the assessment of project iunder criterion j.

Let RI¼{RI1, RI2, …, RIn} be a set of values that model DM'spreferences (or weights) over the selected criteria with

∑n

j ¼ 1RIn ¼ 1; RInZ0.

Let F() be a particular ranking technique: Given PV and RI, F(PV,RI) is able to produce the ranking of a set of projects under study,i.e., R¼[r1, r2, …, rn]T, where rk is the rank position of project k.Although a particular technique is represented as a function F(), itdoes not mean that F() has an analytical definition and it isconsidered as a “black-box” function. For example, the well knownranking methods PROMETHEE [3] (the technique selected inSection 4 to illustrate the proposed approach) defines a multi-step procedure for

a) normalizing PV;b) performing all pair-wise comparisons and distance computa-

tions among projects;c) assessing the differences between projects using “generalized

criterion” expressions;d) computing the preference-weighted aggregation and the posi-

tive and negative outranking flows for the each project;e) determining the overall “quality” of each project using the net

flows andf) producing the final ranking R.

Depending on how the preference function is modeled, thePROMETHEE methods may need the definition of additional para-meters other than the weights of attributes.

Let R0 be the base rank obtained when no uncertainty isconsidered. For example, if m¼4, R0¼[3,1,4,2]T means that project1 occupies the third position, project 2 is the best ranked, and soon. Of course, a reverse ranking order could be used to define thebest project.

Suppose that all of the performance values PV and RI areconsidered as inputs whose uncertainties are modeled as randomvariables properly characterized through known probability dis-tribution functions (pdf). That means that the R is now a randomFig. 1. Maximum area box (solid lines) in the feasible zone defined by dashed lines.

C.M. Rocco S., E. Hernandez / Reliability Engineering and System Safety ∎ (∎∎∎∎) ∎∎∎–∎∎∎2

Please cite this article as: Rocco S. CM, Hernandez E. Robustness and sensitivity analysis in multiple criteria decision problems usingrule learner techniques. Reliability Engineering and System Safety (2014), http://dx.doi.org/10.1016/j.ress.2014.04.022i

variable, and its elements ri have pdfs that represent the distribu-tions of the possible ranks associated to project i. The problemaddressed in this paper is then: Considering uncertainty, underwhat conditions of PV and RI, a given constraint is satisfied? Theterm constraint could mean a set of specifications that thedecision-maker defines in terms of project ranking. If m¼4 andR0¼[3,1,4,2]T examples of such constraints could be: (a) Project2 is still ranked as the first; (b) The rank of project 4 is always r2;Projects 1 and 3 are ranked as the third and fourth respectively.

2.2. Problem solution

To solve the problem previously posed, the following procedureis proposed:

Nsample random deviates are generated for PV and RI, accord-ing to their known probability distribution function: PVk and RIk,k¼1, .., Nsample;

Each sample (PVk, RIk) is evaluated using F(). Then Rk¼F(PVk,RIk).

Let ykA[0,1] be the status of the kth sample: 1 if the constraintis satisfied; 0 otherwise.

Derive the condition of PV and RI using a specific rule genera-tion technique, as described in the next section.

3. Machine learning methods

3.1. Introduction

This section reviews some concepts related to Machine Learn-ing (ML) techniques with special attention to those methodsdesigned to solve the two-class classification problem, where adecision function g: ℜd-{0,1} have to be reconstructed startingfrom a collection of Nsamples (V1,y1), (V2,y2), …, (VN,yN), being V¼(V1, V2, …, Vd) the set of variables under analysis, d the number ofvariables and every yj is a (possibly noisy) evaluation of apredefined function g(Vj) for every j¼1, …, N [22].

In the context of this paper, the vector V is defined by eachperformance value in PV and each weight in RI whereas g(Vj) is thestatus of the jth sample derived from applying the predefinedconstraint to F(PVj,RIj), i.e., g(Vj)¼1 if the constraint is satisfied;0 otherwise.

Since m is the number of projects, and n the number of criteria,then the number of variables to be considered is d¼mnþn¼n(mþ1). The codification of the variables is defined as: the mn PVvalues are mapped column-wise to V1, V2, …, Vmn and the rest ofthe n RI are mapped to Vmnþ1, Vmnþ2, …, Vd. For example, if m¼4and n¼3, then d¼3(4þ1)¼15 and V1¼PV11; V2¼PV21;…;V4¼PV41; V4þ1¼PV21; V6¼PV22; …; V12¼PV43;V13¼RI1; V14¼RI2and V15¼R3.

Two different situations can be identified: ML predictivemethods reconstruct the desired g(V) through a black box device,whose functioning is not directly comprehensible. On the contrary,ML descriptive methods provide a set of intelligible rules describ-ing the behavior of the g(V) for the system at hand. Support VectorMachines [21] and Neural Networks [21] are two examples ofpredictive method, whereas Decision Trees (DT) [21], RIPPER [23],PART [24] and Logic Learning Machine (LLM) [25] are examples ofdescriptive methods that allow discovering relevant properties formulticriteria project evaluations.

3.2. Descriptive methods

Descriptive methods able to produces a set of “If (premise) then(consequence)” rules are known as rule generation techniques andare classified as direct and indirect methods. Direct methods

extract rules directly from data whereas indirect methods extractrules from other classification models. RIPPER [23] and LLM [25]are two examples of direct methods. C4.5 [21] (extracts rules froma DT) and FALCON-ART [26] (extracts fuzzy rules from a NeuralNetwork) are two examples of indirect methods.

A common characteristic of rule generation techniques is thefact that the rules produce not only the subset of variables actuallycorrelated, but also the conditions that satisfy a specified con-straint [25]. Rules are determined for each possible class (1 ifconstraint is satisfied; 0 otherwise), and their performance couldbe assessed using the covering, the error and the relevancestatistics (see [25] for details).

3.3. Classification procedure

In order to apply a classification method it is first necessary tocollect a set of examples (V,y). A subset, defined as the training set(TrS) is used to tune-up a specific model (training phase) while asecond subset, known as the testing set (TeS) is used to assess thepredictive capability of the model (testing phase). It is importantto realize that TrS\TeS¼∅. A common rule of thumb is to use 70%of the samples for training and 30% for testing.

The performance of each classifier during the training phase isevaluated using standard measures of sensitivity, specificity andaccuracy [25]:

sensitivity¼ TPTPþFN

; specificity¼ TNTNþFP

; accuracy¼ TPþTNTPþTNþFPþFN

where [22]

� TP (resp. TN) is the number of examples belonging to the classy¼1 (resp. y¼0) for which the classifier gives the correctoutput,

� FP (resp. FN) is the number of examples belonging to the classy¼1 (resp. y¼0) for which the classifier gives the wrongoutput.

For multicriteria project evaluation, sensitivity gives the percen-tage of samples that satisfies the constraint, and specificity providesthe percentage of samples that do not satisfy the constraint.

A set of indexes has also been defined for assessing theperformance of each single rule, such as “the covering (the fractionof examples in the training set that verify the rule and belong tothe target class) or the error (the fraction of examples in thetraining set that satisfy the rule and do not belong to the targetclass)” [25].

As previously mentioned, each premise in a rule is defined as aspecific condition on a particular variable Vi, or the logical productof several conditions. This fact allows for assessing the relevance ofevery condition in a rule by evaluating the error of the rule withand without the selected condition. On the other hand, it is alsopossible to evaluate the relevance or importance of every variableVi in the set of rules (see [25] for details).

Once the training phase is concluded, the next step is thetesting phase. The set of rules generated are used to predictthe class of the samples in the testing set and the performanceof the classifier.

3.4. Using rule generation techniques

The main idea of this section is to highlight important aspectsthat must be considered when rule learners are used and inparticular to illustrate the capability of some rule generationtechniques as implemented in the package RWeka [27] (underthe R environment), as well in the Rulex software suite 2.0 [28]

C.M. Rocco S., E. Hernandez / Reliability Engineering and System Safety ∎ (∎∎∎∎) ∎∎∎–∎∎∎ 3

Please cite this article as: Rocco S. CM, Hernandez E. Robustness and sensitivity analysis in multiple criteria decision problems usingrule learner techniques. Reliability Engineering and System Safety (2014), http://dx.doi.org/10.1016/j.ress.2014.04.022i

(for illustrative purpose, both software where executed using theirdefault settings i.e., no optimization of the settings is performed).

It is important to mention that different techniques usedifferent criteria for developing the set of rules. As a result, usingthe same training set, different values of a performance indexcould be obtained. It is important to realize that the examplespresented do not constitute a formal comparison among rulegeneration algorithms. (Note: both packages automatically nor-malize the quantities of the set of samples).

Let Y¼V1�V2þ0�V3, the model to be analyzed, where thethird factor has no influence [29]. The constraint selected is Y40.Each Vi is sampled from a uniform distribution in [�0.5, 0.5]. Inthis problem, it is clear that there are two regions that satisfy theconstraint: (a) V1 and V240; and (b) V1 and V2o0.

To illustrate the effects of the number of samples on thedefinition of the rules, two cases are evaluated with differentsamples: Case 1: 1000 samples and Case 2: 5000. In each case,samples are generated using deviates for each factor. The trainingset is randomly selected with the 70% of the total samples and therest of samples are assigned to the testing data set.

Case 1. 1000 samples

a) The DTþC4.5 procedure (implemented in the Rulex software)extracts 9 rules (see Table 1). Note that factor V3 does notappear in any of the rules. This fact suggests, as expected, thatthis factor has no influence.The error of the classifier during the training phase is 0. Notethat some rules seem strange. For example the rule “IF(V140.484) THEN Y¼1” means that no matter the value ofV2, as long as V140.484, Y¼1. Since the error during thetraining phase is 0, it means that there are samples (2.7% of thetotal training set, as evaluated by Rulex) that induce such rule.During the testing phase the following indexes are derived:Accuracy¼96.8%.TP¼97.321%; FP¼2.679%TN¼96.377%; FN¼3.623%

b) The LLM procedure (through the Rulex software) extracts4 rules. As in the previous case, the factor V3 does not appearin any of the rules. Note than in this case a more compact set ofrules are derived even if the conditions for V1 and V2 are notexactly derived, i.e., (a) V1 and V240; and (b) V1 and V2o0.The error of the classifier during the training phase is 0. Duringthe testing phase the following indexes are derived:Accuracy¼98.5%.TP¼98.214%; FP¼1.786%TN¼98.551%; FN¼1.449%Note that the accuracy of the LLM is greater that the accuracyof DT.

c) The JRip procedure (implemented in RWeka) extracts 4 rules.In JRip, the output is expressed as a logical condition: TRUEif the constraint is satisfied; FALSE otherwise, e.g.: IF ((V14¼0.030064) AND (V24¼0.00032)) THEN Y¼TRUE.The error of the classifier during the training and testingphases are

Training phase Testing phase

Accuracy¼99.85% Accuracy¼99.33%TP¼100%; FP¼0% TP¼100%; FP¼0%TN¼99.722%; FN¼0.278% TN¼98.795%; FN¼1.205%

Note that the rules derived by JRip could not be minimal, requiringsome additional simplification. For example, the rule:

(V1o¼0.017804) and (V2o¼�0.002293) and (V1o¼�0.006248)¼4Y¼TRUE is simplified to: (V1o¼�0.006248)

and (V2o¼�0.002293)¼4Y¼TRUE

Case 2. 5000 samples

a) The DTþC4.5 procedure (through the Rulex software) extracts7 rules (see Table 2). Note that factor V3 does not appear in anyof the rules. This fact suggests, as expected, that this factor hasno influence. The error of the classifier during the trainingphase and testing phase is 0.

b) The LLM procedure (through the Rulex software) extracts 2 rules.The error of the classifier during the training phase and testingphase is 0. Note that the two feasible regions are correctlyextracted.

Table 1Rules derived Case 1.

DTþC4.5

IF (V1o¼�0.220 AND V2o¼�0.011) THEN Y¼1

IF (�0.220oV1o¼�0.013 AND V2o¼�0.012) THEN Y¼1

IF (�0.220oV1o¼�0.011 AND �0.012oV2o¼�0.007) THEN

Y¼1

IF (�0.011oV1o¼0.484 AND V24�0.009) THEN Y¼1

IF (V140.484) THEN Y¼1

IF (V1o¼�0.220 AND V24�0.011) THEN Y¼0

IF (�0.013oV1o¼0.484 AND V2o¼�0.012) THEN Y¼0

IF (�0.220oV1o¼�0.011 AND V24�0.007) THEN Y¼0

IF (�0.011oV1o¼0.484 AND �0.012oV2o¼�0.009) THEN Y¼0

LLM procedure

IF (V1o¼�0.011 AND V24�0.007) THEN Y¼0

IF (V14�0.011 AND V2o¼�0.007) THEN Y¼0

IF (V14�0.011 AND V24�0.007) THEN Y¼1

IF (V1o¼�0.011 AND V2o¼�0.007) THEN Y¼1

JRip procedure

(V14¼0.030064) and (V24¼0.00032)¼4Y¼TRUE

(V1o¼0.017804) and (V2o¼�0.002293) and

(V1o¼�0.006248)¼4Y¼TRUE

(V24¼�0.002293) and (V14¼0.001456)¼4Y¼TRUE

Otherwise Y¼FALSE

Table 2Rules derived Case 2.

DTþC4.5

IF (V1o¼�0.000 AND V2o¼�0.001) THEN Y¼1

IF (�0.001oV1o¼0.146 AND V24�0.000) THEN Y¼1

IF (V140.146 AND V24�0.001) THEN Y¼1

IF (�0.000oV1o¼0.146 AND V2o¼�0.001) THEN Y¼0

IF (V1o¼�0.001 AND V24�0.001) THEN Y¼0

IF (�0.001oV1o¼0.146 AND �0.001oV2o¼�0.000) THEN Y¼0

IF (V140.146 AND V2o¼�0.001) THEN Y¼0

LLM procedure

IF (V140.000 AND V240.000) THEN Y¼1

IF (V140.000 AND V2o¼0.000) THEN Y¼0

JRip procedure

(V14¼0.113597) and (V24¼0.00021)¼4Y¼TRUE

(V1o¼�0.002028) and (V2o¼�0.000938)¼4Y¼TRUE

(V14¼0.000661) and (V24¼0.002221)¼4Y¼TRUE

Otherwise Y¼FALSE

C.M. Rocco S., E. Hernandez / Reliability Engineering and System Safety ∎ (∎∎∎∎) ∎∎∎–∎∎∎4

Please cite this article as: Rocco S. CM, Hernandez E. Robustness and sensitivity analysis in multiple criteria decision problems usingrule learner techniques. Reliability Engineering and System Safety (2014), http://dx.doi.org/10.1016/j.ress.2014.04.022i

c) The JRip procedure extracts the following 4 rules (simplified).The error of the classifier during the training and testingphases are:

Training phase Testing phase

Accuracy¼99.97% Accuracy¼99.80%TP¼100%; FP¼0% TP¼100%; FP¼0%TN¼99.942%; FN¼0.058% TN¼99.608%; FN¼0.392%

These two cases presented show that, in general:

1) The performance of a classifier could be different.2) The number of rules derived could be different.3) As the number of samples increase, the performance of the

classifiers is also better.

For a better understanding of the set of rules derived, the usercould employ the procedure rpart.plot (implemented in R) able toproduce the graphical solution of the RPART algorithm [30](another rule generation procedure that extracts an optimizedset of rules), as shown in Fig. 2 (for Case 2, all samples). In thisprocedure, the rule set is optimized: if the constraint is satisfiedthen the output is TRUE; otherwise is FALSE.

Each path from node 1 to the bottom nodes (or leaves) could beassociated to a rule. For example, the path from node 1 to node 8(i.e., 1-2-4-8) means:

IF (V1Z�0.1 AND V2o0 AND V140) THEN Y¼FALSE, i.e.,Yo0 (note that the rule is not minimal).

The numbers shown in the bottom nodes represent the prob-ability of samples that are correctly labeled. For example, innode 9, all samples are correctly classified as TRUE while in node10, all samples are correctly classified as FALSE.

4. Case study

The case study analyzed corresponds to a real situation inportfolio management [31] faced up by policy makers, centraland developing banks, and finance and economy ministries. Theexample is related to a Venezuelan case where a set of projects(public investments) are used by decision-makers and analysts toaddress the growth and development of the economy, likerefineries, bridges, petroleum exploration, among other projects.The decision-maker and their representatives collected the data

and estimated the impacts by using their own prospective model,which includes financial, economic, and social aspects.

Projects were evaluated according to their performance onthree criteria (n¼3) previously selected by the decision unit:Benefit-Cost ratio (BCR) (is the ratio of the benefits of a projectrelative to its costs, expressed in discounted present values): ahigh value is better; Compensation of employees or Remunera-tions (REM) (an economic criterion that represents the aggregateincome that a project could produce (monetary units MMBs): ahigh value is better) and employment (EMP) (the capability to hirepeople (number of employees): a high value is better).

Table 3 lists the set of 20 projects (m¼20) under analysis. Eachcolumn represents the performance value PVi,j for each project(i¼1, .., 20) and for each attribute (j¼1, .., 3). Note that there is nodominant project and a multicriteria approach is required to rankthe projects. The vector RI, as defined in [31] is RI¼[0.66, 0.26,0.08]T. In this case, the author used a weighting method based onthe Analytical Hierarchy Process [32], an approach able to quantifythe preference of the decision maker as well as to evaluate theirconsistencies. The interested reader could refer to [33] for otherweighting methods.

4.1. Base rank

Table 4 shows the base ranking obtained using the PROMETHEEmethods (implemented on a spreadsheet) and using an Usualor Normal generalized criterion (i.e., if there is any differencebetween two projects on a specific criterion, no matter how smallit is, it will generate enough reasons to establish an order betweenthem). Project 4 is ranked in the first position followed by project16 in the second position, project 6 in the third position and so on.

4.2. Uncertainty propagation

Data from Table 3 and RI are now considered uncertain andmodeled as random variables with known probability distribution

Fig. 2. RPART graphical output for example 3.4, Case 2.

Table 4Base ranking for projects shown in Table 3, using PROMETHEE methods.

Project Base rank Project Base rank

1 10 11 72 5 12 83 14 13 44 1 14 135 19 15 206 3 16 27 6 17 98 11 18 169 15 19 18

10 17 20 12

Table 3Performance values for the projects considered (base case).

Project BCR↑ REM↑ EMP↑ Project BCR↑ REM↑ EMP↑

1 1.18474 593.02 103 11 0.63303 862.67 1072 1.85288 474.96 72 12 0.52859 1337.97 1613 0.39002 777.84 118 13 1.19821 597.13 1214 14.1446 799.80 112 14 0.35295 947.38 1345 0.21170 3859.19 582 15 0.20128 5058.33 4116 1.78824 581.04 119 16 8.76614 550.75 1157 0.54774 1144.71 158 17 0.62838 856.55 1198 0.36953 1495.81 169 18 0.29973 1088.68 1539 0.26308 2429.45 422 19 0.26252 2190.28 234

10 0.21600 3765.42 445 20 0.26673 2586.63 406

C.M. Rocco S., E. Hernandez / Reliability Engineering and System Safety ∎ (∎∎∎∎) ∎∎∎–∎∎∎ 5

Please cite this article as: Rocco S. CM, Hernandez E. Robustness and sensitivity analysis in multiple criteria decision problems usingrule learner techniques. Reliability Engineering and System Safety (2014), http://dx.doi.org/10.1016/j.ress.2014.04.022i

function. In this case: m¼20 and n¼3, then there are d¼63random variables. For example: V1¼PV11 (i.e., the BCR of project1); V20¼PV201; V24¼PV42; V36¼PV161; …; V61¼RI1; V62¼RI2 andV63¼R3.

To illustrate the proposed approach, a variation of 730% isdefined for each PV and RI. It is important to mention that thedecision maker determines such variation ranges. In general, it ispossible to define such ranges by assessing the effects that somevariables produce on a given performance value. For example, thevalues related to the Benefit-Cost ratio (BCR) depend on thepossible uncertainty associated to the discount rate.

A uniform distribution is assumed for all random variables,centered at the base case values shown in Table 3 and at RI¼[0.66,0.26, 0.08]T. Even if a constant variation is used in the example, theproposed approach could consider any range variations.

As previously mentioned, the procedure is as follows:

1) A random deviate is generated for each factor; and2) The PROMETHEE methods are used to rank the projects.

Step (1)–(2) are repeated Nsample times.To illustrate the approach, 10,000 samples are generated. As a

result, an approximation of the probability distribution functionof the rank position of each project is obtained. Table 5 shows thepossible ranks for project 4 along with their probability. Fig. 3shows the approximated probability distribution.

Several questions can be answered from Table 5:

1) What is the probability that project 4, ranked in the first positionin the base case, is still ranked in the same position whenuncertainty is considered?

2) What is the rank that could be occupied with the highestprobability?

3) What are the probabilities for the best and the worst ranking?

In this case, it is easy to detect that the most probable positionfor project 4 is the first (prob¼0.78), followed by the secondposition (prob¼0.139). Now the decision-maker knows the effect

of uncertainty factors on the rank of project 4. Note that theprobability of project 4 of being in the first position is high but itcould be ranked in lower positions but with very low probability.

4.3. Rule extraction

At this point the analyst could perform the proposed approachto determine the key factors and the conditions for a specificbehavior, by defining a constraint. Suppose that the decisionmaker is interested in knowing the conditions for project 4 forbeing in the first rank position (i.e., its ranking position in the basecase). In this case, the class associated to each sample is defined as:

IF project 4 is ranked in the first position THEN y¼1;0 otherwise.

The number of samples to be used in the rule extractionapproach is a compromise between the performances of the rulelearner used (both for training and testing) as well as the cpu timerequirements to evaluate the decision model (in this case thePROMETHEE technique). As mentioned in Section 3.4, moresamples produces better rules.

The training set is selected with the 70% of the total samplesand the rest of samples are assigned to the testing data set.However, in this case, to avoid unbalance in the sample set(i.e., the fact that there are more samples in the data set belongingto class y¼0 than y¼1 and therefore derives naïve classifiers) a setof 4734 balanced samples is selected (with approximately 50% ofeach class) from the previous 10,000 samples generated.

As previously mentioned, a higher number of samples couldderive in better classifiers. This point could pose a big constraint ifthe model F() is computational-time intensive. However, in thespecific area suggested in this example (multicriteria projectevaluations), this is not a major constraint, and additional samplescould be considered.

The LLM procedure of the Rulex software extracts 446 rules(the error of the classifier during the training phase is 0). Only 17out of 63 variables considered appear in the rule premises. Thismeans that the rest of variables have no significant effects on theselected model behavior. The ten most important variables Vi inthe set of rules (automatically determined by the Rulex software)are shown in Fig. 4. For example V24, V4 and V61, located at the toplevel, correspond to attributes REM and BCR of project 4, and RI1respectively.

The first five more important rules extracted by Rulex (i.e., withhigher covering index) are:

� IF (V4412.874 AND V244664.543 AND V56o¼122.981

AND V6140.651 AND V63o¼0.083) THEN Y¼1

� IF (V4412.874 AND V244664.543 AND V26o¼587.506

AND V6140.651 AND V63o¼0.083) THEN Y¼1

� IF (V4412.874 AND V244664.543 AND V36o¼586.048

AND V56o¼122.981 AND V6140.651) THEN Y¼1

Table 5Possible ranks for project 4.

r Prob (Rank¼r) r Prob (Rank¼r)

1 0.7812 9 0.00222 0.1344 10 0.00143 0.0381 11 0.00094 0.0159 12 0.00065 0.0093 13 0.00046 0.0069 14 0.00047 0.0049 15 0.00018 0.0033

0

0.2

0.4

0.6

0.8

1

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Prob(Rank=Position)

Position

Fig. 3. Approximated probability distribution functions for project 4.

0 0.1 0.2 0.3 0.4 0.5

V32V15V63V44V56V16V36V61V4V24

Relevance

Fig. 4. Importance of variables as determined by Rulex.

C.M. Rocco S., E. Hernandez / Reliability Engineering and System Safety ∎ (∎∎∎∎) ∎∎∎–∎∎∎6

Please cite this article as: Rocco S. CM, Hernandez E. Robustness and sensitivity analysis in multiple criteria decision problems usingrule learner techniques. Reliability Engineering and System Safety (2014), http://dx.doi.org/10.1016/j.ress.2014.04.022i

� IF (V16o¼9.085 AND V244664.543 AND V36o¼586.048

AND V46o¼121.664 AND V6140.651) THEN Y¼1

� IF (V4o¼12.874 AND V1540.199 AND V24o¼664.543

AND V364586.048 AND V61o¼0.651) THEN Y¼0

During the testing phase the following indexes are calculated:Accuracy¼87.35%.TP¼90.244%; FP¼9.756%TN¼88.947%; FN¼11.043%On the other side, the procedure JRip extracts a more compact

set of rules (14 rules) but with lower performance indexes:

Training phase Testing phase

Accuracy¼98.46%. Accuracy¼83.32%.TP¼98.291%; FP¼1.709% TP¼88.135%; FP¼11.865%TN¼98.635%; FN¼1.365% TN¼79.391%; FN¼20.609%

Finally, Fig. 5 shows the graphical output from procedure RPART(all samples). The numbers shown in the bottom nodes representthe probability of samples that are correctly labeled. Note that7 out of 9 final leaves have a percentage of samples perfectlyclassified 470%.

From a decision-making point of view, Fig. 4 (the set of importantfactor) and Fig. 5 (the RPART graphical output), provide relevantinformation to define the conditions for project 4 remaining in thefirst position. Using Fig. 5, for example, it can be observed:

� Path (1-2-4):IF REM Project 4 (V24)o721 MMBs AND REM Project 16 (V36)Z598 MMBs, THEN project 4 does not remain in the firstposition in the global ranking.Based on the high percentage of samples perfectly classifiedas false (87%), the rule could be considered as a robust rule.

In addition, it can be observed that the position of project 4 isrelated to the performance of projects 4 and 16 on REM.

� Path (1-2-5-10):IF REM Project 4 (V24)o721 MMBs AND REM Project 16 (V36)o598 MMBs AND RI of CBR (V61)o0.61 THEN project 4 doesnot remain in the first position in the global ranking.In this case, the position of project 4 is related not only to theperformance of projects 4 and 16 on REM, but also to the RI ofCBR criterion.

� Path (1-3-6-13):IF REM Project 4 (V24)4721 MMBs AND CBR of Project 4 (V4)o11 AND CBR of Project 16 (V16)4¼8.7 THEN project4 remains in the first position in the global ranking.

Again the final position of project 4 depends not only on itsown performance (REM and CBR) but also on the CBR performanceof project 16.

5. Conclusions

In this paper, Machine Learning techniques are used to assessthe robustness of a ranking in multicriteria decision problems. Aset of “If-Then” rules derived from model samples, allows describ-ing the condition of the input variables (performance of alter-natives under a defined set of criteria and/or decision-makerspreferences), given a model behavior specification, represented bya global constraint.

The approach presented is considered as an additional tool fordecision-makers interested in characterizing a decision problem.Indeed, the information provided by previous works on the effectof uncertain inputs on the final ranking of alternatives or theknowledge of the important inputs that affect ranking variations,could be now complemented by an approach (based on rulegeneration techniques) able to describe the feasible space of aset of inputs responsible of a given model behavior constraint.

Fig. 5. RPART graphical output for case study.

C.M. Rocco S., E. Hernandez / Reliability Engineering and System Safety ∎ (∎∎∎∎) ∎∎∎–∎∎∎ 7

Please cite this article as: Rocco S. CM, Hernandez E. Robustness and sensitivity analysis in multiple criteria decision problems usingrule learner techniques. Reliability Engineering and System Safety (2014), http://dx.doi.org/10.1016/j.ress.2014.04.022i

Additionally, such techniques produce the subset of variables actuallycorrelated and its importance under the specified constraints.

The case study presented (a real portfolio of 20 projectsevaluated through 3 criteria) shows that the approach is able toapproximately synthesize the ranking model responses, under aspecific constraint, through a group of simple rules that relates thevariables and their possible variation ranges. In addition, it is alsopossible to determine the set of important variables that affect theselected behavior. Although the analysis is based on the use of thePROMETHEE methods, any other multicriteria technique couldbe used.

Different rule generation techniques could be selected. In thispaper several procedures were tested, using their default settings,to only illustrate their use and capabilities. However no compar-ison among techniques was performed.

The quality of the extracted classifier (mainly assessed throughsensitivity, specificity and accuracy indexes) also depends on thecardinality of the set of samples used. For the area suggested inthis paper (i.e., multicriteria project evaluations), this is not alimitation since the set of samples are randomly generated using amodel that is not computationally intensive. However, the set ofsamples must be carefully selected to avoid unbalanced classesand therefore derive naïve classifiers. This situation could happenwhen the constraint imposed produces a feasible region with asmall hyper-volume.

In any case, the performance indexes along with the quality ofthe rules extracted must be taken into consideration before makingthe final decision.

Acknowledgment

The authors thank Prof. M. Muselli for providing a copy of theRulex software suite 2.0 and the anonymous reviewers for theircomments and suggestions.

Reference

[1] Roy B. Méthodologie Multicritère d'aide à la Décision. Paris: Económica; 1985.[2] Figueira J, Grecco S, Ehrgott M. Multicriteria analysis: state of art surveys. USA:

Springer Science; 2005.[3] Brans JP, Vincke PH. A preference ranking organization method: the PRO-

METHEE methods for multiple criteria decision making. Manag Sci 1985;31(6):647–56.

[4] Hwang CL, Yoon KS. Multiple attribute decision making: methods andapplications. Berlin: Springer; 1981.

[5] Bruggemann R, Bücherl C, Pudenz S, Steinberg C. Application of the concept ofpartial order on comparative evaluation of environmental chemicals. ActaHydrochim Hydrobiol 1999;27:170–8.

[6] Bruggemann R, Schwaiger J, Negele RD. Applying Hasse diagram technique forthe evaluation of toxicological fish tests. Chemosphere 1995;30:1767–80.

[7] Al-Sharrah G. Ranking using the copeland score: a comparison with the Hassediagram. J Chem Inf Model 2010;50(5):785–91.

[8] Rios Insua D, French S. A framework for sensitivity analysis in discrete multi-objective decision-making. Eur J Oper Res 1991;54(2):176–90.

[9] Triantaphyllou E, Sanchez A. A sensitivity analysis approach for some determi-nistic multi-criteria decision-making methods. Decis Sci 1997;28(1):151–94.

[10] Wolters WTM, Mareschal B. Novel types of sensitivity analysis for additiveMCDM methods. Eur J Oper Res 1995;81(2):281–90.

[11] Yu O-Y, GuikemaS S, Briaud JL, Burnett D. Sensitivity analysis for multi-attribute system selection problems in onshore environmentally friendlydrilling (EFD). Syst Eng 2012;15(2):153–71.

[12] Rocco CM, Tarantola S. Evaluating ranking robustness in multi-indicatoruncertain matrices: an application based on simulation and global sensitivityanalysis. In: Rainer Brüggemann Lars Carlsen, Wittmann Jochen, editors.Multi-indic Syst Model Partial Order. Springer; 2014. p. 275–92 (Chapter 13).

[13] Saltelli A, Tarantola S, Campolongo F, Ratto M. Sensitivity analysis in practice. aguide to assessing scientific models. John Wiley & Sons; 2004 (Probability andStatistics series).

[14] Saltelli A, Ratto M, Andres T, Campolongo F, Cariboni J, Gatelli D, et al. Globalsensitivity analysis. The primer. Chichester: John Wiley & Sons Ltd.; 2008.

[15] Rocco CM, Moreno JA, Carrasquero N. Robust design using a hybrid cellular-evolutionary and interval-arithmetic approach: a reliability application. ReliabEng Syst Saf 2003;79:149–59.

[16] Zimmermann M, von Hoessle JE. Computing solution spaces for robust design.Int J Numer Methods Eng 2012. http://dx.doi.org/10.1002/nme.4450.

[17] Graff L, Harbrecht H, Zimmermann M. On the computation of solution spacesin high dimensions; November 2012, Preprint-Number SPP1253-138.

[18] Hornberger GM, Spear RC. An approach to the preliminary analysis ofenvironmental systems. J Environ Manag 1981;12:7–18.

[19] Beven KJ, Binley A. The future of distributed models: model calibration anduncertainty prediction. Hydrol Processes 1992;6:279–98.

[20] Spear RC, Grieb TM, Shang N. Factor uncertainty and interaction in complexenvironmental models. Water Resour Res 1994;30:3159–69.

[21] Witten H, Frank E. Data mining: data mining: practical machine learning toolsand techniques. 3rd edition. San Francisco: Morgan Kaufmann; 2011.

[22] Rocco C, Muselli M. Empirical models based on machine learning techniquesfor determining approximate reliability expression. Reliab Eng Syst Saf2004;83(3):301–9.

[23] Cohen WW. Fast effective rule induction. In: Prieditis A, Russell S, editors.Proceedings of the 12th international conference on machine learning. MorganKaufmann; 19951-55860-377-8. p. 115–23. ⟨http://citeseer.ist.psu.edu/cohen95fast.html⟩.

[24] Frank E, Witten IH. Generating accurate rule sets without global optimization.In: Shavlik J, editor. Machine learning: proceedings of the fifteenth interna-tional conference. San Francisco, CA: Morgan Kaufmann Publishers; 1998.⟨http://www.cs.waikato.ac.nz/eibe/pubs/ML98-57.ps.gz⟩.

[25] Cangelosi D, Blengio F, Versteeg R, Eggert A, Garaventa A, Gambini C, et al.Logic learning machine creates explicit and stable rules stratifying neuro-blastoma patients. BMC Bioinform 2013;14(Suppl. 7):S12.

[26] Viloria A, Chang C, Pineda MC, Viloria J.. Estimation of susceptibility tolandslides using neural networks based on the falcon-art model. In: Proceed-ings of the IEEE 11th international conference on machine learning andapplications (ICMLA), vol.1; 2012. p. 655–60.

[27] RWeka R package; 2013. ⟨http://cran.r-project.org/web/packages/RWeka⟩.[28] Rulex software suite 2.0. [⟨http://www.impara-ai.com/⟩].[29] Pappenberger F, Iorgulescu I, Beven KJ. Sensitivity analysis based on regional

splits and regression trees (SARS-RT). Environ Model Softw 2006;21:976–90.[30] rpart R package; 2013, ⟨http://cran.r-project.org/web/packages/rpart⟩.[31] Hernandez E, Multicriteria A. Methodology to rank investment projects [M.Sc.

thesis in Operational Research]. Venezuela: Universidad Central de Venezuela;2006 (In Spanish).

[32] Saaty T. The analytic hierarchy process. New York, USA: Mc Graw Hill; 1980.[33] Nardo M. Handbook on constructing composite indicators -methodology and

user guide. Ispra: OECD; 2008.

C.M. Rocco S., E. Hernandez / Reliability Engineering and System Safety ∎ (∎∎∎∎) ∎∎∎–∎∎∎8

Please cite this article as: Rocco S. CM, Hernandez E. Robustness and sensitivity analysis in multiple criteria decision problems usingrule learner techniques. Reliability Engineering and System Safety (2014), http://dx.doi.org/10.1016/j.ress.2014.04.022i