Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
1
Impact Assessment Tools:
Multi-criteria Analysis
(the Maximum Likelihood Approach)
Michaela Saisana
European Commission
Joint Research Centre
Econometrics and Applied Statistics Unit
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
2
Issues we will touch upon:
• Impact assessment studies
• Cost Benefit Analysis (+ limitations)
• MCA rooted in Social Choice Theory
• 5 methods (Relative majority, Condorcet, Borda, Successive eliminations, Median ranking)
• Weighted Sum (most common; limitations)
• Weights as importance coefficients (BA and AHP)
• MCA: Maximum likelihood approach (steps, suitability)
• Sensitivity Analysis of MCA results
• Conclusion
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
3
Impact Assessment Studies • Locating a new plant
• Human resources management
• Evaluating projects
• Selecting an investment strategy
• Electricity production planning
• Regional planning
• Evaluation of urban waste management systems
• Environmental applications
• Health Risk Prediction
• Systemic Risk Assessment ( JRC collaboration with the European Systemic Risk Board)
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
5
Basic steps of cost-benefit analysis (CBA)
1. Determine if CBA is worth doing
2. Identify objectives and policy alternatives
3. Determine stakeholders
4. Identify costs and benefits of each alternative
5. Sort into measurable and non-measurable costs and benefits
6. Estimate costs and benefits that can be measured in monetary terms
7. Conduct sensitivity analysis
8. Compare costs-benefits across alternatives
9. Adjust for non-measurable costs and benefits(?)
10. Make a decision
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
6
Cost-benefit guidelines
• UK Department of the Treasury, Appraisal and Evaluation in Central Government (The Green
Book), London:2002, http://www.hmtreasury.gov.uk/data_greenbook_index.htm
• NZ Treasury guidelines
www.treasury.govt.nz/publications/guidance/planning/costbenefitanalysis>
• Australian Government, Office of Best Practice Regulation,
http://www.finance.gov.au/obpr/cost-benefit-analysis.html (see especially Handbook of Cost-
Benefit Analysis, and Best Practice Regulation Handbook)
• Queensland Government, Department of Infrastructure and Planning, Cost Benefit Analysis,
www.dip.qld.gov.au/resources/guideline/project-assurance-framework/pafcost-benefit-
analysis.pdf
• Government of Western Australia, Department of Treasury and Finance, 2005, Project
Evaluation Guidelines,
www.dtf.wa.gov.au/cms/uploadedFiles/project_evaluation_guidelines_2002.pdf
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
7
Limitations of CBA
• Results often highly sensitive to specific assumptions, such as
discount rate
• Difficult to balance non-quantifiable costs/benefits against
quantifiable ones
• Anthropocentric in its underlying social vision
How much is life, education (literacy), welfare, health, ecological
sustainability, employment (business confidence) worthy?
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
8
Multi Criteria Analysis (MCA) - Definition
“Multi Criteria Analysis is a decision-making tool, developed for
complex multi-criteria problems that include quantitative and/or
qualitative aspects of the problem in the decision making process.”
(Center for International Forestry Research, CIFOR, 1999)
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
9
MCA - Steps
1. Establish the decision context
2. Identify the criteria and the options
3. Describe/rate the performance of each option against the criteria
4. Assign weights across criteria
5. Combine the information to obtain a ranking of the options
6. Examine the results and review
7. Conduct sensitivity analysis
8. Final decisions
A story next!
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
10
Story: particularly insidious form of malaria that devastated Egypt 1942
Cause identified: mosquito
Yet, the real causes were previous expert led interventions:
engineering of railways, irrigation of canals,…
Timothy Mitchell, 2002
Animation by University of California, Berkeley: https://www.youtube.com/watch?v=8jqEj8XUPlk
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
11
Tania Murray Li, 2007
These boxes hide:
•Context,
•History (problems are
treated as snapshots)
•Politics (disregard
questions of power and
inequality)
Experts draw themselves
outside the picture
Animation by University of California, Berkeley: https://www.youtube.com/watch?v=8jqEj8XUPlk
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
12
MCA - Performance matrix
Criterion 1
(/20)
Criterion 2
(rating)
Criterion 3
(qual.)
Criterion 4
(Y/N) …
Action 1 20 135 G Yes …
Action 2 9 156 B Yes …
Action 3 15 129 VG No …
Action 4 9 146 VB No …
Action 5 7 121 G Yes …
… … … … … …
Criteria should not be dependant on each other and not
redundant (to avoid double counting)
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
13
MCA - Performance matrix
• Who decides the ratings?
MCA very flexible wrt who gets a say in either the criteria or rating the
options:
Democratic decision-making - all members of the decision-making body, or each
organizational branch/unit, independently allowed to rate options
Panel of experts asked to make judgments; can use different panel to judge different
criteria
Consensus model - decision-making body ‘thrash it out’
Stakeholder inclusion
Different groups can rate options on different criteria
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
14
MCA - Result
The outcome of MCA can be used to:
•Identify a single, most-preferred option
•Rank options
•Short-list a limited number of options for subsequent detailed
appraisal through other methods such as CBA
•Distinguish acceptable from unacceptable options
•Combine different options based on relative strengths
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
15
Social Choice Theory Problem:
• A group of voters have to select a candidate among a group of
candidates (election)
• Each voter has a personal ranking of the candidates according to
his/her preferences
•Which candidate must be elected?
What is the «best» voting procedure?
Analogy with multi-criteria analysis:
• Candidates actions
• Voters criteria
Best interest of society
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
16
Social Choice Theory
Social choice theory methods would be ideally suited for assessing
multiple options through multiple criteria … and were already available
between the end of the XIII and the XV century, …
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
17
1. Ramon Llull (ca. 1232 – ca. 1315) proposed first what would then become known as the
method of Condorcet.
2. Nicolas de Condorcet, (1743 –1794) His „Sketch for a Historical Picture of the Progress of the
Human Spirit (1795)‟ can be considered as an ideological foundation for evidence based policy
(modernity at its best!).
3. Nicholas of Kues (1401 – 1464), also referred to as Nicolaus Cusanus and Nicholas of Cusa
developed what would later be known as the method of Borda.
4. Jean-Charles, chevalier de Borda (1733 – 1799) developed the Borda count.
1 2 3 4
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
18
Five methods (among many others)
1. Relative majority
2. Condorcet
3. Borda
4. Successive eliminations
5. Median ranking
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
19
3 candidates: Adam, Brian, Carlos
11 voters
10 voters
9 voters
A B C
B C B
C A A
A 11
B 10
C 9
Adam is elected
30 voters:
Method 1 : Relative majority
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
20
3 candidates: Adam, Brian, Carlos
11 voters
10 voters
9 voters
A B C
B C B
C A A
A 11
B 10
C 9
Adam is elected
30 voters:
Method 1 : Relative majority
Problem: B and C preferred to
A by a majority of voters!
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
21
11 voters
10 voters
9 voters
A B C
B C B
C A A Brian is elected
B preferred to A 19
votes
B preferred to C 21
votes
C preferred to A 19
votes
3 candidates: Adam, Brian, Carlos
30 voters:
Method 2 : Condorcet
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
22
4 voters
3 voters
2 voters
A B C
B C A
C A B
A preferred to B 6
votes
B preferred to C 7
votes
C preferred to A 5
votes
Method 2 : Condorcet
3 candidates: Adam, Brian, Carlos
9 voters: Problem: Nobody is elected!
(cycle)
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
23 39 x 2 + 31 x 1
Points
2
1
0
30 voters
29 voters
10 voters
10 voters
1 voter
1 voter
A C C B A B
C A B A B C
B B A C C A
Scores
A 101
B 33
C 109
31 x 2 + 39 x 1
11 x 2 + 11 x 1
Carlos is elected!
Method 3 : Borda
3 candidates: Adam, Brian, Carlos
81 voters:
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
24
3 voters
2 voters
2 voters
C B A
B A D
A D C
D C B
Points
3
2
1
0
Scores
A 13
B 12
C 11
D 6
Ranking
A
B
C
D
Adam is elected
Method 3 : Borda
4 candidates: Adam, Brian, Carlos, David
7 voters:
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
25
3 voters
2 voters
2 voters
C B A
B A C
A C B
Points
2
1
0
Scores
A 6
B 7
C 8
Ranking
C
B
A
Carlos is elected
Method 3 : Borda
4 candidates: Adam, Brian, Carlos, David
7 voters:
Problem: Fully Dependant on
irrelevant alternatives (easy to
manipulate)
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
26
Method 4 : Successive eliminations
6 voters
4 voters
1 voters
A C C
C A B
B B A
3 candidates: Adam, Brian, Carlos
11 voters:
Ranking
A
C
B
A tour-wise procedure, whereby
the worst candidate (most voted
in the last position) is eliminated
progressively until one is left.
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
27
Method 5 : Median ranking
6 voters
4 voters
1 voters
A C C
C A B
B B A
3 candidates: Adam, Brian, Carlos
11 voters:
A: 11111122223
B:23333333333
C:11111222222
•Ranking of candidates for each
voter
•Median rank for each candidate
across voters
Ranking
A
C
B
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
28
5 candidates: Adam, Brian, Carlos, David, Edison
8 voters
7 voters
4 voters
4 voters
2 voters
A B E D C
C D C E E
D C D B D
B E B C B
E A A A A
25 voters: Relative majority Adam elected
Condorcet: Carlos elected
Borda:
David elected
Successive eliminations:
Edison elected
Median ranking:
Carlos elected
?
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
30
Kenneth Arrow
(Nobel prize in economy, 1972)
Impossibility theorem (1952):
With at least 2 voters and 3 candidates, it is impossible to build a voting procedure that simultaneously satisfies the 5 following properties:
•Non-dictatorship •Universality • Independence with respect to third parties •Monotonicity •Non-imposition
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
31
Most common approach: Weighted sum
Problems: 1) Fully compensatory (elimination of conflicts)
weights
X1
(50%)
X2
(50%)
Y
a 90 10 50
b 10 90 50
c 50 50 50
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
32
Most common approach: Weighted sum
Problems: 2) Does not encourage improvement in the weak dimensions
weights
X1
(50%)
X2
(50%)
Y
a 100 10 55
b 20 90 55
c 50 50 50
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
33
Most common approach: Weighted sum
Problems: 3) Weights are used as if they were importance coefficients
while they are trade off coefficients
Y = 0.5 ×X1+ 0.5 ×X2
R12 = 0.08, R2
2 = 0.83, corr(X1, X2) =−0.151, V(x1) = 116, V(x2) = 614, V(y) = 162
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
34
Most common approach: Weighted sum
• Weighted sum approach only possible under special circumstances (e.g.
standardized variables, uniform covariance matrix…)
• Hence we need to move away from weighted sums …
Effective weights are compared with nominal weights to
ensure coherence between the two.
[Paolo Paruolo, Michaela Saisana, Andrea Saltelli, 2013, Ratings and
rankings: Voodoo or Science?, J. R. Statist. Soc. A, 176 (3), 609-634]
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
35
MCA: Maximum likelihood Approach Features (1/2):
• no need for outlier treatment;
• no need for data normalisation;
• no need for uniform covariance matrix;
• no need to attach monetary value to indicator scores;
• no need for data aggregation;
[Kemeny (1959), Young and Levenglick (1978)] Led to: Condorcet-Kemeny-Young-Levenglick (C-K-Y-L) ranking procedure
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
36
MCA: Maximum likelihood Approach Features (2/2):
• works with both continuous and categorical variables;
• weights attached to indicators are indeed importance coefficients;
• a compromise between conflicting opinions;
• reasonably resistant to manipulation;
• produces a ranking that is statistically optimal (anonymous, neutral, Pareto optimal, satisfies reinforcement
and local independence of irrelevant alternatives)
[Kemeny (1959), Young and Levenglick (1978)] Led to: Condorcet-Kemeny-Young-Levenglick (C-K-Y-L) ranking procedure
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
37
MCA - Performance matrix
Criterion 1
(20%)
Criterion 2
(30%)
Criterion 3
(20%)
Criterion 4
(30%)
Action 1 20 135 G Yes
Action 2 9 156 B Yes
Action 3 15 129 VG No
Action 4 9 146 VB No
Action 5 7 121 G Yes
• Criteria should not be dependant on each other and not redundant (to
avoid double counting)
Where do
weights come
from?
(…next couple
of slides)
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
38
43
32
25
29
38
33
38
34
28
39
26
36
39
32
2931
33
36 35
30
34
38
33
29
33 33 33 3335
32
0
10
20
30
40
50
60
70
Co
nsu
mp
tion
Acc
ess
Sta
bili
ty
Nu
triti
on
Qu
alit
y
Qu
alit
y
Ava
ilab
ility
Acc
ess
Sta
tus
Acc
ess
Qu
alit
y
To
ilet
Fa
cilit
ies
Wa
ste
Ma
na
ge
me
nt
Pra
ctic
es
Qu
alit
y
Fa
cilit
ies
En
erg
y
Qu
alit
y
Ava
ilab
ility
Acc
ess
Te
nu
re
Qu
alit
y
Inp
uts
Ski
lls
Se
rvic
es
Ass
ets
Exp
osu
re
Co
pin
g a
bili
ty
Re
cove
ry a
bili
ty
Fo
od
Ed
uca
tion
He
alth
care
Education Farm
Assets
Exposure &
Resilience to
Shocks
Gender
Equality
In 4 dimensions of poverty, the average expert
weight is similar to equal weighting Tiredness
in filling in the questionnaire on weights??
Weights based on Budget
Allocation (42 experts)
[Cohen, Saisana, 2014]
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
39
PAIRWISE COMPARISONS;
RELATIVE IMPORTANCE OF ONE CRITERION
OVER ANOTHER 1-9 scale
1 EQUAL 3 MODERATE 5 STRONG 7 VERY STRONG 9 EXTREME
1 2 3 4 5 6 7 8 9
Patents vs. x Royalties x
x Patents vs. Internet x
x Patents vs. Technology exports x
x Patents vs. Telephones x
x Patents vs. Electricity x
Patents vs. x Schooling years x
Patents vs. x University Students x
x Royalties vs. Internet x
Royalties vs. x Technology exports x
x Royalties vs. Telephones x
x Royalties vs. Electricity x
Royalties vs. x Schooling years x
Royalties vs. x University Students x
Internet vs. x Technology exports x
x Internet vs. Telephones x
x Internet vs. Electricity x
Internet vs. x Schooling years x
Internet vs. x University Students x
x Technology exports vs. Telephones x
x Technology exports vs. Electricity x
Technology exports vs. x Schooling years x
Technology exports vs. x University Students x
x Telephones vs. Electricity x
Telephones vs. x Schooling years x
Telephones vs. x University Students x
Electricity vs. x Schooling years x
Electricity vs. x University Students x
x Schooling years vs. University Students x
Which Indicator Do You Feel Is More Important? To What Degree?
Questionnaire
Patents Royalties Internet Tech.Exports Telephones Electricity Schooling University St.
Patents 1 1/3 5 4 3 9 1/6 1/8
Royalties 3 1 3 1/4 5 9 1/3 1/4
Internet 1/5 1/3 1 1/6 2 2 1/7 1/6
Tech.Exports 1/4 4 6 1 5 9 1/4 1/5
Telephones 1/3 1/5 1/2 1/5 1 7 1/9 1/9
Electricity 1/9 1/9 1/2 1/9 1/7 1 1/9 1/9
Schooling 6 3 7 4 9 9 1 2
University St. 8 4 6 5 9 9 1/2 1
solve for the
Eigenvector
Patents 0.109
Royalties 0.103
Internet hosts 0.029
Tech exports 0.117
Telephones 0.030
Electricity 0.014
Schooling 0.301
University st. 0.297
Weights
Inconsistency
17.4 %
Weights based on Analytic Hierarchy Process
[Saisana, Saltelli, 2008]
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
40
USING PAIRWISE COMPARISONS, THE
RELATIVE IMPORTANCE
OF ONE CRITERION OVER ANOTHER CAN BE
EXPRESSED
1 EQUAL 3 MODERATE 5 STRONG 7 VERY STRONG 9 EXTREME
Patents Royalties Internet Tech.Exports Telephones Electricity Schooling University St.
Patents 1 1/3 5 4 3 9 1/6 1/8
Royalties 3 1 3 1/4 5 9 1/3 1/4
Internet 1/5 1/3 1 1/6 2 2 1/7 1/6
Tech.Exports 1/4 4 6 1 5 9 1/4 1/5
Telephones 1/3 1/5 1/2 1/5 1 7 1/9 1/9
Electricity 1/9 1/9 1/2 1/9 1/7 1 1/9 1/9
Schooling 6 3 7 4 9 9 1 2
University St. 8 4 6 5 9 9 1/2 1
Weights based on Analytic Hierarchy Process
P=5I
R=3I
We expect:
P > R
Expert said:
R > P (R=3P)
Inconsistency
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
41
Performance
matrix
Criterion
1
Criterion
2
Criterion
3
Criterion
4
Criterion
5
Weights 10% 20% 10% 30% 30%
Option A 50 0.6 400 0.6 4000
Option B 70 0.3 500 0.7 5000
Option C 90 0.4 600 0.4 3000
Step 1 - Input matrix to the multicriteria analysis
Example: Three options need to be ranked according to five criteria. The importance
of the criteria is reflected in the respective weights.
MCA: Maximum likelihood Approach
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
42
Step 2 – Options are compared pairwise
For each comparison, e.g. option A versus option B, all the weights corresponding to the criteria
that favour A versus B are added up (abbreviated as AB). In this case AB gets the weight of
Criterion 2 only (=0.2). The comparison BA gets the sum of the weights of the remaining
criteria: 1, 3, 4, 5 (=0.8). For n options, there are n (n-1) comparisons to be made. All the values
from the pairwise comparisons are entered in a so called outranking matrix.
Outranking
matrix Option A
Option B
Option C
Option A 0 0.2 0.8
Option B 0.8 0 0.6
Option C 0.2 0.4 0
MCA: Maximum likelihood Approach
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
43
Step 3 – Calculate support for all permutations and select the maximum
•All 3! (=6) permutations of the options are considered and the support score for each ranking is calculated.
•ABC has a support of 1.6 (=0.2+0.8+0.6), which is the sum of elements above the diagonal in the
outranking matrix.
•Support scores for all six rankings: ABC= 1.6 |ACB=1.4 | BAC=2.2 | BCA=1.6 | CAB=0.8 | CBA=1.4
•The ranking selected is the one with the maximum likelihood score: BAC
Outranking
matrix Option A
Option B
Option C
Option A 0 0.2 0.8
Option B 0.8 0 0.6
Option C 0.2 0.4 0
MCA: Maximum likelihood Approach
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
44
How to shake coupled stairs How coupled stairs are shaken in most of
available literature
MCA: Maximum likelihood Approach
Important to assess sensitivity of results to the weights
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
45
Frequency matrix – Sensitivity of the final ranking to the assumptions (e.g.
weights)
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
46
• The main limitation of this method is the difficulty in computing the ranking when
the number of options grows (e.g. 50).
• For 10 options 10 = 3,628,800 permutations …still trivial for today’s PCs
• To solve this NP-hard problem when the number of options is very large there are
plenty of numerical algorithms (JRC works on them!)
MCA: Maximum Likelihood Approach
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
47
Concluding: How to use MCA in your work
1. Decide on the criteria that you want to use in your assessment;
2. Identify appropriate indicators for each of the criteria (more than one indicator for
each criteria is OK);
3. Score the options on each criterion based on their performance on that criterion;
4. Determine the weights of all the criteria (use for instance AHP);
5. Calculate the overall ranking of the alternatives (e.g., using Maximum Likelihood);
6. Examine the results: explain why some options turn out to be better than others;
7. Do a Sensitivity Analysis: what happens to the results if you change the weights of
the criteria?
8. Make a final decision
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
48
“
[Peyton Young, 1995, Optimal Voting Rules,
Journal of Economic Perspectives 9:51-64]
Peyton Young Professor Emeritus, Research
Professor in Economics,
Johns Hopkins University
The more important issue is whether the (maximum
likelihood) method is intuitively easy to grasp, and
whether it improves on methods currently in use. On both
these counts I think that the answer is affirmative, and
I predict that the time will come when it is considered
a standard tool for political and group decision
making.
Michaela Saisana
5th Impact Assessment Course
JRC & SecGen, Brussels, 20-21/01/2015
49
http://composite-indicators.jrc.ec.europa.eu
Further reading
MCA •Balinski, M. and R. Laraki (2010). Majority Judgment. Measuring, ranking and electing. MIT Press.
•Balinski, M. and R. Laraki (2014). Judge: Don’t vote!. Operations Research, 62, 483-511.
•Dodgson, J. S., Spackman, M., Pearman, A., and Phillips, L. D. (2009). Multi-criteria analysis: a manual. Department
for Communities and Local Government: London.
•Keeney, R. L. and Raiffa, H. (1976). Decisions with multiple objectives: preferences and value tradeoffs. Wiley, New
York. Reprinted, Cambridge Univ. Press, New York (1993).
•Munda G. (2007), Social multi-criteria evaluation, Springer-Verlag, Heidelberg, New York, Economics Series.
•Young, H. P. (1995). Optimal voting rules. Journal of Economic Perspectives, 9, 51-64.
Weights through expert opinion •Saisana, A. Saltelli, S. (2008) Expert Panel Opinion and Global Sensitivity Analysis for Composite Indicators, Lecture
Notes in Computational Science and Engineering 62, 251-275.
•Cohen, A., Saisana, M., 2014, Quantifying the qualitative: Eliciting expert input to develop the Multidimensional
Poverty Assessment Tool, J of Dev. Studies, 2014, 50(1)).