Post on 14-Jun-2015
description
transcript
Expert Mining for Evaluating Risk Indicators Scenarios
Oscar Franco-Bedoya, Dolors Costal, Soraya Hidalgo, Ron Ben-Jacob
Monday 21st July 2014
2
Background
Workshop procedure
Generalization of the approach
Applications and lessons
learned
Related work
Conclusions and further
work
Outline
3
Bac
kgro
un
d
4
Bac
kgro
un
d
5
Bac
kgro
un
d
Managing risk in
open source adoption
6
Identification
Mitigation
methods
Management
Advanced tools
Provides
Platform
Methods
OSS
adoption
projects To supporta
In
Uses
Ecosystem
modelingStatistical
tools
Risk
Management
i.e. i.e.i.e.
Bayesian
Networks
Social
network
analysis
Expert
scenarios
assessment
e.g. e.g. e.g.
Bac
kgro
un
d
IdentificationRisks
Management
Systematic
protocoluses
7
Bac
kgro
un
d
Project Site
Code Version Repository
Bug Tracker IRC Mailing List
Ecosystem hubs
Project indexes
Social Media
Twitter Facebook
I
Raw
DataSNA
Measures
Risk Indicators:• Project
• Community
• ContextualII
IndicatorsScenario-based
Assessment DomainExpert
Business Analysis• Business goals
III
Business
Goals
3-Layered
RISCOSS approach
Number of
downloads
Number of
event
referencesCentrality
Number of
open bugs
8
Ou
tlin
e
Background
Workshop procedure
Generalization of the approach
Applications and lessons
learned
Related work
Conclusions and further
work
Outline
9
The RISCOSS tactical workshop is designed to permit
experts to assess risk indicators
Wo
rksh
op
pro
ced
ure
10
Tactical workshop
protocol
Wo
rksh
op
pro
ced
ure
Pre-Tasks
Part II: Community dinamics
Part I : Community data
TacticalWorkshop Organizer
DomainExpert
6.Make presentation of RISCOSS
project summary.
7.Explain the RISCOSS analytics platform.
8.Explain the tactical
workshop Part I and
Part II
9.Explain the risk driver
selection WS Excel File
Excel File: risk driver selection WS
TabsTimelinessActivenessCommunity
10.Study the use case scenario
11.Assess the use case scenario
overall state
12.Determine the level of
the risk indicator
13.Study the use case scenario
14.Assess the use case scenario
overall state
15.Determine the level of
the community
risk indicator
PDF File:KPA RISCOSS Analytics
[more risk drivers]
[not more risk drivers]
RISCOSS Analytics
Team
1.Determine drivers and
risk indicators
2.Construct Bayesian networks
3.Define scenarios
4.Identify and contact experts
5.Workshops planning and preparation
16.Send the scenarios
judgement
Populate BN nodes
11
Pre-Tasks
TacticalWorkshop Organizer
RISCOSS Analytics
Team
1.Determine drivers and
risk indicators
2.Construct Bayesian networks
3.Define scenarios
4.Identify and contact experts
5.Workshops planning and preparation
Correspond to
tasks that must
be done before
conducting the
tactical workshop
Wo
rksh
op
pro
ced
ure
12
Pre-Tasks
TacticalWorkshop Organizer
RISCOSS Analytics
Team
1.Determine drivers and
risk indicators
2.Construct Bayesian networks
3.Define scenarios
4.Identify and contact experts
5.Workshops planning and preparation
Wo
rksh
op
pro
ced
ure
Risk Driver
Forum posts per day
Forum messages per thread
Mail per day
Overall community size
Number of developers involved
Number of testers (individuals
providing feedback)
Number of companies using the
software
Companies supporting the project
(adding to code)
Risk Indicator Activeness
13
Pre-Tasks
TacticalWorkshop Organizer
RISCOSS Analytics
Team
1.Determine drivers and
risk indicators
2.Construct Bayesian networks
3.Define scenarios
4.Identify and contact experts
5.Workshops planning and preparation
Wo
rksh
op
pro
ced
ure
14
Pre-Tasks
TacticalWorkshop Organizer
RISCOSS Analytics
Team
1.Determine drivers and
risk indicators
2.Construct Bayesian networks
3.Define scenarios
4.Identify and contact experts
5.Workshops planning and preparation
Wo
rksh
op
pro
ced
ure
15
Pre-Tasks
TacticalWorkshop Organizer
RISCOSS Analytics
Team
1.Determine drivers and
risk indicators
2.Construct Bayesian networks
3.Define scenarios
4.Identify and contact experts
5.Workshops planning and preparation
Wo
rksh
op
pro
ced
ure
16
The tactical
workshops begin
with an exposition
about the main
topics that will be
covered
Wo
rksh
op
pro
ced
ure
6.Make presentation of RISCOSS
project summary.
7.Explain the RISCOSS analytics platform.
8.Explain the tactical
workshop Part I and
Part II
PDF File:KPA RISCOSS Analytics
TacticalWorkshop Organizer
17
Wo
rksh
op
pro
ced
ure
9.Explain the risk driver
selection WS Excel File
Excel File: risk driver selection WS
TabsTimelinessActivenessCommunity
TacticalWorkshop Organizer
18
Wo
rksh
op
pro
ced
ure
Part I : Community data
DomainExpert
10.Study the use case scenario
11.Assess the use case scenario
overall state
12.Determine the level of
the risk indicator
19
Wo
rksh
op
pro
ced
ure
Part II: Community dinamics
13.Study the use case scenario
14.Assess the use case scenario
overall state
15.Determine the level of
the community
risk indicator
16.Send the scenarios
judgement
DomainExpert
20
Ou
tlin
e
Outline
Background
Workshop procedure
Generalization of the approach
Applications and lessons
learned
Related work
Conclusions and further
work
21
Ge
ne
raliz
atio
n o
f th
e a
pp
roac
h
22
Ou
tlin
e
Outline
Background
Workshop procedure
Generalization of the approach
Applications and lessons
learned
Related work
Conclusions and further
work
23
Applications and lessons learnedA
pp
licat
ion
s an
d le
sso
ns
lear
ne
d
We have conducted 10 technical workshops In private and official organizations and academic institutions.
The experts were from different countries.
France, Israel, Italy, Spain, and Netherlands
There are some inconsistencies in the scenarios The scenarios were designed using random number generators
While the domain experts are conducting the tactical workshops, The degree of "calibration" of their judgement improves
24
Ou
tlin
e
Outline
Background
Workshop procedure
Generalization of the approach
Applications and lessons
learned
Related work
Conclusions and further
work
25
Re
late
d W
ork
Related Work
Delphi method
(QUELCE)
Quantifying Uncertainty
in Early Cost
Estimation[2]
Reliable consensus of
opinion of a group of
experts [1]
[1]N. Dalkey and O. Helmer, “An experimental application of the Delphi method to the use of experts,”.
[2] R. W. Ferguson, D. Goldenson, J. M. McCurley, R. W. Stoddard, and D. Zubrow, “Quantifying
Uncertainty in Early Lifecycle Cost Estimation ( QUELCE ),”
26
Ou
tlin
e
Outline
Background
Workshop procedure
Generalization of the approach
Applications and lessons
learned
Related work
Conclusions and further
work
27
DomainExpert
Conclusions and further workC
on
clu
sio
ns
and
fu
rth
er
wo
rk Pre-Tasks
Part II: Community dinamics
Part I : Community data
TacticalWorkshop Organizer
DomainExpert
6.Make presentation of RISCOSS
project summary.
7.Explain the RISCOSS analytics platform.
8.Explain the tactical
workshop Part I and
Part II
9.Explain the risk driver
selection WS Excel File
Excel File: risk driver selection WS
TabsTimelinessActivenessCommunity
10.Study the use case scenario
11.Assess the use case scenario
overall state
12.Determine the level of
the risk indicator
13.Study the use case scenario
14.Assess the use case scenario
overall state
15.Determine the level of
the community
risk indicator
PDF File:KPA RISCOSS Analytics
[more risk drivers]
[not more risk drivers]
RISCOSS Analytics
Team
1.Determine drivers and
risk indicators
2.Construct Bayesian networks
3.Define scenarios
4.Identify and contact experts
5.Workshops planning and preparation
16.Send the scenarios
judgement
Populate BN nodes
DomainExpert SNA
Project Site
Code Version Repository
Bug Tracker IRC Mailing List
Step-by-step
protocol
Data used to
construct
Bayesian
networks
Future work
Combines opinion
of domain experts
with OSS raw data
Empirical
application
&
lessons
28
Ou
tlin
e
Outline
Background
Workshop procedure
Generalization of the approach
Applications and lessons
learned
Related work
Conclusions and further
work
29
SEe
D4
FI
Thanks for your attention
Comments and Questions
31
SEe
D4
FI