5. More on assessing performance Dr Mike Yearworth
Reader in Systems Engineering
© University of Bristol 2014
Notation
• If E is a proposi8on, an interval number is used as a probability measure such that
P(E)=[Sn(E), Sp(E)] • Where Sn(E) is the lower bound and Sp(E) the upper bound of probability of P(E) e.g. – P(E)=[0,0] – indicates belief that E is certainly false – P(E)=[1,1] – indicates belief that E is certainly true – P(E)=[0,1] – indicates belief that E is unknown
2
Interval Numbers Notation and the Italian Flag
• P(E)=[1.0,1.0]
• P(E)=[0.0,0.0] • P(E)=[0.0,1.0] • P(E)=[0.4,0.9]
3
Using ‘Hard Data’ - Mapping Value Functions
4
(Marashi & Davis 2006)
Using ‘Hard Data’ - Mapping Value Functions
5
(Marashi & Davis 2006)
Example of Mapping Likert Scales
6
(Deakin Crick, Green, Barr, Shafi & Peng 2013)
Summary of Judgements Required
• Evidence -‐ for and against separated • Sufficiency -‐ How much of the evidence is directly relevant to
the parent process? – 1 = it is sufficient on its own to fully determine the success of the
parent • Necessity -‐ Will the parent fail if the sub-‐process fails?
– 1 = it is certain the parent process will fail if the child process fails – Takes over if evidence against is large
• Dependency -‐ How much overlap of evidence is there between the sub-‐processes? – Issues of bias and redundancy
7
How this works in a simple example
8
Practical guide – heuristics
• Experience has shown that es8ma8ng necessity, sufficiency, and dependency condi8ons is hard
• In prac8ce PeriMeta is used in in a way that assumes that the number of processes per level is >2 and usually ~5±2 – Using default parameters of Necessity=0.3, Sufficiency=0.4, Dependency=0 (independent)
• Note that the point of using HPM is problem structuring with the aim of interven8on, not predic8on
9
Review PeriMeta model
• In groups… • If model is not already in PeriMeta, now is the 8me… • Review model and check necessity and sufficiency condi8ons – Default can be used if the number of dependent processes is ~5
– If <5 review the necessity and sufficiency links (check with Mike/Dan)
• Does this change anything in terms of deciding where to place focus for interven8on?
10
References Blockley, D., & Godfrey, P. S. (2000). Doing it differently : systems for rethinking construc7on.
London: Thomas Telford. Davis, J. P., MacDonald, A. D. J., & Marashi, S. E. (2007). Integrated performance measurement to
support strategic decision making in engineering organisa7ons. Paper presented at the IEEE Interna8onal Conference on Industrial Engineering and Engineering Management, Singapore.
Deakin Crick, R., Green, H., Barr, S., Shafi, A., & Peng, W-‐J. (2013). Evalua7ng the wider outcomes of schooling: the Oasis ECHO project (pp. 212). Bristol: University of Bristol.
Hall, J. W., Blockley, D. I., & Davis, J. P. (1998). Uncertain inference using interval probability theory. Interna7onal Journal of Approximate Reasoning, 19(3-‐4), 247-‐264.
Marashi, E., & Davis, J. P. (2006). An argumenta8on-‐based method for managing complex issues in design of infrastructural systems. Reliability Engineering & System Safety, 91(12), 1535-‐1545. doi: 10.1016/j.ress.2006.01.013
Marashi, S. E., Davis, J. P., & Hall, J. W. (2008). Combina8on methods and conflict handling in eviden8al theories. Interna7onal Journal of Uncertainty Fuzziness and Knowledge-‐Based Systems, 16(3), 337-‐369. doi: 10.1142/s0218488508005315
11