Date post: | 28-Mar-2015 |
Category: |
Documents |
Upload: | hailey-orr |
View: | 216 times |
Download: | 0 times |
Performance DecisionsPerformance Decisions
Civil Engineering Systems Civil Engineering Systems
University of BristolUniversity of Bristol
Recognise these questions?Recognise these questions? Have we really taken into account all the Have we really taken into account all the
factors affecting our decision?factors affecting our decision? Tell me again about the assumptions?Tell me again about the assumptions? Just what are our levels of risk?Just what are our levels of risk? But where did these numbers come from?But where did these numbers come from? Have we got any evidence to prove that?Have we got any evidence to prove that? Where are we vulnerable?Where are we vulnerable? Are we really ready to make that decision Are we really ready to make that decision
yet?yet? What is the most important factor?What is the most important factor?
59%
53%
47%
44%
24%
15%
3%
0% 20% 40% 60% 80% 100%
Percentage of respondents
The quantitative analysis of direct drivers in decisions:
Specialist analysis hard to understand / interrogate / test Not enough analysis Clash of personal / departmental cultures Input numbers are shaky Assumptions not clearly identified / tested Decision factors / influences poorly / incompletely framed Uncertainty, risks and unknowns hidden or forgotten
Yet why?Yet why? Breadth depth & balance of issues not fully Breadth depth & balance of issues not fully
graspedgrasped Undue trust in quantitative analysisUndue trust in quantitative analysis Different influencers see only part of the Different influencers see only part of the
big picturebig picture Delays due to lack of consensusDelays due to lack of consensus Poor knowledge management or corporate Poor knowledge management or corporate
learninglearning Qualitative judgement and quantitative Qualitative judgement and quantitative
analysis done in isolation of each otheranalysis done in isolation of each other Lack of frameworkLack of framework
What is the PeriMeta What is the PeriMeta Approach?Approach?
A means to:A means to: Enhance decision making in the context of Enhance decision making in the context of
incomplete, sparse and conflicting informationincomplete, sparse and conflicting information Communicate complex systems simplyCommunicate complex systems simply Manage uncertainty explicitlyManage uncertainty explicitly Integrate ‘hard’ and ‘soft’ influences Integrate ‘hard’ and ‘soft’ influences
By:By: Modelling systems as hierarchies of processesModelling systems as hierarchies of processes Recording the attributes of the processesRecording the attributes of the processes Embedding a rich uncertainty calculusEmbedding a rich uncertainty calculus Using all available evidence in whatever formUsing all available evidence in whatever form
Added ValueAdded Value Shared understanding of the state of the Shared understanding of the state of the
asset asset across teams across teams up and down the organisationup and down the organisation with all stakeholders ..simplywith all stakeholders ..simply
Meta level system - health overview Meta level system - health overview Identify success,failure and vulnerabilityIdentify success,failure and vulnerability Sensitivity and value of informationSensitivity and value of information Vehicle for testing intervention strategiesVehicle for testing intervention strategies Decision recording Decision recording
corporate memory, transparency, auditabilitycorporate memory, transparency, auditability
Programme Development Programme Development
Civil Engineering SystemsBristol
Juniper
Oil Industry
Texaco
CSIRO
TMX & HLSPeriMeta
FaberMaunsell
Highways Agency
Halcrow
New Applications
Design Information Group
Bristol
CMAM Industrial Partners
EPSRC
Political risk & City vuln 3.02 & CMAM
At its heart………At its heart……… A way to communicate uncertainty A way to communicate uncertainty
and its converse dependability:and its converse dependability:
Sn(A) Sn(A) = Evidence that A is successful = Evidence that A is successful1 - Sp(A)1 - Sp(A) = Evidence that A is not successful = Evidence that A is not successfulSp(A) - Sn(A) = Lack of evidenceSp(A) - Sn(A) = Lack of evidence
Sn(A)Sn(A) Sp(A)Sp(A) 1100
Evidence that AEvidence that Ais successfulis successful UncertaintyUncertainty
Evidence that AEvidence that Ais not successfulis not successful
Interval Probability Theory Interval Probability Theory
Coin Toss:Coin Toss: 0 1
headsclassically:
‘open world’:
Life elsewhere ?:Life elsewhere ?:
Safety
Environment
Pitlochry system
M&E assets
Water control
Tail race
Turbines
***** Drum gates
*****
*******
Pitlochry system
M&E assets
Water control
Tail race
Turbines
***** Drum gates
*****
*******
Cost
Pitlochry system
M&E assets
Water control
Tail race
Turbines
***** Drum gates
*****
*******
Views of performance in selected areas of interest
Model manager
The asset - a
precious resource
Overview
Reservoir F
Dam Water control
Tail race
Towers ***** Drum gates
*****
*******
Hierarchical model of the system
Database of performance
indicators
•Inspections•Reports•Instrumentation•Analysis
0
0.2
0.4
0.6
0.8
1
1.2
0 200 400 600 800
Performance indicator
Val
ue
Values and objectivesRegulatory standards
Library of value functions
Key principles: 1/2Key principles: 1/2 Simple – only one kind of blob and link!Simple – only one kind of blob and link!
The blobs are processes with objectivesThe blobs are processes with objectives
The degree to which a process meets its The degree to which a process meets its objectives is expressed explicitlyobjectives is expressed explicitly
Its performance or dependabilityIts performance or dependability
The asset or system of interest is The asset or system of interest is described described hierarchicallyhierarchically
Each hierarchy layer represents a fairly Each hierarchy layer represents a fairly complete description of the system of complete description of the system of interestinterest
Key principles: 2/2Key principles: 2/2
Layers in the hierarchy can be related Layers in the hierarchy can be related to different levels of decision-making & to different levels of decision-making & to the contingency planning processto the contingency planning process
Evidence of performance is assembled Evidence of performance is assembled from all available sourcesfrom all available sources from expert judgement, visual from expert judgement, visual
inspection reports, instrumentation inspection reports, instrumentation model analysis etcmodel analysis etc
Best built as a challenged group Best built as a challenged group activityactivity
Generating specific viewsGenerating specific views
Multi-attribute weights can be changed Multi-attribute weights can be changed to emphasise specific points of view to emphasise specific points of view on system performance, e.g. safety on system performance, e.g. safety or economics. or economics.
If any performance indicator is If any performance indicator is irrelevant to that aspect of irrelevant to that aspect of performance is can be set a weight of performance is can be set a weight of zero.zero.
Key featuresKey features
Rapid prototyping of models for Rapid prototyping of models for exploring decision scenariosexploring decision scenarios
Comprehensive model building Comprehensive model building process for operational modelsprocess for operational models It is not just a software packageIt is not just a software package
Rapid communication, exploration Rapid communication, exploration and modification with the built modeland modification with the built model
Presidential safety
The Human Decision MakerThe Human Decision Maker
- Creative- Creative - Responsible- Responsible - Operating in an Open World- Operating in an Open World - Achieving :- Achieving :
SatisfactionSatisfactionSelf esteemSelf esteem
Therefore:Therefore: enhance rather than replaceenhance rather than replace
Evidence for a DecisionEvidence for a Decision
Generating the Generating the OptionsOptions
Assembling the Assembling the EvidenceEvidence
Making Making the Choicethe Choice
Determining the Determining the ObjectivesObjectives
Modelling Option PerformanceModelling Option Performance
Comparing Options with Comparing Options with Objectives and states of natureObjectives and states of nature
Taking Action to maximise value and mitigate riskTaking Action to maximise value and mitigate risk
Assessing Assessing Risks & ValuesRisks & Values
Mind mappingMind mapping
Monte-Monte-CarloCarlo
Fuzzy Fuzzy etcetc
Decision TreesDecision Trees AHPAHP
Evidence for a DecisionEvidence for a Decision
Generating the Generating the OptionsOptions
Assembling the Assembling the EvidenceEvidence
Making Making the Choicethe Choice
Determining the Determining the ObjectivesObjectives
Comparing Options with Comparing Options with Objectives and states of natureObjectives and states of nature
Risk Risk ManagementManagement
QRA QRA etcetc
PeriMetPeriMetaa
Assessing Assessing Risks & ValuesRisks & Values
Taking Action to maximise value and mitigate riskTaking Action to maximise value and mitigate risk
Modelling Option PerformanceModelling Option Performance
Philosophical problemsPhilosophical problems
““Seventeenth century natural scientists Seventeenth century natural scientists dreamed of uniting the ideas of dreamed of uniting the ideas of rationality, necessity and certainty into a rationality, necessity and certainty into a single mathematical package, and the single mathematical package, and the effect of that dream was to inflict on effect of that dream was to inflict on Human Reason a wound that remained Human Reason a wound that remained unhealed for three hundred years – a unhealed for three hundred years – a wound from which we are only recently wound from which we are only recently beginning to recover”beginning to recover”Stephen Toulmin 2001 “Return to Reason” Stephen Toulmin 2001 “Return to Reason”
Harvard University Press, p13Harvard University Press, p13
Culture informs process which Culture informs process which defines toolsdefines tools
Why before How
Culturee.g.
Processese.g.
Tools e.g.
WHY
PartneringBest value
Managing Value - RiskGetting AgreementsIncentivising successBuilding the teamEmpowering the supply chainContracts ProfilesProcedures Process mapsWorkshopsRich pictures
WHYHOW
HOW
Building the modelBuilding the modelModel the Model the ProcessProcess
Why before howWhy before how Assemble the evidenceAssemble the evidence
All sources – mapped to common expressionAll sources – mapped to common expression Record the Record the AttributesAttributes
Why Who What When Where How?Why Who What When Where How? Use a Rich Use a Rich Uncertainty CalculusUncertainty Calculus
Gives a powerful handle on the Gives a powerful handle on the dependabilitydependability of of each of the processeseach of the processes
The process of constructing the model The process of constructing the model encourages creative collective reflection encourages creative collective reflection on how the asset system performson how the asset system performs
Evidence from sub-Evidence from sub-processesprocesses
PeriMeta maps the conditional probabilities PeriMeta maps the conditional probabilities to linguistic variables to linguistic variables
H
E1
E2
E3
Propagation of uncertainty requires input of a Propagation of uncertainty requires input of a set of conditional probabilities set of conditional probabilities
Summary of Judgements Summary of Judgements RequiredRequired
EvidenceEvidence - for and against separated - for and against separated SufficiencySufficiency - How much of the evidence is - How much of the evidence is
directly relevant to the parent process? directly relevant to the parent process? 1 = it is sufficient on its own to fully determine 1 = it is sufficient on its own to fully determine
the success of the parentthe success of the parent
DependencyDependency - How much overlap of - How much overlap of evidence is there between the sub-evidence is there between the sub-processes?processes? Issues of bias and redundancyIssues of bias and redundancy
NecessityNecessity - Will the parent fail if the sub- - Will the parent fail if the sub-process fails?process fails? Higher necessity puts more weight on the redHigher necessity puts more weight on the red
The Essence of Risky The Essence of Risky DecisionsDecisions
Examples of UseExamples of Use Oil industryOil industry
Expert interpretation of sparse data managing Expert interpretation of sparse data managing the oilfield assetthe oilfield asset
Political riskPolitical risk Water sectorWater sector
Assessing sustainability of supplyAssessing sustainability of supply Assessing safety of contractAssessing safety of contract
Civil EngineeringCivil Engineering Flood defence decisionsFlood defence decisions Measuring ‘Egan performance’Measuring ‘Egan performance’ Highways Agency PFI MACHighways Agency PFI MAC
Assessment of Terrorist RiskAssessment of Terrorist Risk
Reservoir estimate Reservoir estimate dependabilitydependability
‘Things’ coming out of the woodwork
Field Development Success?Field Development Success?
Spend?
Value of InformationValue of InformationTesting strategiesTesting strategies
Key Drivers for the Highways Key Drivers for the Highways AgencyAgency Procurement have asked for a generic Procurement have asked for a generic
performance specification to apply to new performance specification to apply to new MAC contractsMAC contracts
The modernising government initiative The modernising government initiative (OGC/Treasury/NAO) requires outcome (OGC/Treasury/NAO) requires outcome based processes to improve efficiency and based processes to improve efficiency and effectivenesseffectiveness
SSR are responding to HA’s business needsSSR are responding to HA’s business needs Mapping of rate of progress showed that Mapping of rate of progress showed that
standard approach would be too slowstandard approach would be too slow
PropositionProposition To develop a performance regime that To develop a performance regime that
connects outcomes to what people do (i.e. connects outcomes to what people do (i.e. process) through consultation and ‘learning process) through consultation and ‘learning to improve’:to improve’: Provide a framework for high level performance Provide a framework for high level performance
specification and decision supportspecification and decision support
Motivating people by helping them to understand Motivating people by helping them to understand
where they fit and their contribution to outcomes where they fit and their contribution to outcomes
Provide confidence and competence through Provide confidence and competence through
systematic rigour and recognition of uncertaintysystematic rigour and recognition of uncertainty
Alignment Alignment WOOs
Work on Outputs and OutcomesPurpose: To match route based outputs to network level outcomes/targets.
SUNSSetting up Network StrategyPurpose: To identify business improvements and best working practices. Output: Desk instructions John Bagley (Leeds)
Network Strategy + KPIs
Purpose: Publish KPIs against which to measure performance of key Agency asset management and service delivery activities.
PRIDe Performance Measurement Group
Purpose: To develop better metrics for Area Performance Indicators to facilitate benchmarking across the network. To develop CCC compliance indicators.
Maintenance Contractors
Individual Providers have various systems for demonstrating good performance.
Performance RegimePurpose: To develop performance regime for PFMAC Output: performance specification to allow greater freedom to innovate and improve
Keith Shaw
Keith Shaw
Glynn Harrison
John Fitch
Nick Harding
Halcrow
We are not re-inventing the wheel!
OD Quality Management
(OD Process Mapping)
Keith ShawSimon Smith
Corporate Planning Team
Lisa Scott / Dick TysonPPDG - Performance Planning Development GroupHA Performance Man. – Framework for HA
Supply Chain Management
Integrated teams and continuous
improvementDavid Parker
?
?
Keith Shaw
Cabinet Office and OGC are
promoting the recognition of
uncertainty
Stakeholder DiagramStakeholder Diagram
HAMAC - Internal
Other Stakeholder
s
Road Users
Government
Other Stakeholder
s
Road Users
CommutersFreight
Many Others
Consultants
Contractors
Others
Many Others
Local Authorities Emergency
Services
Suppliers
Measurement Boundaries Measurement Boundaries
Measurement needs to be at the Contract BoundaryMeasurement needs to be at the Contract Boundary
HAMAC
HA Service Boundary
Contract Boundary
Service Delivery
Delivery Slice
Demonstrate
Detail
MAC
Measured Process
Indicators used to measure delivery of
outcomes
Measured Process
Indicators used to measure delivery of
outcomes
DDeliver
DDemonstrateProcess
Demonstrating competence
Process
Demonstrating competence
Process
Demonstrating competence
Process
Demonstrating competence
DDetail Procedures / QMSProcedures / QMS Procedures / QMSProcedures / QMS
DDev
elop
Con
tinuo
us Im
prov
emen
t ove
r tim
e
DDev
elop
Con
tinuo
us Im
prov
emen
t ove
r tim
e
Contract Boundary
Outcomes
Where are we going?Where are we going?
No right ‘answer’, so how can you be sure No right ‘answer’, so how can you be sure you have the right answer!? you have the right answer!?
We are searching for a model that:We are searching for a model that: is robust, and with which the owners are is robust, and with which the owners are
comfortable. comfortable. enables exploration of the performance enables exploration of the performance
regime by all sorts of stakeholdersregime by all sorts of stakeholders Will drive improvement in outcomesWill drive improvement in outcomes Enables alignment to paymentEnables alignment to payment
Journey to Process Model Journey to Process Model
Who? Process owners a useful guideWho? Process owners a useful guide
When ?Reflecting process cycles
Why
How
ProcessWinter ServiceOwnerNetwork ManagerPurpose- Safe mobility in winterRelevant outcome measures- Journey times- Accident rates- Material used (proxy sustainability)
Procedure/Process
Owner
Purpose
Customer Perspective
Safety
Measurement
Procedure/Process
Owner
Purpose
Customer Perspective
Safety
Measurement
Procedure/Process
Owner
Purpose
Customer Perspective
Safety
Measurement
Performance Measurement
Staff (A Good Employer))
HIGHWAYS AGENCY KEY OBJECTIVES
Attributes Attributes of of ProcessProcess
Innovation & Learning
Delivering in Partnership
Best Value (Finance)
Sustainability
NOTE: HA Balanced Scorecard
Perspectives in Blue
Customer perspective
Safety
Types of EvidenceTypes of Evidence Types of evidence considered:Types of evidence considered:
MeasuredMeasured Performance Measurement Group (PMG) Performance Measurement Group (PMG)
measuresmeasures Existing contract Area Performance Indicators Existing contract Area Performance Indicators HA High Level KPI’s (e.g. Balanced Scorecard, HA High Level KPI’s (e.g. Balanced Scorecard,
Government Objectives, etc.)Government Objectives, etc.) Instrumentation, analytical and computer modelsInstrumentation, analytical and computer models
LinguisticLinguistic Expert judgement from form reports or Expert judgement from form reports or
interviewsinterviews
Value FunctionsValue Functions Performance targets are Performance targets are
expressed as value expressed as value
functionsfunctions
Translate PI’s onto a Translate PI’s onto a
non-dimensional 0-1 non-dimensional 0-1
scalescale
0 = failure0 = failure
1 = total success1 = total success
Allows different types of Allows different types of
evidence to be brought evidence to be brought
togethertogether
Safer Travel
Reduce the number of people killed and seriously injured on
trunk roads in 2002-03 by 694 to 4297 (compared with the 1994-98
average 4991)
Mapping Value FunctionsMapping Value Functions
Performance Indicator
Val
ue
Highways Agency Highways Agency measurementmeasurement
Library of Value Functions – HA KPI’s & Targets – 2003/04Library of Value Functions – HA KPI’s & Targets – 2003/04
Maintain at least 85% of the network in good condition.
Effective Maintenance Safer Travel
Reduce the number of people killed and seriously injured on
trunk roads in 2002-03 by 694 to 4297 (compared with the 1994-98
average 4991)
Winter serviceWinter service
Worked exampleWorked example
Winter service processWinter service process
WHY
HOW
WHEN
Propagated
Outcome measures of Outcome measures of successsuccessCurrent Total number 29Current Total number 29 8 Key performance measures +8 Key performance measures +
12 Covered in quality plan 12 Covered in quality plan 9 Other measures 9 Other measures
Reduced to 3 key measuresReduced to 3 key measures Accident rateAccident rate Average journey time through systemAverage journey time through system Quantity of grit used (environmental constraint)Quantity of grit used (environmental constraint)
Plus conformance with plan Plus conformance with plan to fulfil public dutyto fulfil public duty
Benefit of outcome measuresBenefit of outcome measures Simple measures common for many processesSimple measures common for many processes Moderated to be relevant to process context Moderated to be relevant to process context
eg MOORI (Met office open road indicator). eg MOORI (Met office open road indicator). Benchmarked to achieve continuous Benchmarked to achieve continuous
improvementimprovement Empowers those doing the job to improve Empowers those doing the job to improve
performance by performance by Better measurement of their processBetter measurement of their process Continuous improvementContinuous improvement InnovatingInnovating
ConclusionsConclusions Progress is on programme Progress is on programme Strong evidence of need to improve measurementStrong evidence of need to improve measurement Key generic measures to be testedKey generic measures to be tested
Safer travel –KSI reductionSafer travel –KSI reduction Mobility – hours of congestionMobility – hours of congestion Customer Satisfaction – local surveysCustomer Satisfaction – local surveys Minimise adverse environmental impactMinimise adverse environmental impact
Robust process model to enable context based Robust process model to enable context based measurement at MAC boundarymeasurement at MAC boundary
Next stage wider engagement - validationNext stage wider engagement - validation
Overall PeriMeta ConclusionsOverall PeriMeta Conclusions Shared understanding of the state of the Shared understanding of the state of the
asset asset across teams across teams up and down the organisationup and down the organisation with all stakeholders ..simplywith all stakeholders ..simply
Performance Regime Performance Regime health & vulnerability overview health & vulnerability overview
Sensitivity and value of informationSensitivity and value of information Decision recording Decision recording
corporate memory, transparency, corporate memory, transparency, auditabilityauditability
Prediction
Uncertainty
Models
Uncertainty
Risk
Decision
Monitoring
Uncertainty
Hazards
Environment
Vulnerability
Systems
Uncertainty
Information
Surprise
Outcome
Uncertainty is a property of information – fuzziness, incompleteness and randomness
Risk is the likelihood of an uncertain event or behaviour, and its consequences for our intended purpose or objectives, set in a context that needs to be understood
Hazard - set of incubating preconditions for failure
Vulnerability – susceptibility to disproportionate damage from an event or behaviour
Surprise - an unexpected event – an unrecognised risk
TerminologyTerminology
FuzzinessFuzziness - Imprecision of - Imprecision of definition definition
Incompleteness - That which we do Incompleteness - That which we do notnot know, choose not know, choose not toto include or cannot include or cannot afford to include afford to include
RandomnessRandomness - Lack of a - Lack of a specificspecific pattern pattern
The Nature of UncertaintyThe Nature of Uncertainty
The Nature of UncertaintyThe Nature of Uncertainty
Fuzziness Incompleteness Fuzziness Incompleteness RandomnessRandomness
Tools for Uncertainty Tools for Uncertainty ManagementManagement
Analogue StudiesAnalogue Studies Case Based ReasoningCase Based Reasoning Parametric StudiesParametric Studies Safety FactorsSafety Factors Monte Carlo Simulation Monte Carlo Simulation
Bayesian ReasoningBayesian Reasoning Fuzzy MethodsFuzzy Methods Neural NetsNeural Nets
Genetic AlgorithmsGenetic Algorithms Evidential Evidential Reasoning Reasoning Process modelsProcess models
Just because...Just because...
... past futures have resembled past pasts, ... past futures have resembled past pasts,
it does not follow that future futures will it does not follow that future futures will
resemble future pasts.resemble future pasts.
Bryan Magee Bryan Magee “Popper” “Popper” 19731973
Understanding risksUnderstanding risks
Predict from
history
Understanding what we do not know
Look out
for change
Priority
actionoccurence
consequence
frequent
infrequent
highlow
VulnerabilityVulnerability
Now done in many areas manuallyNow done in many areas manually Complexity of systems means that is Complexity of systems means that is
now unreliable to depend on unaided now unreliable to depend on unaided human identificationhuman identification
Need systems approach as opposed Need systems approach as opposed to a reductionist paradigmto a reductionist paradigm
JuniperJuniper
‘‘There is a need for “generating new There is a need for “generating new processes for imagining imaginative processes for imagining imaginative outcomes” ’.outcomes” ’.
Schneider “The future of climate potential for Schneider “The future of climate potential for interaction and surprises” In Downing: Climate interaction and surprises” In Downing: Climate Change and World Food Security. Springer Berlin Change and World Food Security. Springer Berlin (1996 p79)(1996 p79)
Risk Management PlansRisk Management Plans
( KnowRisk – Australian RM software )( KnowRisk – Australian RM software )
Uncertainty measurementsLayout that gives view on Uncertainty
QRA QRA
Limited scope – applicable in tightly Limited scope – applicable in tightly constrained physical environmentsconstrained physical environments
Needs frequency databaseNeeds frequency database Makes bold assumptionsMakes bold assumptions Difficult to bring in ‘soft’ systemsDifficult to bring in ‘soft’ systems
Yet we need to mix hard and soft……..Yet we need to mix hard and soft……..
HA HA BalanceBalanced d ScorecarScorecardd
Staff Perspective How effective are we at managing, developing and motivating our workforce?
Efficiency & Finance
How good are we at managing our financial resources?
Customer Perspective
How effective are we are delivering customer requirements?
Delivering in Partnership
Do we work in synergy with our partners?
Innovation & Learning
How good are we at preparing for the future and learning from the past?
HA
Vision
Safe RoadsReliable Journeys
Informed Travellers‘Travelling with Confidence’
7 Oct 027 Oct 02