Portfolio Management Using Portfolio Management Using Questionable Quality DataQuestionable Quality Data

SPE 90395SPE 90395

Jim DuBoisJim DuBoisPortfolio Decisions Inc.Portfolio Decisions Inc.

September 28, 2004September 28, 2004

SPE 90395SPE 90395

Opening ThoughtsOpening Thoughts

This paper is not about improving data, but This paper is not about improving data, but about how to work with the data you have- about how to work with the data you have- today.today.Until you work with your data in a holistic Until you work with your data in a holistic manner, it is very unlikely that it will improve.manner, it is very unlikely that it will improve.Treat data quality as another unknown, like Treat data quality as another unknown, like reservoir size or Ps. Learn what can and reservoir size or Ps. Learn what can and what can’t hurt you, and to what extent.what can’t hurt you, and to what extent.May result in choosing a portfolio that is May result in choosing a portfolio that is slightly less “optimal”, but also less sensitive slightly less “optimal”, but also less sensitive to perceived data issues.to perceived data issues.

SPE 90395SPE 90395

Avoid “Answerism”Avoid “Answerism”

Many delay using advanced decision methods Many delay using advanced decision methods because “without perfect data, I won’t get the because “without perfect data, I won’t get the right answer”. right answer”. ButBut::

There is no “right”There is no “right”

There is no “answer”There is no “answer”

There are, however, better insights, which There are, however, better insights, which lead to lead to better decisionsbetter decisions..

SPE 90395SPE 90395

Better DecisionsBetter Decisions

The question facing you is not, “Can I The question facing you is not, “Can I make perfect decisions”, but can I make make perfect decisions”, but can I make better decisions than I am making now?better decisions than I am making now?

You are going to make decisions.You are going to make decisions.

Improve in cycles as you learn.Improve in cycles as you learn.

SPE 90395SPE 90395

ProcedureProcedure

Define the suspected data problem.Define the suspected data problem.

Describe the decision process.Describe the decision process.

Plan and execute the analysis.Plan and execute the analysis.

Draw conclusions and Draw conclusions and communicate.communicate.

SPE 90395SPE 90395

Step One- Define What You Don’t Step One- Define What You Don’t like About Your Datalike About Your Data

Systemic Over or Under EstimationSystemic Over or Under Estimation

Specific BiasSpecific Bias

Random ErrorRandom Error

– Competence/Training/ConsistencyCompetence/Training/Consistency

– ResourcesResources

– Emphasis/Management PressureEmphasis/Management Pressure

SPE 90395SPE 90395

Step Two- Describe How the Data Step Two- Describe How the Data is Currently Used for Decisionsis Currently Used for Decisions

Used in PM, we just don’t trust it.Used in PM, we just don’t trust it.

Used in another decision process (Hurdles, Rank Used in another decision process (Hurdles, Rank and Cut, etc.) and Cut, etc.)

Different data for approva, planning, budgeting, Different data for approva, planning, budgeting, etc.etc.

Used as a starting point, then “massaged” by Used as a starting point, then “massaged” by decision makers, intermediaries.decision makers, intermediaries.

– Massaged how? Massaged how?

Used as a smoke screen to legitimize intuitive Used as a smoke screen to legitimize intuitive decision making.decision making.

IgnoredIgnored

SPE 90395SPE 90395

Step 2 AssumptionStep 2 Assumption

For now, for today’s decision, this For now, for today’s decision, this is the best the data is going to get.is the best the data is going to get.

(Always try to improve for the next decision, (Always try to improve for the next decision,

but don’t abdicate today’s decision.)but don’t abdicate today’s decision.)

SPE 90395SPE 90395

Step Three- Plan Your Analysis Based Step Three- Plan Your Analysis Based on Answers to Two Previous Questionson Answers to Two Previous Questions

Two types of consequence from questionable Two types of consequence from questionable data:data:

– Performance- If you act on a set of data which is Performance- If you act on a set of data which is flawed, the flawed, the rangerange of possible portfolio outcomes of possible portfolio outcomes will be different (usually worse) than you expect.will be different (usually worse) than you expect.

– Lost Opportunity- Had you had “correct” data, you Lost Opportunity- Had you had “correct” data, you would have made different decisions. This is often would have made different decisions. This is often seen as a seen as a fairnessfairness issue issue..

Both types of consequence will Both types of consequence will alwaysalways exist. exist.

SPE 90395SPE 90395

Step Three- Plan Your Analysis Step Three- Plan Your Analysis Based on AnswersBased on Answers

Basic Procedure:Basic Procedure:

– Analyze honoring current data.Analyze honoring current data.

– Adjust data by increments in the direction feared. Adjust data by increments in the direction feared.

PsPs

Reserve SizeReserve Size

Capital CostCapital Cost

Timing, etc.Timing, etc.

– (May be possible to use previous corporate behavior (May be possible to use previous corporate behavior as a guide). Do not reoptimize yet. (Examples to as a guide). Do not reoptimize yet. (Examples to follow).follow).

(con’t)(con’t)

SPE 90395SPE 90395

Step Three- Plan Your Analysis Step Three- Plan Your Analysis Based on AnswersBased on Answers

Basic Procedure (continued):Basic Procedure (continued):

– Describe Describe portfolioportfolio implications on two fronts: implications on two fronts:

Metric performanceMetric performance

ValueValue

– Reoptimize at an interesting sensitivity pointReoptimize at an interesting sensitivity point

– Describe implications on three fronts:Describe implications on three fronts:

Metric performanceMetric performance

SelectionsSelections

ValueValue

(con’t)(con’t)

SPE 90395SPE 90395

Step Three- Plan Your Analysis Step Three- Plan Your Analysis Based on AnswersBased on Answers

Basic Procedure (continued):Basic Procedure (continued):

– Examine implication of new selections when Examine implication of new selections when paired with original dataset.paired with original dataset.

– Continue analysis, seeking a portfolio with Continue analysis, seeking a portfolio with acceptable performance if the data is “right”, acceptable performance if the data is “right”, but which has resilience if the data is “wrong”.but which has resilience if the data is “wrong”.

– Can combine factors, but don’t overwhelm Can combine factors, but don’t overwhelm your ability to understand what is going your ability to understand what is going on.Examine the impact of each factor on.Examine the impact of each factor separately first, then in combination.separately first, then in combination.

SPE 90395SPE 90395

Step Four- Draw Conclusions and Step Four- Draw Conclusions and Communicate ThemCommunicate Them

Did the feared data problem have the Did the feared data problem have the magnitude of impact you assumed?magnitude of impact you assumed?

In what time frame was the impact?In what time frame was the impact?

At what point did the impact become At what point did the impact become significant?significant?

If there are other decision methods being If there are other decision methods being used, use them with the same data variations. used, use them with the same data variations. Are the impacts more or less than using the Are the impacts more or less than using the preceding analysis?preceding analysis?

SPE 90395SPE 90395

Step Four- Draw Conclusions and Step Four- Draw Conclusions and Communicate ThemCommunicate Them

If appropriate, reexamine at a coarser If appropriate, reexamine at a coarser granularity.granularity.

– Impactful changes vs. incidentalImpactful changes vs. incidental

– Changes that result in major shifts of capital, jobs, Changes that result in major shifts of capital, jobs, etc., obviously require more investigationetc., obviously require more investigation

What was constant through the cases?What was constant through the cases?

At what point (amount of bias) did impactful At what point (amount of bias) did impactful changes start?changes start?

Communicate impacts in Communicate impacts in bothboth directions. directions.

SPE 90395SPE 90395

ExamplesExamples

Three Examples in the Paper:Three Examples in the Paper:

Ps Sensitivity AnalysisPs Sensitivity Analysis

Capital Sensitivity AnalysisCapital Sensitivity Analysis

Prod and Reserve Bias AnalysisProd and Reserve Bias Analysis

SPE 90395SPE 90395

ExamplesExamples

WE want to understand:WE want to understand:The effect of possible under-estimation on The effect of possible under-estimation on performance resultsperformance resultsThe effect of possible under-estimation on The effect of possible under-estimation on project selectionproject selectionThe level of under-estimation at which these The level of under-estimation at which these effects become critical effects become critical If there exists a portfolio that performs better if If there exists a portfolio that performs better if there is under-estimation, but still performs well there is under-estimation, but still performs well if there is not.if there is not.

SPE 90395SPE 90395

Base Data MetricsBase Data MetricsNPV = $18427 MNPV = $18427 M

SPE 90395SPE 90395

Base Data SelectionsBase Data Selections

SPE 90395SPE 90395

Ps –5% MetricsPs –5% MetricsNPV = $17516 MNPV = $17516 M

SPE 90395SPE 90395

Ps –10% MetricsPs –10% MetricsNPV = $16604 MNPV = $16604 M

SPE 90395SPE 90395

Ps –20% MetricsPs –20% MetricsNPV = $14782 MNPV = $14782 M

SPE 90395SPE 90395

Ps –30% MetricsPs –30% MetricsNPV = $12959 MNPV = $12959 M

SPE 90395SPE 90395

Ps –20% Metrics ReoptPs –20% Metrics Reopt

SPE 90395SPE 90395

Ps –20% Metrics (Compare)Ps –20% Metrics (Compare)NPVNPV = $14782 M= $14782 M

SPE 90395SPE 90395

Ps-20% SelectionsPs-20% Selections

SPE 90395SPE 90395

Ps –20% Selection ComparisonPs –20% Selection Comparison

SPE 90395SPE 90395

Ps –20% Selections- Original PsPs –20% Selections- Original Ps

SPE 90395SPE 90395

Ps Decrease Vs. NPVPs Decrease Vs. NPV

SPE 90395SPE 90395

Ps Sensitivity ConclusionsPs Sensitivity Conclusions

Not as sensitive to Ps as suspected, largely Not as sensitive to Ps as suspected, largely because of extra room in late years.because of extra room in late years.Largest effect is in late reserves- makes sense Largest effect is in late reserves- makes sense since this was a critical metric.since this was a critical metric.A 20% across the board reduction in exploratory A 20% across the board reduction in exploratory Ps results in Reserves violations 2007-2009, Ps results in Reserves violations 2007-2009, minor early production violations.minor early production violations.The 20% reduction case can be reoptimized to The 20% reduction case can be reoptimized to honor all the constraintshonor all the constraints

SPE 90395SPE 90395

Ps Sensitivity ConclusionsPs Sensitivity Conclusions

Reoptimized solution moves away from DW and Reoptimized solution moves away from DW and Int’l, and towards the mid-continent.Int’l, and towards the mid-continent.

If we use the original PS with the 20% reduction If we use the original PS with the 20% reduction selections, there is little loss of performance, but selections, there is little loss of performance, but we do exceed capital constraints in 2009. we do exceed capital constraints in 2009. Reserve performance is better.Reserve performance is better.

Overall, loss of Value is more severe than loss Overall, loss of Value is more severe than loss of needed performance.of needed performance.

SPE 90395SPE 90395

Some General Data Some General Data ConsiderationsConsiderations

In PM, frequency is usually more important than In PM, frequency is usually more important than amplitude. In ranking and hurdling, amplitude is amplitude. In ranking and hurdling, amplitude is more important than frequency.more important than frequency.Beware the specific case. The general case is often Beware the specific case. The general case is often more telling, and more subject to common sense QC.more telling, and more subject to common sense QC.Hurdles cause bias.Hurdles cause bias.Any decision is valid. Manipulating data to justify a Any decision is valid. Manipulating data to justify a decision already made is not.decision already made is not.A vast amount of our “experience” has to do with A vast amount of our “experience” has to do with generated data, not with actual data.generated data, not with actual data.Beware unchecked internal limits- i.e. limiting only by Beware unchecked internal limits- i.e. limiting only by internal cash flow- multiplies the effects of bias.internal cash flow- multiplies the effects of bias.

SPE 90395SPE 90395

FinallyFinally

Bottom Line:Bottom Line:

– Shift from data paralysis to Shift from data paralysis to understanding the impact of data understanding the impact of data shortcomings.shortcomings.

– Understand what does and doesn’t Understand what does and doesn’t depend upon improving the data.depend upon improving the data.

– Be an analyst, not merely a compiler.Be an analyst, not merely a compiler.