Best Available Copy
ANEALAIN FCMPNN
ID I
DEPENDMENCE ON COT-EARIS ANALYSI
TIHUIERSIS Y
Robrigt- DevaoneAi Philip T.se PhopiFirs Lieuteantla USAn CaptamUA
for TGSM/SQ/85-2
..........................................16TI1C
AFIT/GSM/LSQ/85
V
AN EVALUATION OF COMPONENT f
DEPENDENCE IN COST-RISK ANALYSIS
THESIS
Robert E. Devaney Philip T. PopovichFirst Lieutenant, USAF Captain, USAF
L ~AFIT/GSM/LSQ/85S-27
Li~lu
Approved for public release; distribution unlimited
Best Available Copy1_ , • - 'w j" - • , '. , 1 "' a • - , "• , " ' -I - . • - - ,• . .' ' . • , . . . . o. d , " . ' - . : - . •r, 1
The contents of the document are technically accurate, andno sensitive items, detrimental ideas, or deleteriousinformation are contained therein. Furthermore, the viewsexpressed in the document are those of the author(s) and donot necessarily reflect the views of the School of Systemsand Logistics, the Air University, the United States AirForce, or the Department of :efense.
Accession ForNTIS GRA&IDTIC TABUnannounced ElJustification
ByDistribution/
Availability Codes
Avail and/orDist Special
Best Available Copy
Vi r ~ ~ u r I~ IU t N M~ . ' I I
V,
DISCLAIMER NOTICE
x THIS DOCUMENT IS BEST QUALITYPRACTICABLE. THE COPY FURNISHEDTO DTIC CONTAINED A SIGNIFICANTNUMBER OF PAGES WHICH DO NOTREPRODUCE' LEGIBLY.
'4.7
AFIT/GSM/LSQ/85S-27
n
,i'0 AN EVALUATION OF COMPONENT
DEPENDENCE IN COST-RISK ASSESSMENT
THESIS
Presented to the Faculty of the School of Systems and Logistics
of the Air Force Institute of Technology
Air University
"In Partial Fulfillment of the
Requirements for the Degree of
Master of Science in Systems Management
~lJ
Robert E. Devaney, B.S. Philip T. Popovich, M.B.A.
First Lieutenant, USAF Captain, USAF
September 1985
Approvcd fOr pubI ic relcase; distr 1butino uwi ,i w .tod
,. ,',
Preface
The purpose of this research effort was o ,levelop a
cost-risk assessment method that incorporated the effects' of
cost dependence among system components. Current cost-risk
assessment techniques make-limiting assumption.s of component
cost independence or total component cost dependence,
neither of which we felt truly represented system
relationships.
The method we developed and tested provides insight
into the modelling of component cost dependency and its
application to cost-risk assessment. Further research in
this area will improve the ability of the Department of
Defense to estimate cost-risk and to limit weapon system
cost growth.
During the course of this research effort we have had a
great deal of help from a number of people. We are grateful
for the direction given to us by our tl esis advisor, Mr Rich
Murphy. His ideas guided us through the entire project. We
wish to thank Col Deep of the Business Management Research
Center and his staff for their assistance and cooperation.
Finally, we wish to thank Cathy Devaney and Sandy Rue for
their patience and support throughout the past months.
Phi Lip T. Popovich
Robert E. Devaney
46 I
1.)& IkI d_
Table of Contents
Page
Preface .. .................. . . . . . . . . . ii
List of Figures ..................... .... . . v
List of Tables .. .................... . ...... ... . vi
Abstract ....... ............. . . .... vii
I. Introduction . . . . . . . . . . . . 1
General Issue . .......... 1A Macro View . ...... . ........ . 1The Micro View . . . . . . . . a . . 3Definitions . . . . . . . . . . . 6
Specific Problem Statement . . . . . . . 7Research Objective . , # .. . .* . . .* 12
II. Literature Review . . . . . . . . . . . 14
Purpose . . # . . #. . . 6 14Scope and Limitations . . . . . . . . 14Organization . . .. . . . . . . . . . 15Background . . . . . . . . . . . . . 15The Risk Assessment Process . . . . . 16
Stop One . . . . . . . . . . . . . 18Step Two . .... . . ....... . . . 21Step Three . . . .. . . . .. . 23
Current Perspectiveson Component Dependency ... ........ 24The Authors' Viewpoint . . . . . . . . . 28Conclusion . . ....... . . ... . ............ 29
III . Methodology . ........ . . . . ... . ............. 30
CosL-r-sk Assessment Methodology .... 30Total System Cost Variance .... 30
Step One. ............ 32Step Two . . . . .... . . . .......... 32Step Three. ........... 33
Limiting Combinations andCovariance Tersrms . . . . . . . . 33Covariance Interpretation ...... 34Correlation Coefficients andVariance-Covarlance Relationships . 36The Independence Assumption ..... ... 37
iii
S I• ., ... •, .• ,, • .,• - .•• ,.*,hr -,r.. . • ,• . , , , . '. • " .V..'*.*.*• " "
Page
The Maximum Positive LinearDependency Assumption ........... . 38The Maximum Negative LinearDependency Assumption ... ........ .. 40Conclusions About TotalLinear Dependence . . . . . . . . . . 40Estimating the Covariance . . . . . . 41
Direction of the Dependence . . . 41Strength of the Dependence .... 42
Validation Methodology ... ... ... ... ... 46What is Simulation? . . . . . .. . . 47
Description of the Simu 'lation, Model .48
Sensitivity Analysis . . . . . . . . 50Is Dependence Important? . . . ... . 51Is the Methodology Internally Valid? 54Conclusion . . . . . . . . . . . 60
IV. Analysis . . . . . . . . . . . . . . . . 61
Ratio of Variances . . . . . . . . . . . 61Internal Validity ..
V. Conclusions and Recommendations . . . . . . 77
Conclusions . . . . . . . . . . . . . . 77Limitations . . . . I .. .. . . . 79Recommendations . . . . . . . . ... . . 79
Appendix A: Calculation of Sample Size ..... A.1
Appendix B: Simulation Program . . . . . . . .. B.1
Appendix C: BMDP Programs . . . . . . . . . . . C.1
Appendix D: HisEtograms for RUN1-RUN54 ... .. D.1
Appendix E: Overlap Percentages forVariance Distributions . . . . . . E.1
Bibliography . . . . . . . . . . . . . . . . . . 262
Vito . . ... . ................. . . . 265
V iLa ................. . ... . ................... 266
Nil"
='-
ivR, iV
List of Figures
Figure Page
1.1 Risk Analysis Taxonomy ...... .......... 7
2.1 Risk Assessment Process . . . . . . . .. . 17
2.2 Cost Distribution Examples . . ........ 20
2.3 Cumulative Cost Distributions . . . .... 24
2.4 Cumulative Distribution Function . .... 27
2.5 Component Cost Dependence Relationships . . 29
3.1 Generalized Cost-risk Assessment Method . . 30
3.2 R2 Measurement Scale . .......... .44
3.3 True vs Hypothesized Cost Distributions 46
3.4 Generic Sequential System ......... .. 48
3.5 Cost-risk Assessment with Rho Errors 55
k t
List of Tables
T' Table Page
3.1 Rho Value Assignments ... ........ .. . . . 45
3.2 System Configurations . . . . 51
3.3 Acceptance Standards . . . . .. .... . . .... 59
4.1 System Configurations by Run Number . . . . 62
4.2 Ratio of Variance ...... ....... ...... . 64
4.3 m and p Values for RUN1-RUN9 . . . . . . . 67
4.4 Internal Validity Result
Matrix for Standard 1 . . . . . . . . . . . 71
4.5 Internal Validity Result
Matrix for Standard 2 . . . . . . . . . . . 73
A.1 Simulation Replications . . . . . . . . . . A.3
v i
AFIT/GSM/LSQ/85S-27
Abstract
This research project developed a cost-risk assessment
method that incorporated the effects of cost dependency
between components in a system. The method uses program
personnel's subjective assessments of component dependency
as inputs. A simulation model was developed and employed to
test the method under various levels of component dependence
sLrength and direction, estimation error, and system size.
The analysis was eccomplished by performing sensitivity
analysis on the predictive capabilities of the cost-risk
assessing method. Results indicate that the method has
strong predictive capability when component size is small
and when the direction of the component dependencies is
mixed. It was also determined that the use of component
dependency assessments produced more realistic total system
variances than those produced under the assumiiption of
colnponent cost independence.
V 4
S~vii
A 4 ,v?.. ;••, I. •• , • .. .,,• *= •. •..'. a .',.2, •t. .. ,. ' :,...,•; .,,• . i• ; . .. . ., .,,.', ', ., .,• , • . .=.,.^ ., ',.,: ,. , .:,,'• =.• ', • , ,. ,"?.=..
AN EVALUATION OF COMPONENT
DEPENDENCE IN COST-RISK ASSESSMENT
I. Introduction
General Issue
The basic economic problem in the world is scarcity.
Scarcity in critical human skills, raw materials, products
and services characterizes an environment of limited re-
sources. Like all other nations, the United States must
operate within its resource constraints. Resource scarcity
pervades and influences decisions'at all levels of our
government and industry.
SMacro View. Productive resources are distributed
between the competing demcnds of the public and private
sectors of the United States economy. Furthermore, the sum
total of resources available for public and private use is
generally considered to be fixed in the short run. Conse-
quently, the Government's share of our nation's scarce re-
sources can only be increased at the expense of private
consumption and/or investment. Even when there are unem-
ployed resources available, the utilization of these re-
sources in the public sector may still have a negative
impact on the private sector if they are paid for with tax
dollars.
Within the public sector, the gevernment must allocate
human and material resources to fulfill both national de-
[1
fense and social responsibilities. Increasing our defense
capabili'ty by producing the B-lB Bomber or the Peacekeeper
missile requires either a cutback in the resources allocated
to social programs, or a shifting of resources from the
private to the public sector. In addition, political deci-
sions in a democratic society cannot deviate substantially
from the public will without a loss of public trust. Conse-
quently, the dual constaints of public will and scarcity
limits both the quantity of resources allocated to the
public sector and their distribution among competing public
needs.
Since money is used'by free market societies as a
medium for valuing resources, the government's allocation of
its human and material assets is accomplished through the
budget process. The government indicates, through its ap-
propriation of public Zunds, how it wants resources divided
between defense and social programs. The government's re-
,ponsibility to see that these funds are wisely spent is
ingrained in the words of Thomas Paine:
Public money ought to be touched with the mostscrupulous consciousness... It is no- the produce ofriches only but of the hard earnings of labor andpoverty. It is drawn even from the bitterness ofwant and misery. Not a beggar passes, or perishes inthe streets, whose mite is not in that mass [22:1].
As ;.i steward of public funds, the DoD shares the same ethi-
cal responsibility common to all government agencies in
expending those funds.
Another reason the military establishment must use its
2
resources effectively and efficiently is to gain and main-
tain credibility. Since Congress determines what portion of
the federal budget is to be appropriated for national de-
tense, Congress must feel confident that DoD's funds are
wisely used. The leveJ of DoD appropriations is also indi-
rectly influenced through the attitudes of Congressional
constituents. When the public supports defense programs, it
is easy for 'pro-defense' J-'islators to vote their con-
science. Even those who traditionally favor social programs
will be more compelled to support defense activities in
order to maintain favor with their constituencies. For
these reasons, the military must maintain a strong public.
image, based on the efficient use of resources to meet
essential goals. To do otherwise would undermine its
ability to secure the resources necessary to provide for a
strong national defense.
The Micro View. Resource scarcity influences decisions
at all levels of government; the Department of Defense is no
exception. Resource limitations create controversy con-
cerning the correct combination of programs to maximize our
military capability. What is the optimal blend of strategic
and tactical forces? Must the purchase of new weapon sys-
tems be sacrificed to retain manpower at desired levels?
These are but a few examples of trade-off decisions that DoD
managers continually make while pursuing our defense goals.
Annual Program Objective Memorandum (POM) decisions reflect
the internal competition within the DoD for scarce resources.
41*I
3
4N.
Resource scarcity critically impacts the way DoD de-
velops and procures its weapon systems. With an emerging
emphasis on readiness and maintainability issues, a larger
share of future funding will likely be diverted from those
who acquire systems to those who operate and maintain them.
Even within the acquisition community, there are competing
rusource demands. Trade-offs are frequently made between
the amount of development effort and the number of systems
we produce. Lack of adequate funding can often limit de-
velopment effort, in areas such as test and evaluation, in
order to maintain desired production quantities. Another
example of resource trade-offs is-the 'stretching out' of
efficient production schedules to redirect funds to com-
peting programs. These examples clearly indicate that the
distribution of scarce resources, a problem at all levels of
our economy, forces major trade-offs in the acquisition of
weapon systems.
DoD resource trade-offs are couched in an environment
of imperfect information. A by-product of this uncertainty
is cost growth in weapons acquisition. Cost growth is the
increase from the initial program cost estimate to the final
program cost (3:105). Clearly, not all cost growth can be
viewed in a negative context. Cost-effective specification
changes, for example, often enhance weapon system perform-
ance beyond the initial design. Likewise, boosting produc-
tion quantities is also an intentional management decision
to increase both cost and defense capability. It would be
4
1,6
naive, however, to imply that all cost growth is controlled
or favorable. Most weapon system programs, regardless of
their degree of complexity, face a certain amount of risk...
unforeseen and possibly unfavorable.
The concept of risk has often been used to denote the
probability of an event (4;11-2). In the acquisition envir-
nment, cost-risk is the probability of achieving some de-
fined program event, in this case, a cost outcome. If the
cost exceeds the anticipated outcome a cost overrun results.
Conversely, a cost underrun is any program cost which is
less than the anticipated outcome.
The consequences of cost overruns are well recognized.
The affected program may either be cancelled, delayed or
funded. Cancellation results in unanswered defense needs,
while delays result in inefficient program execution - often
at a higher final cost. Funding a fiiiancially-troubled
program in order to avoid these consequences may come at the
expeise of other related defense programs.
Beverly, and others, state that cost-risk has commonly
been attributed to three areas: technical risk, schedule
risk and estimating risk. Technical risks arise from
striving to achieve maximum performance and technological
superiority. The 'state-of-the-art' is often advanced be-
yond its current boundaries. Schedule risk hinges on the
ability of the contractor to meet contractual delivery terms
with a product of acceptable quality. Estimating risk com-
monly results from vague early system definition, lack of
5
sufficient historical data, or inherent fluctuations in the
prices of labor and materials. While all three risk areas
impact cost, their effect may be somewhat complimentary
(2:265). For example, schedule delays may stem from techni-
cal failures. They may also result when a program cannot
continue due to a lack of funds caused by an inadequate cost
estimate. Because of their direct and complimentary effects
on cost, this research uses coat-risk in the context of an
aggregate measure of technical, schedule and estimating
risk. A program cost goal will be viewed as that program
cost which balances an accepted level of technical, schedule
and estimating risks,
Definitions. The application of risk assessment to
defense weapon system acquisition is relatively new; there-
fore, widespread agreement on common terms and definitions
has not yet emerged (14035-39). One conclusion from a 1981
risk symposium was the need for basic definitions and common
classifications (15:175,2). The solidification of termi-
nology should result as the risk assessment discipline ma-
tures.
Rowe and Somers categorize risk analysis as diagrammed
in Fig. 1.1. They view risk assessment as the estimation of
the risks associated with given program alternatives, while
risk management is the action taken to reduce these risks.
Risk analysis is considered the combination of risk assess-
ment and risk management (13:10).
6
' •' .. .T ''' ' .. . •' •' . .. ... iUP I
r ----- - - -- -
I Risk Assessment '
Risk Analysis
!Risk Management
Fig. 1.1 Risk Analysis Taxonomy Adapted from (4:1-3)
This paper accepts that taxonomy, and. will focus solely-on
risk assessment. For further clarification, this paper uses
a modified Defense Systems Management College (DSMC) defini-
tion of cost-risk assessment, "the mathematical probabili-ty
of achieving or not achieving acquisition cost.., goals".
(4:B-5). Identifying the magnitude of the deviation from.
the cost goal, along with the deviation's probability 'of
occurrence, is an implicit part of the risk assessment
process. Implied in this definition is Worm's concept of
reasonably efficient and economical practices in the con-
tractor's and government's operation (20:1).
Specific Problem Statement
While program cost estimates are traditionally pre-
sented awet single, unique value, the probability of a-
chieving that cost has generally been ignored. Not until
Ij, recently has there been an emphasis on quantifying cost-
risk, yet risk assessment has potential benefits when the
magnitude of unfavorable consequences is significant.
7
Effective cost-risk assessment identifies and helps manage-
ment focus attention on potential cost problems. The DoD
manager may then make better resource allocation decisions
in the face of uncertainty by a, •lying risk management to
appropriate program areas. Also, budget decisions made with A,
cost-risk as an additional consideration should reflect more
accurate program cost goals. Objective identification of
adequate contingency funding will also result. Request for
these contingencies during the budget process, a concept
that the Army is using (8:2), may further DoD long-run
credibility by reducing the occurrence of cost overruns.
In order to make better decisions, acquisition person-
nel need to know the probability of deviating from the
program's cost goal or the probabilityý,of an outcome falling
within a specified range of values. The following
information is needed to determine this probibility.
1. The shape of the total cost distribution is
the underlying statistical distribution for the cost of an
entire system. This distribution describes the relative
likelihood of each possible cost outcome for the system.
2. The measure of location describes where the
mean, median or mode are located on the cost distribution.
The mode, or most likely cost outcome is typically the poinL
cost estimate for the program (16:130),
3. The measure o, scale indicates the amount of
variance in the distribution. A large variance indicates a
I arge dispersion around the mean and, thereforce, a greater
8
potential program cost-risk. A good variance estimate is
crucial in cost-risk assessment. While items I and 2 are
necessary information for cost-risk assessments, they do not
impact the thrust of this research.
While this information is sufficient for identifying
cost-risk, it is seldom available for the entire system.
Current cost-risk techniques divide the system into compo-
.nents, or identifiable activities, which collectively in-
clude all required program tasks. System components are
typically Work Breakdown Structure (WBS) elements; however,
Contract Line Items or other cate8orizations may be used
(16:129). The cost of each system component is estimated
separately. Because the shape of the cost distribution for
Seach system component is rarely, if ever, known, analysts
make distributional assumptions. The Normal, Beta and Tri-
angular distributions are commonly used (18:195). The var-
iance of each component is determined by estimating the most
optimistic, most pessimistic and most-likely cost outcomes.
The component variances are then added together to yield a
total system cost variance. This allows cost-risk assess-
ments to be made around the totol system cost estimate. This
process will be discussed more in Chapter 2.
It: is the authors' opinion that the methods currently
used to assess total system cost-risk are inadequate. Al-
though cost-risk assessment is inherently subjective, cost-
risk I.s. also mis.estimated because of invalid assumptions
about the relationships between system components. Most
9
V.*~. :
often, components are assumed to be completely independent
of each other; that is, cost performance in each system
component is not associated with the cost outcomes of other
system components. In other words, an increase in the cost
of one of the system components does not imply an increase
or decrease in the cost of any other system component,
Assuming component independence, the cost variance for the
total system is simply the sum of the component variances.
This simplifying assumption is unrealistic because it under-
states the system cost variance whenever its component costs
have a positive relationship to each other.
The counter-assumption to independence is the assump-
tion that there is maximum positive linear dependency among
all system components. With this assumption, the cost var-*
iance for the total system can also be expressed entirely in
terms of individual component variances, Under this assump-
tion, an increase in the cost of a system component would
cause a totally predictable increase in all of the other
system components, However, this approach overstates the
system cost variance by assuming that the strength of a
component's cost influence on all other components is so
overwhelming as to be totally predictable. The overstate-
ment or risk occurs because cost changes are assumed to
affect components that, in reality would not be affected,
could be negatively affected, or have only a weak positive
affect. For this reason, assuming maximum positive linear
dependency is unrealistic. This will be addressed further
10.;
in Chapter 3.
Intuitively, cost-risk dependence among some system
component costs exists, but the strength of the relation--
ships will vary between component pairings and should seldom
be strong enough to correspond to maximum dependency. Sta-
tistical theory exists for identifying total system variance
under these conditions.. The details will be discussed in
Chapter 3. For now, suffice it to say that the total vari-
ance in a function of the individual variances and a covar-
iance term for each component pair. Wherever dependency
exists between component pairs, the covariance indicates the
direction of dependencei positive or negative. A positive
covariance between two components indicates that a rise in
the cost of one component will be reflected by a rise in the
cost of the second component. Similarily, a decrease in the
cost of one component will be reflected by a decrease in the
cost of the second component. On the other hand, a negative
covariance indicates that component costs will tend to move
in opposite directions. The strength of the relationship
can be determined after a simple scaling procedure.
Current risk assessment techniques do not require an
estimate of the covariance among system components. As
such, the information that is currently used is insufficient
to accurately measure total system cost variance for a
program with dependent components. The resulting system
cost-risk is inherently distorted.
11
Research Objective
The purpose of this research is to develop a methodolo-
gy which produces a more realistic evaluation of cost-risk.
Spectfically, the goal is to estimate the total system cost
variance given the existence of component dependencies,
Current statistical theory provides a solution to the
problem. However, the solution requires reasonable esti-
mates of the covariances between component pairs. Histori-
cal databases generally provide an inadequate basis for
estimating these values, which means that program management
personnel or knowledgeable system experts must be called
upon to provide subjective measures of dependency. The
problem is that these experts seldom, if ever, think about
dependencies in terms of covariances. It would be a gross
understatement to say that most experts would feel uncom-
fortable using covariance to describe dependence within a
program, The key to solving this problem is to solicite
information in terms that relate to the experts frame of
reference. Failure to do so may result in receiving no
information or infurmation which provides a distorted pic-
ture of the expert's true opinion.
The methodology developed in this research must be able
to convert this information into covariance estimates that
capture the component dependencies implied in the expert's
responses. Obviously, the adequacy of the conversion pro-
cess is critical if the total system cost variance is to
adequately reflect the 'real' cost-risk.
12
Development of this methodology, therefore, requires
that the following specific research objectives be met:
1. Develop a methodology to translate subjective
inputs into a measure of component dependency
with a logical and rational basis.
2. Convert the component dependency measurement
and other related cost data into a range-
stated estimate.
3. Validate the predictive abilities of this
technique.
The methodology to meet these objectives will be detailed in
Chapter 3.
13
I
II. Literature Review
] Purpose
V Methodologies for quantifying program cost-risk within
DoD are still evolving. Although these methodologies differ
somewhat from each other, they follow a common risk assess-
ment approach. The purpose of this chapter is to describe
this general risk assessment process through a review of the
current literature.
§Scoe and Limitations
This chapter provides an overview of cost-risk assess-
ment. It does not intend to provide a how-to procedure nor
an in-depth explanation of any one risk assessment tech-
nique. Instead, it summarizes only the general concept
fi •behind risk assessment, with special interest on the more
limited problem of determining the total system cost vari-
ance. As mentioned in Chapter 1, determining the shape and
measure of location of the total cost distribution are
beyond the scope of this research. This paper does assume
that all component cost estimates (including the low and
high estimates, which will be discussed later) are avail-
able.
Professional Journals were included in the literature
search, but they are oriented toward risk assessment in
commerce and personal investment. Their contribution to
this specific risk assessment problem was not significant.
14
For this reason, concepts in this chapter are primarily
documented in government studies, articles from symposium
proceedings, and official literature from the cost analysis
and program management communities of the three military
services.
Organization
Following a brief background of risk analysis, this
chapter will review the general risk assessment approach.
It discusses the techniques used to determine distributional
shape, measure of location, and variance for the components
and how these are transformed into the equivalent informa-
tion for the total system. It concludes with a portrayal of
this total system information in a coat-probabil$.ty rela-
tionship.
Background
The history of DoD risk assessment is quite recent.
The post-Korean conflict era found the military services
with ample funding for weapon system development, and acqui-
sition. Cost growth was generally tolerated and easily
absorbed by the national budget. Events changed in the
1960's and 19 7 0's as the Federal budget expanded in Rocial
entitlements. As competltion for scarce Federal resouzces
intensified, military cost growth quickly became nore polit-
ically sensitive. Tl.s scrutiny of DoD management fathered
the need to apply risk concepts in order to identify and
control acquisition costs.
15
..... .. .
vil"".4
In the wake of this period of cost growth sensitivity,
the 1981 Defense Acquisition Improvement Program highlighted
DoD's cost-risk awareness. Mr Carlucci's Initiative #11
'recommended' an increased effort to quantify'risk within
the DoD, and directed the military services to adopt a
method to budget funds for risk and uncertainty (10:55).
Early work with DoD risk assessment was done primarily
by the RAND Corporation. Eventually, the problem received
increased, cooperative attention from industry, government
and academe. The first symposium to include the management
of risk in the defense acquisition process was hosted by the
University of Southern California (U.S.C.• in 1979. A sec-
ond workshop, held at the Air Force Academy in 1981, was co-
sponsored by U.S.C. and the Air Force Business Research
Management Center. The most recent workshop met at' the DSMC
in 3,983 (13:6). The workshop has now evolved into a biannual
Defense activity.
Despite this increased emphasis on risk applications,
most current Do'D acquisition policies do not require formal,
quantitative risk assessments. One exception is the Army's
Total. Risk Assessing Cost Fstimate (TRACE) program which
requires risk asseisments fur selected programs and incorpo-
rates the results into the budget process (10:61).
The Risk Assessment Process
Cost-risk assessment techniques follow the general
approach illustrated in Fig. 2.1. This process has three
16
basic steps (21:3-4).
Component X 1
P(cost)
**-...- AggregationMethod
Component X2
C6ot
Component X3
Fig. 2.1. Risk Assessment Process Adapted from (18:194)
First, the weapon system is broken into its components at
some specified level of detail. The breakdown of each
system should go as low as necessary to include those compo-
nents which are considered to have particularly high risk
(11:250). The cost uncertainty for each component is then
represented by a distribution of the possible costs for the
component. Second, the component distributions are trans-
formed into a cost distribution for the total system using
some aggregation method in order to reflect the amount of
uncertainty in the total system cost. Third, this total
17
distribution is expressed in probabalistic terms to aid
program management decisions. The following sections de-
scribe these steps in more detail.
Step One. Component costs are point estimates that are
derived from cost estimating techniques such as: parametric
(cost estimating relationships), detailed (bottoms-up),
analogy, and constant multiplier ýfactor) (D:136). These
techniques require information about component physical and
performance characteristics, manhour and material require-
ments, or a subjective estimate about the similarities of
the component to other components-for which the costs are,
known.
The potential cost variability around these point esti-
mates is often represented by a probability density function
(p.d.f.). The shape of the p.d.f. often has the following
characteristics:
1. fixed, positive upper and lower bounds
2. not necessarily symmetric
3. unimodal
4, computationally simple (18:195)
Flexibility is also desirable, such that changing the values
of the distribution's parameters allows it to assume many
variations of its general shape. This characteristic offers
adaptability in representing a variety of cost patterns
(21:5). Most analysts use the Beta distribution. Othersprefer the Gamma, Normal or Weibul distributions because
these are un-bounded in at least one direction, and there-
.18
fore, leave open the possibility of unusually high and/or
low cost outcomes (18:195). The unbound portion(s) of the
curve run asymptotic to the x-axis; therefore, the proba-
bility of extremely high and/or low values approaches zero
(21:5). The triangular distribution may also be appropriate
(18:195). The selection of a distribution shape is largely
a matter of analyst preference and insight into the possible
cost outcomes. Two or more distribution shapes may be
reasonable choices in many situations.
Once a distribution has been selected, the program
management team often needs to estimate only three values to
determine the distribution's variance: low cost, most-
likely cost and high coat. The low estimate should be that
cost which results from the most optimistic conditions. The
probability of achieving a cost below this value is zero for
a bounded distribution. The most-likely estimate is the
most-probable cost, or the mode of the distribution. The
high estimate should be that pessimistic cost which reflects
the worst conoit:ions. The probability of exceeding this
high value is also zero for a bounded distribution
(17:A-50).
Given these three values, a formula may then be used to
calculate the variance. For example, the approximate vari-
ance of a Beta distribution is calculated by squaring one-
sixth of the difference, between the high and low estimates
(9:138; 12:30).
Naturally, the low, most-likely, and high estimates are
19
MIJ
subjective. Kazanowski feels that the most-likely value is
generally the most difficult to estimate because of estima-
tor indifference to a broad mid-range of values (9:138).
For certain distributions this is true. Fig. 2.2(a) repro-
sents such a case where the estimator may be reasonably
certain about the upper and lower bounds, but a large degree
of indifference exist, for the most-likely value.
1 m h 1 m h 1 m h
(a) (b) (c)
Fig. 2.2. Cost Distribution Examples
However, the distributional shapes shown in Fig.. 2.2(b) and
2.2(c) are so greatly skewed in one direction that the high
or low cost may be the most difficult to estimate.
Analysts may feel uncomfortable in estimating absolute
low or high costs, particularly if they perceive extreme
skewness and have selected an unbounded distributional
shape. In those cases, the low and high estimates need not
be absolutes. Instead, the analyst may select a reasonably
low cost and estimate the associated probability of under-
running it. He also selects a reasonably high cost and
estimates the probability of overrunning it. For example,
_ A 20
McNichols provides a revised formula for calculating the
approximate variance for a Beta distribution when the 5 and
95 percentile costs are given. The formula is the differ-
ence between the hi-gh (x. 9 3) and low (x. 0 5 ) estimates di-
vided by 3.2, quaritity squared (12036). Similarly, with a
triangular distribution, Wilder and Black provide a means of
calculating the absolute high and low costs using any per-
centile (17:A-50). Once the absolute high and low costs
have been calculated, they may be used in a formula to find
the variance for the distribution.
Sta. Two. The next step in the process is to transform
the cost estimate, distribution shape and the variance for
each component into an overall cost distribution for the
total weapon system (21:4). A common technique for deter-
mining the shape of the distribution is the method of mo-
ments, which was first applied by McNichols in 1976
(18:195A), The technique is generally used in conjunction
with parametric cost estimating methods. As a result, it
generally relies on a large historical database to determine
the probability distributions for the components and the
total system (12:20).
The method-of-moments characterizes each component cost
distribution by four additive moments. The first additive
moment is the mean of the distribution, that is, the first
moment about the origin. The second and third additive
moments are central moments (moments taken about the mean).
The second moment is the variance, which measures squared
21
lo 'mII II III II I II I u
deviations about the mean, while the third moment is used to
measure asymmetry. The fourth additive moment measures
peakedness, and is formulated usin8 the second and fourth
central moments (16:130). The four additive moments are ex-
pressed analytically so that the formulas have additive
properties. For example, the second moment (variance) for
the total system is the summation of all the component
second moments. Therefore, the shape of the total cost
distribution for a system with independent component's is
found by summing the similar moments of every component.
The analyst may then 'fit' a selected distribution to the
four moments with the aid of computerized routines (io1130).
The measure of location, or the point estimate, for the
total system cost has not been calculated in a wholly con-
sistent manner. The method-of-moments uses the sum of the
component first moments (means) as the total system cost
estimate (W:F-251 21036). Kazanowoki, who used a less so-
phisticated approach to risk assessment than the method-of-
moments, also summed the component means. He calculated
individual component means with a formula using low, most-
likely, and high costs, similar to the method discussed
earlier for calculating component variances (9:15S). In
contrast, the Army's TRACE model uses the sum of the point
estimates (modes) as the total system coat (8:46). Another
method is used by the Air Force's computerized RISK model,
where the median value of the total system cost distribution
was selected as the best measure of total cost (7:7). The
22
median is that value who!, cumulative probability of
occurrence is 50%.
The literature is significantly more consistent on
calculating the total system cost variance Lhan it is on the
measure of location. With the few -exceptions that will be
discussed shortly, analysts assume independence between
components and sum their variances for the total system cost
variance (16s130; 21:391 12:20). The method-of-moments
typifies this approach, where the variance of the total
system is the sum of each component second moment (16:130).
The literature clearly documents the need to include covar-
iance terms in order to compute'total system cost variance
when components are dependent (21:39; 9:153).
e Three. The final step translates the total system
cost distribution into a cost-probability relationship.
While the probability density function provides the decision
maker with a graphic picture of the cost variability, prob-
ability assessments are difficult in this format because
probabilities are represented by the area under the curve
(4:11-8). A cumulative distribution function (c.d.f.)
translates the area under the curve for each cost value into
a cumulative probability. The c.d.f.'s in Fig. 2.3 allow
the program manager, or other decision maker, to assess the
probability of over or underrunning a given program cost.
For example, accordin8 to Fig. 2.3(a), there is an 80%
chance that a hypothetical program will cost less than $200.
23
P(c) P( c)
.20....10
0 100 200 300 0 100 200 '300Cosat Cost
(a) (b)
Fig..2.'3 Cumulative Cost Distributions
Stated another way, the probability of overrunning $200 is
* 20X. Cumulative distribution functions also reveal the
proIbability *of a program outcome-between any two cost.,
Figure 2.3(b),,for example, shown the probability of a-
chieving a coat between $100 and $200 is 801 (901-10%). The
probability of a similar cost outcome on Figure 2.3(a) in
only 601. A steeper c.d.f., therefore, indicates a smaller
cost variance aind less program cost-risk.
Current Perspectives on Component Dependencyl
-Most current risk assessment techniques, such as the
method-of-moments, rely on the erroneous assumption of inde-
pendent system components. The problem with this asaumption
is that, in most circumstances, it understates true pro'gram
cost-risk (9t150, 16:130).
The literature explains why analysts continue to rely
24
IQ
on this assumption in spite of its weakness. Black calls
the independence assumption "troublesome" (16:130), but cites
computational ease as one reason for its use. Analysts have
not been able to apply the method-of-moments, for example,
to situations with realistic component dependency. Because
the method-of-moments technique is mathematically rigorous,
derivation of moment functions with additive properties is
not possible unless component dependencies are perfectly
predictible (16:133). The technique becomes unwieldy if it
incorporates any deviations from a strict linear relation-
ship between components.
Worm feels that the continued reliance on the inde-
pendence assumption rests in the difficulty in expressing
dependency between components in a meaningful wiyi Quanti-
fying covariance terms has been elusive because of the lack
of historical data or of a method to estimate covariance
from subjective information (21:39),
Only limited efforts have been taken to resolve the
dependency problem. Wilder and Black's approach was to
place an upward bound on the cost-risk of component depend-
ency rather than estimate its true affect. Two situations
were assessed - independence and complete linear dependence.
If we make the opposite assumption, i.e., thatthere is complete [positive] linear dependenceamong the project elements, in effect we say thatany problem with any element will be reflectedin all elements, and conversely any 'good luck'will be similarly reflected. This assumption maynot. be valid in many situations, but we feel thatit is closer to reality than the independencyassunipLion, and the region between the two
25
assumptions may be considered to bound the set of
intermediate outcomes [.6:130].
The method-of-moments was used to develop a cumulative
cost distribution for both assumptions. The two curves, as
shown in Fig. 2.4, bracket the actual, unknown curve. To
apply this concept, Wilder and Black derived a series of
dependent moments. The first dependent moment is the mean
of the distribution, while the remaining kth dependent me-o
ments are the kth root of the kth central moment, The
relationship between the dependent and the independent mo-
ments is shown below:
A1 01 D(2.1)
A2 2 22 (2.2)A 3 C3 D D33 (2.3)
A4 = C4 -3C 22 (2.4)
C 4 D 44 (2.5)
where
D . k th dependent additive momentAk - k t independent additive momentC - kt central moment01 - f~st origin moment
The dependent moments have the same additive properties as
f the independent moments. Therefore , the dependent moments
of the total program are the sums of the dependent moments
of each component (1:130-131). Referencing Fig. 2.4, the
0 independent curve underestimates the program risk for sys-
26
% .
tems with predominantly positive dependence, while the de-
pendent curve overestimates it. "It is a matter of Judgement
to determine where in this area the 'truth' lies" (16:131).
1pendent
P(cost) .50
independent0
Cost
Fig. 2.4. Cumulative Distribution FunctionsAdapted from (16:132)*
Worm offers a second approach. Rather then bound the
problem, he attempts to structure the dependency between
components . Using contract pricing items for system compo-
nents, Worm breaks each component into independent and de-
pendent portions. The dependent variation in each component
'1 is thought to stem from a common set of exogenous factors
which affect all system components. To illustrate this
concept, two components, such as labor and material costs,
may both be excessive because of design immaturity. The
dependency between these components is reflected by the
influence of the common factor, design maturity, on both
components. When more than one of the exogenous factors are
present, their effect on each component is estimated cumula-
27
n ! I I I4I I I I I I I I I I I I-
tively. The contribution of all exogenuus factors is summed
for each component and the aggregated value collectively
becomes a single, separate and independent element, Each
component is then viewed as the sum of two indepenident
random variables and the traditional additive properties of
the method-of-moments apply (21:39-43).
The Authors' Viewpoint
The successful inclusion of component dependence in
cost-risk assessment has been elusive. Two approaches to
dependency were observed, The authors' believe that neither
approach satisfactorily resolves this problem.
Wilder and Black bounded the problem by taking a total-
ly dependent and independent perspective. Neither assump-
tion is valid, They both intentionally mis-state true pro-
gram risk. Quantification of covariance terms was unneces-
sary because the components had either no influence on each
other or were maximum linearly dependent. Components with
negative dependence were not considered. While some insight
wans guined, its information value is limited by the broad
range of subjectivity left to the decision maker.
Worm's approach toward dependency is also inadequate.
Even :if all the exogenous factors are correctly identified,
an significant weakness remains. By structuring dependency
firound n set of external influences, the interrelationship
of thi, components themselves tire ignored. As shown in Fig.
2.5(a), Worm's cost estimating relationships do not account
,~ • 28
•.•,V,
for Lhe direct association jetween 8ystem components. The
trite cost dependence relationships which need to be modeled
are shown in fig. 2.5(b).
CompnentX IComponent X
Ex og n ou sComonet XComponent X2
Copoen X Componen't.X3
(a) (b)
Fig. 2.5 Component Cost Dependence Relationships
Conclusion
This chapter reviewed the general risk-assessment pro-
cess. Techniques were briefly described for estimating a
component's' cost, variance, and distributional'shape.
Methods were described to transform this component informa-
tion into the equivalent information for the total system.
The di~fficultics in incorporating conponenL dependence into
a total. system cost variance were discussed. Two approaches
to this problem were reviewed, along with their limitations.
29
III. Methodology
The literature review on cost-risk assessmeiit, sum-
marized in Chapter 2, revealed that cost dependent relation-
ships are not used in current cost-risk assessment methods.
This research attempts to overcome that limitation. The
chapter is divided into two sections. The first section
develops a cost-risk assessment method that incorporates
compunent cost dependence. The second section explains the
approach used to demonstrate the significanco of component
dependence in cost-risk asses.sment,ýand dsscribes the pro-
ccdure used to verify the internal validity of the proposed
method. Fig. 3.1 shows a general scheme for the-research.
r - - r-ISubjective I Risk I Total I
-0 Assessment I - w, System CostInputs I Process I I Variance
L ----------- I--------
-------------------------------------------Validation I
--------------
Figure 3.1 Generalized Cost-risk Assessment Method
Cost-risk Assessment Methodology
Total System Cost Variance. In the generalized form,
V total system cost variance can be expressed as the sum of
compon•(?t variances and the covariances between components.
This generalizcd expression is shown in the following equation:
30
.,-
[ I m~ | •m m"iI
n nTOT SYS COST VAR E f ~Cov(Xi.X.P (3.1.)
in-i 01m]
where
n - number of system componentsS- ith component of the system*iX component pair within the system*
and
Cov(X±*X) Var(Xi) wben imj (3.2)
Winkler and Hayes (191186) define covariance in terms,
of expected values as
Cov(X11 X) E [(Xi E(Xi)) (X~ E(X).3 (3.3)
When dealing with population samplest the expression is
modified to
Cov(XitX). E [ (X1 - d (X~ (3.4)
The associative property, when applied to expected values,
states that E(ab) n E(ba) .The order in which the terms
aire multiplied has rno effect on the expected value. There-
fore, Eq (3.4) can be written as:
31
SE (Xxi - i) (xj -JYJ) ]
=E (X - ) (Xi Yi) (3.5)
Therefore,
Cov(XiXj) - Cov(XjXi) (3.6)
"Tho covariance between any two pair of components appears
twice in Eq (3.1), once an Cov(XiXj) and once a~s
Cov(XXo)x. By substituting Cov(XiXj) for Cov(Xj.X 1 )
whenever j is greater than i, the total number of covariance
terms in Eq (3.1) is reduced by, half. 9nly the covariance
terms Cov(XiXj) for i less than J remain in'the equation.
Furthermore, each term is multiplied byitwo to account for
the substitution.
In the cost-risk assessment process, for a given system
having two components, X1 and X2 , the determination of total
system cost variance requires three steps.
Step One. Determine the variance for the cost
distribution of Component X1 and Component X2 . This is
usually accomplished by determining the low, moat-likely,
and high cost values and applying a variance assessmenttechniq-e as described in Chapter 2.
Step Two. Determine the covariance between compo-
nents Xand X2 Currently, this is a major problem in
acquisition program cost estimation. A latter part of this
chapter will offer a subjectively-based means of measuring
32
the strength of a dependency relationship.
Step Three. The cost variances and covariances of
each component and its pairwise influence are summed. For
the two component system, the following terms are generated:
Var(X 1 ), Var(X 2 ), Cov(X 1 ,X 2 ), Cov(X 2 ,X 1 ). From Eq (3.6),
* the third and fourth terms are equal, so
Cov(X 1 ,X 2 ) + Cov(X 2 ,X 1 ) I 2Cov(X ,X 2 ) (3.7)
The total system cost variance can then be expanded in
equation form from Eq (3.1) to
TOT SYS COST VAR - Var(Xl) + Var(X 2 )
+ 2Cov(X 1 ,X 2 ). (3.8)
Limiting Combinations and Covariance Terms. In a sys-
tem with more than two components, more than two covariance
terms must be expressed, one for each pairwise combination.
T1ý number of covariance terms requIred is the combination
expression:
2) - ni / 21(n-2)1 (3.9)
wheren - number of components in the system
The number of pairwise combinations rises quickly as the
33
number of components in the system rises. For example, in a
system with three components, the number of covariance terms
required is:3,)2)-/31 21(3-2)1 . 3 (3.10)
fHowever, the required combinations for a system with ten
components is:
10) M .10 21(10-2)! -, 45 (3.11)
Since much larger systems are not uncommon, the authors will
simplify the research by defining a system as a linear
series of components. Components will have, at most, only
M, one preceding and one subsequent task. A major assumption
*i with this approach is that all component dependence is
captured in a sequential manner. For example, dependence
between components X' and X3is assumed to be accounted for
Sby the intervening component, X2 . Component X1 has no cost
relation to Component X3 , except as indirectly transferred
through Component X2 . The system, therefore, is greatly
simplified because it has only 'n-l' covariance terms rather
than the theoretical maximum in Eq (3.9).
Coveriance Interpretation. The preceding paragraphs
have discussed the definition of covariance, its properties,
and how it affects the total system cost variance. While a
covariance term can assess the association between two com-
34
ponents, it is not without its flaws. There are three
problems with covariance - all caused by its units of meas-
urement.
First, the magnitude of the covariance term is extreme-
ly sensitive to the components' units of measurement. If the
* units of Component X1 and Component X2 are expressed in
dollars, the covariance term is expressed with these units:
Cov(X 1 ,X 2 ) (dollars) (dollars). If! however, the units are
expressed in other terms, such as cents, the magnitude of
the covariance will change even though the relationship
between the components remains the same. Essentially, the
magnitude of the covariance term can be manipulated by
changing the units of the dependent components.
The second issue concerning units in the covariance
term is that the units do not make sense. If Component X1
and Component X2 have units of pounds and miles respective-
ly, the resulting covariance term will be expressed in
pound-miles. This expression is confusing and serves no
useful purpose in the determination of dependence between
the two components.
The third problem with units deals with the compar-
ability of different covariance terms. The different units
that are used with the covariance term prohibit the compari-
son of the dependence between different pairs of components.
The following example illustrates this problem.
Cov(X 1,X 2 ) is expressed in the same units used to measure
Component X1 and Component X2. If Component X1 is measured
35
*41
in pounds and Component X2 is measured in miles, the co-
variance between the components is measured in pounds times
miles. Likewise, if Component X3 and Component X4 are
measured in volts and square feet, respectively, the co-
..variance between them is measured in volts times square
feet. Obviously, this difference in units of measure makes
it impossible to compare Cov(X 1 ,X 2 ) with Cov(X 3 ,X 4 ). It is
a comparison of apples and oranges.
Correlation Coefficient &a Variance-Covariance
Relationships. In order to compare covariances, the vari-
ance, of the components (component costs in this case) are
used to scale the covaridnce terms. The outcome is the
correlation coefficient, Rho (R).
Rx x - Cov(Xi,Xj) / [ Var(X,) Var(X )P]} (3.12)
The correlation coefficient ranges from -1 to +1. A value
of -1 corresponds to maximum negative linear dependence; a
value of +1 corresponds to maximum positive linear depend-
ence; and a value of 0 corresponds to independence between
the components' costs. These terms were defined in
Chapter 1.
Eq (3.12) makes it possible to solve the problems of
units of measure that were discussed earlier. For Component
X1 and Component X2 in pounds and miles, Eq (3.12) scales
the covariance term by eliminating the units of measurement.
In order Lo avoid confusion and to illustrate this point, Eq
36
(3.12) is modified to show only the units in the equation.
R X = (pound-miles) / ( (pounds) 2 (miles) 2 ]½ (3.13)
The units in the equation cancel. RhoX X2 is now a unitless.
value whose magnitude has been scaled. The relationship
between Rho, component variances and component covariances
is a critical relationship for this research. The popular
aseumptions of component independence and maximum positive
linear dependence are based on this relationship. Later,
this relationship will be used as a major part of the cost-
risk-assessment method that this research proposes.
The Independence As aumDtiond In Chapter 2' it was men-
tioned that since the quantification of dependency between
the cost of system components is difficult to assess, many
cost-risk assessment methodologies assume component inde-
paidence. This assumption allows for the exclusion'of the
covariance terms from the total system cost variance equa-
tion. The independence assumption means that there is no
association between the cost of one component and the cost
of any other system component. The independence assumption
also implies a Rho value of zero (19:187).
Therefore,
RX X2 - Cov(XiXj)/[ Var(Xi) Var(Xj) ]• - 0 (3.14)
If Rho is zero, the numerator of the ratio must be zero.
37
Therefore, the independence assumption implies
NCov(Xi.X)m 0 (3.15)
The equation for total system coat variance, then, can be
expressed as.
TOT SYS COST'VAR Var (Xi) (.6
The eoffect of this'deletion is obvious. Assessment of
covariance terms is unnecessary under the independence
assaumpt ion.
The MaAimuF. gositivej Linear Dependeace Asumption.
Since the assumption of total independence seems, to violate
our intuitive understanding of how systems are developod,
some coot-risk assessment methodologies assume total posi-
tive linear dependence between component costs. Total posi-.
tive lineair dependence means thiat if the cost of one comnpo-
nent is higher than expected, not only will the the cost of
all subsequ'ent components be higher than'expected, but the
m~agnitude of the overrun can he predicted with absolute
certainty. In essence, the total systemi cost io entiralji
detoirmined by the cost of the fix-st component in rbe s)atem.
This assumpi-.ion also enables us to compute the totail
system cost variance without e.stimating covariance terms.
Aýa~. the expression for R~ho is used to demonstrate this
phenomenon. Total positive linear dependence betwveen two
38
components' costs is represented by a correlation coeffi-
cient equal to +1. When Rho is one, Eq (3.12) gives the
following solution for the covariance:
RXx X Cov(XlX 2)/[ Var(X 1) Var(X2 ) ]~.1 (3.17)
and
Cov(XlPX 2) [-(Var(X,) Var(X 2) J~(3. 18)
In 8eneral,
Cov(X±t.X) *RX X Var(X,) Var(X~)] (3.19)
The fect that Rho falls-between -1 and +1 means that the,
Cov(X11 X2) is at its maximum value when Rho is one. . There-
fore, Eq (3.1) for the total system cost variance will also
be m~aximized when total positive linear dependence is as-
sumed.
F~or the two component example, making the substitution
shown in Eq (3.18) in the oquation for total system coat
variance gives the following expression:
TOT SYS COST VAR w Var(X 1) + Var(X,2 )
+ 2 [ Var(X1 ) Var( X2) ]~(3.20)
Under the assumption of total poaii~t1ve linear dependence,
39
.. 0' At If
total system cost variance can be expressed without the use
of covariance terms. As mentioned in previous chapters,
this approach is not intuitively appealing because it as-
sumes that extreme cost outcomes will ripple through the
entire system. Using this assumption, therefore, inflates
the total system cost variance.
The Maximum Negative Linjar Denendence Assumption.
Component cost dependencies are not always positive - a low
cost in one component doee not always mean that other compo-
nents' costs will be lowl a high cost in one component does
not always mean'that other components' costm will be high.
When a low cost in one component is associated with a high
cost in another component, the dependency relationship is
negative. The assumption of maximum negative linear do-
pendency sets the Value of Rho for all pairvise combinations
of components eqUal to -1. Eq (3.19) Indicates that if Rho
is -1, the covariance term takes on its smallest possible
value, which also happens to be negative.
Cov(XiXj) -1 ( Var(Xi) Var(X3 ) 1} (3.21)
Therefore, Eq (3.1) will be minimized when total negative
liniear dependence is assumed.
Conclusions About Tojel Linear Dopendence, These two
assumptions, total positive linear dependence and total
negative linear dependence, generate the maximum and minimum
bounds for the true value of the total system cost variance.
40
Normally, it is expected that some covariance terms would be
positive and some would be negative. The sum of the co-
variances may be negative or positive, depending on the sign
and magnitude of the individual terms. To the extent that
this sum differs from zero, the total system cost variance
,obtained under the assumption of total independence will
either over- or underestimate the true system cost variance.
Estimating, te Cov,ariance. As Chapter 1 mentioned, the
database does not exist 'to calculate covariance terms. This
necessitates the use of subjective information to determine
the degree of cost dependence between components. This
information, solicited in terms that are understandable to
program personnel, must be converted into a quantitative
covariance value for risk assessment,
Direction o2f the Dependence. One piece of infor-
mation that tho analyst needs in order to estimate the
covariance is the direction of the component dependencies.
The direction can be implied simply by asking the program
specialist how the extreme cost outcomes for one component
(high or low) relate to the extreme cost outcomes for the
other component. The high and low costs have the same
meaning here as they did in the variance estimating techni-
ques in Chapter 2, If, in the specialist's Judgement,
component cost outcomes tend to be complimentary (high-high
or low-low), then the implied sign of the relationship is
positive. If, on the other hand, the cost outcomes are
41
If . � , t '�..�"' % .SAM
perceived to be generally opposite (high-low or low-high),
then the relationship is assumed to be negative. The most-
likely cost outcome could also be used, but the extreme
values give the analyst the, beat insight into the direction
of the component dependencies.
Strensith of the Deoendence. Program personnel
will not be able to directly estimate the magnitude of the
covariance because the covariance lacks any intuitive mean-
ing, As mentioned earlier in this chapter, covariance may
be expressed in units which are nonsensical and the magni-
tude of the covariance will fluctuate with the bcaie of the
units of measurement. Conceivably, these problems could be
overcome if subjective inputs were solicited-in terms of the
scaled covariance, Rho. Rho is unitless and is standardized
on the interval (-1,1). A more appropriate technique, how-
ever, would be to solicit subjective inputs in terms of the
coefficient of determination, Rho squared.
Winkler and Hays describe the coefficient of determina-
tion ab the proportion of total variation in a variable that
is accounted for by a linear relationship with another
variable (19:633). For example, a Rho squared value of .49
(Rho-.7) indicates that the relationship between the two
variables accounts for 49% of the total variation which
would occur if the variables were independent. Essentially,
Rho squared and Rho are the same measure of association
because the magnitude of one can be determined from the
magnitude of the other. Rho lies on the interval (-1,1).
42
Rho squared is always positive and is bound by (0,1).
The advantage of using Rho squared Lo assess the
strength of dependency is that it weasures variation in
terms of proporttons. Therefore, identical increments any-
where alongI.s interval have the same 'explanatory power'
because each fixed increment represents a constant"propor-
tion of total explained variation. For example, changes in
Rhosquared from .1 to .2 and from .8 to .9 both increase
explained variation by 10%. Rho, however, is not as 'well-
behaved" as Rho squared. Increases in Rho that are near the
endpoints of its interval have grujcer 'explanatory power'
than the same increment near the* ceamte of the interval.
For example, a similar-change in Rho from 1. to .2 and.from
.8 to .9 would explain 32 and 17% more variation, respec-
tively. Program personnel are more likely to give more
representative pssessments of the magnitude of component
dependence if the analyst uses Rho squared.
To apply the coefficient of determination, the program
specialist is asked to rate the likelihood of the cost
relationship that he identified earlier for the direction of
the dependence. Jiny program personnel should be able to
"interpret a question such as, "On the scale shown in
* Fig. 3.2, what is the predictability of a component's cost
outcome (high or low), given that the cost of the other
component was high, or low"?
•,'• 43
%/
totally moderately completelytncertAin certain certain
0--. --. 2--.3--.4--.5--.6--.7--.8--.9--l.0
Fig. 3.2 R2 Measurement Scale
The values on the interval in Fig. 3.2 measure the cost
association in a manner tha~t is analogous-to Rho squared.
Each value on the interval represents the proportion of
c.artainty in the cost dependenccu 6etween components. This
is similar to Rho squared, which measures the proportion of
totai variation that Is explained by the relationship with
another Variable. Values near 0 imply component indepen'd-
ence and values near I imply total-linear dependence. Once
the amount of dependence is assessed, the iuagnitude of ?ho
is calculated by taking the square root of the Aumber that
was selected on Fig. 3.2. Rho will, be used later to ,calcu-
late an estimate for the covariance.
An alternate method estimaCes the strength of the de-
pendency without relying on any numerical input from the
program specialist. Using descriptive categorios, for exam-
pie, the program specialist should be able to categorize the
cost association between components as either weak, mo-
derate, strong, or none. The number of categories could be
.9 expanded, but for simplicity, only these four will be con-
sidered here. An assignment for the magnitude of Rho can be
made to each descriptive category. Devore lends credibility
44
7 AIN
to such an approach by giving general descriptive q~ualities
to various Rho values (5:184). An example of these assign-
mernts are shown below in Table 3.1.
-- * - -- - -- - - -- - - -- - -
I Tabla-3.1
Rho Valuo A's 6i'Onm n til
g~n~ent. RelatIons-hip ~ Value
;NoneWeak I1 t o .3
Moderate .3 to .7Strong .7 to .9
Th~e actual. magnitude used fo.r Rho could be the Mi~dpoint of
'thi appropriate interval. The width of each interval willnarrow as the number of descripLi've categories i's increased.
The magniý'ude of Rho, based on either technique, is
*. . merged with the assessment of the dependency direction.
Combined, the two result in a positive or negative Rho
value. An ebtimate of the covariance can now be made from
the Rho value and the statistical relationships which were
previously discussed with Eq (3.19).
CoV(X~i) vk R X V'ar(X,) Var(X )(3.19)
The covarianrne is calculated from the component variances
and the Rho estimate thait was doat.rmined above.
'5Z
Validation Mfthodoo2o
The application of this cost-risk assessment technique
to a real system would not provide an adequate basis for
validating the methodology because each program reprisents
only a single event from a!unique distribution. The final
cost outcome for any si•gle program provides insufficient
information tO test the accurasy,'or lack thereof, of the
variance produeed by risk assessment techniques. For exam-
ple, a cost outcome that deviates substantially from the
initial estimate could imply that the true, system cost
variance was underestimated., KThis. is illustratedin
Fig. 3.3(a). However, it could also be true that the total
syste~m variance is accurately estimated, as in Fig.'3.3(b),
but the 'most'-likely' cost estimate is in error,.* Finally,
both the point estimate and the variance may be accurately
Ne estimated, but the outcomn is a 'rare event', as in
Fig. 3.3(c). The problem with only one data point is that the
analyst. will never know the true state of affuirs.
Trle..1. , ypTrue-Tru I : /
x x
I (a) (b) (r)
Fig. 3.3 a/b/c True vs Hypothesized Cost Distributions
46
m A . .,. . .
Likewise, a cost outcome near the mode of the hypothe-
sized distribution may give the analyst a 'warm feeling'
which is entirely unwarranted. The fact that the outcome
occurred near the mode of the hypothesized distribution says
very little about the accuracy of the estimated total system
variance. Furthermore, the outcome could be an outlying
observation from a true distribution that differs signifi-
ýantly front the hypothesized: distribution. Again, the evi-
dence is inadequate to draw any reasonable conclusions.
Because the cost-risk methodology can never be vali-
dated, in the true sense, by reel-world application, the
authors propose a simulation technique as a quasi-validation
tool. If the subjective inputs used in the method are
assumed to be correct, the effect of component dependence on
the total system cost variance is indicated by comparing the
results of the simulation with the case where no Jependen-
cles exist. When errors in the subjective inputs are con-
sidered, simulation will lend insights on the sensitivity of
the total system cost variance to those errors.
What is Simulation? Banks and Carson (1:2) describe
simulation as the imitation of a real-world system. The
behavior of the real system is studied by developing a model
which describes the salient characteristics of the real
system through mathematical and logical relationships.
Models that do not use random variables as inputs are deter-
ministic. These models have a known set of inputs which
re-ult in a unique output. A stochastic simulation has one
47
or more random inputs and leads to random mode.I outputs
(1:10). The simulation model proposed in this thesis will
generate repeated estimates of the total system variance
from stochastic inputs which measure component dependency.
Multiple iterations of the simulation generates an estimate
of the true distribution of total system variance for the
system described by the model.
Desicrption of the Simulation Model. The model used in
this thesis simulates a generic system as illustrated in Fig
3.4, where 'n' is the number of system components. The
components are completed in sequence such that each compo-
nent has only one precedent task.
I Component i ComponentI, I I I .ln, • , o , • ,•
X I 2 I nL -------- j -----------.
Fig. 3.4 Generic Sequential System
This simplification restricts the number of possible systems
to consider when compared to a WBS configuration. Given 'n'
system components, a WBS hierarchy has the flexibility of
describing the component relationships in a variety of ways,
while the sequential system has only one. Such a restric-
tion is desireable because it allows us to examine dependen-
cy relationships in their simplest form. This simplicity
also makes the system easier to model because the system
definition can be changed by varying system size without
48
having to redefine its 'shape'. Note, however, that this
simplification is made for research purposes only. In no
way does it restrict the cost-risk assessment method from
application to a WBS hierarchy. As mentioned earlier, the
simplification redutes the number of component pairs for the
system to 'n-i'. This methodology can easily be extended to
any WBS framework with pote.ntially ase many component cos€t
relationships as described by-Eq (3.9).
The simulation model stochastically generates Rho
values for the 'n-i' component pairings based on a tri-
angular distribution with (0,1) bound!. The triangular
distribution-was chosen because it is unimodal and easy to
manipulate. With the fixed endpoint., the researchers have
complete control over the distribut1on by changing only one
value, the mode. The mode of the distribution represents a
Rho value that could likely result from a program special-
ist's assessment of component dependence. Manipulating the
mode allows the researchers to vary the program specialist's
estimaLes of the component relationships. The mode applies
for all 'n-l' component pairs, but the exact Rho value
generated for each pair will randomly deviate from the mode
according to the distribution. The mode is specified in the
simulation as either .50, .707, or .866.
A uniform distribution is then used to randomly select
Rho values which are subsequently given a negative sign to
indicate a negative dependency. The approximate proportion
of the Rho values identified as negative dependencies is
49
controlled by the modeler's selection. The proportion is
set to either 1.0, .75, or .50.
The model then uses component variances, which are hold,
constant throughout the simulation, and the 'n-i' Rho values
to caIculat.o the covariance terms for each sequential pair
of components, Total system variance is then calculated by
summing the variance and covariance terms, Each observation
'from the simulation .represents one total system variance
estimate using randomly generated Rho values.
The model is then re-executed so that it generates a
new series of Rho values using the same triangular di-tribu.-
tiozi and proportion of negative Rhos, Covariance terms and
total system variance are recalculated as before. The out-
put of this simulation is another estimate oftotal system
variance, Multiple iterations are performed to provide a,
distribution of probable total system variances under the
stated characteristics of the system.
SensitivitZy Analysis,. As mentioned at the beginning of
this chapter, the objectives of the validation effort are 1)
to demonstrate the significance of component dependence in
Y'.i cost-risk assessment, and 2) to verify the internal validi-
ty of the proposed method. The researchers do not believe
that the answers from these two objectives will be similar
for all systems. Instead, modeling componont dependence may
be more critical for certain systems. Similarly, there may
be some system 'limits' beyond which the proposed methcd i&
not a desireable technique. For this reason, the simulation
50
model tests a variety of sysLemrn 'OenAigurations. Table '.2
summarizes those model inputs which characterize each system
and the values they will assume. These values were chosen
in ordetr to i'nvestigate the broadspectrum of dependence
scenarios that could occur in a system acquisition pro8ram,
--- ,,-------------------------------------------------------------
Table
System Confisuratt,,jns
Model], Parameter Symbol *.' Values Assumed
Mode of triangular m .5 .707 .866distribution foe rho
Proportion of positive rhos p 1.0 .75 .30
Number of components n 5 50 100
Error in rho squared . e .10 .20
* Values represent rhos which explain 25%, 50%, and75% of total variation.
** Tu be discussed later in this chapter.
Simulations will be performed for all 54 combinations of
input parameters. The number of iterations required to
provide an appropriate estimate of total system variance for
each system conigurution is documented in Appendix A. The
computer algorithm and associated documentation to perform
these q:Lmula•t:ions is found in Appendix B.
Is Dependence Important? One objective of the valida-
Lion effort is to determine if component dependence should
be an Ilmportant consideration in cost-risk asscssmont. Spe-
51
-0 1- . .....
cifically, what impact does component dependency have on the
total system cost variance? The simulation model was de-
signed to answer this question. Recall that under the
independence assumption, the total system cost variance is
merely the sum of all of the component variances. The
simulation model sums the component variances to attain the
total system variance for the independent System. The model
also simulates 54 different system configurat'ions wi~th var-
ious levels of component dependency. A distribution of the
total system cost variance will be ou tput for each of these
systems. By comparing the total system cost variance under
the independence assumption with total system variance dis-
tribution under the different system configurations, the
impact of component dependence can be assessed.
A measure of significance is needed to compare inde-
pendence with dependence. The ideal measure would be to
compare the affect of both variances on the confidence
intervals for a system's cost estimate using Eq (3.22).
Interval Estimate = Point Ebtimate + [(k) (s)] (3.22)
In Eq (3.22), k represents the niumber of standard errors
that must be added and subtracted from the point estimate to
arrive at the interval estimate. It is a function of the
probability disLribution of cost outcomes and the desired
level of confidence that the actual cost will fail within
the interval, The standard error is represented by s. The
52
g ,,. &$ b~~~V 1
collective term, [(k) (s)], is the precision of the inter-
val.
The above confidence interval in Eq (3,22) would bq
incorrect if it used the independent system standard error
(sI) when, indeed, the system has component dependence. The
dependent standard error (sD) would have to be used t6
provide th.; true confidence interval., Comparing the two
confidence intervals would demnons.trate the ultimate impact
of component dependence on system cost uncertainty. This
measure, however, is not available here bec'uuse no system
cost estimate was used, or assumed. This risk. assessment
inmothod simply assumed that A cost estimate is available from
current estimating techniques. Furthermore, k requires a~n
assumption about the cost distribution's shape, which is
beyond the scope of this research".
Although this research cannot directly compare the two
cost intervels, it ran make a direct comparison of the
interval precision, as shown in Eq (3.23).
(k) (n) I / (k) (SI) I (3.23)
Because k has a constanL value, Eq (3.23) simplifies to
I(%D) / (sT) 1 . The mignitude of this ratio indicates the
reltinye impact of dependence relationships on the system's
contidonce interval. The, ratio indicates to what. extent the
precision of the interva.l is over or understated. If the
ratio is close to one, component dependency iii not signifi-
53
cant and the difference between the cost interval using SD
and sI is small. As the ratio moves away from one, in
either direction, the component dependency relationships are
more significant. Under these conditions, confidence inter-
vals for weapon system cost estimates become increasingly
"unreliable using the independeit standard error,
Because this research focuses on ve>ces rather than
standard orror~s, the authors will use the ratio of -the two
total system cost variances'as the measure of significance.
The relationship between the ratio of the variances and the
ratio of the standard errors is shown in Eq (3.24).
RA•IO 2 2 2of ( D / 1) a,.) (3.24)
VARIANCES
The mean of the total system cost variance distribution will
be used as the .2D for each system. The accuracy of this
value in estimating the true mean of the distribution is
discussed in Appendix A. The authors consider 1.2 an
overestimate and .80 an underestimate of the precision
[(k) (s)] in Eq (3.23) to be acceptable. Therefore, any
ratio of total system cost variance that is between .64 and
1.44 meets the acceptance criteria. For, those systems, the
affects of component dependence is not considered to be
significant.
In the Mathodololy Internally Valid? A second ob-
54
..... ....
Jective of the validation effort is to determine the inter-
nal validity of the cost-risk assessment method. The inter-
nal validity indicates the ability of the analyst to predict
the total system cost variance for a system with component
dependence when theprogram specialist's subjective assess-
ments are erroneous, Realistically, the specialist will not
be able to estimate the strength of dependencies accurately.
Additional error will be induced by the method which rcon-
verts the specialist's inputs into a Rho. value. This is
especially true with the flexibility of Rho assignments to
descriptive categories. As a result, errors in the Rho
estimate will affect the calculation of total system cost
variance, The simulation.s described so for have assumed
that the analyst has an accurate Rho estimate. We assumed
that the mode of the triangular distributio'n and the propor-
tion of negative dependencies were correctly identified,
The iimulaLion will output a distribution of the total
system variance under this assumptiong; we refer to it as the
'normal' case. However, this distribution would be affected
by errors in estimating Rho, as demonstrated in Fig. 3.5.
S--- ......... "R- &- Total Sys Var (low)Subjective I Rho I I
-1' EstLimation i----HR .- D---Total Sys Var (norm)Tnpucs Tv chn i qu e I
L ----------- -- R+ -- Total Sys Var (high)
F'ig. 3.5 Cust-risk Assessment: with Rho Error
The SmuL.Lntion modvl ovaluatres the impact. of the error in
5.5
subjective inputs by placing upper and lower bounds on the
Rhos that are randomly generated. These bounds represent
the analysts confidence in the Rho estimate. If an analyst
feels uncertain about his estimate of Rho, he could place
bounds' around' his estimate and also calculate, the total
, ,syoter V'ariance Using these bounds as a way of evaluating
the potential effect of his estimating error.
A,,method to bouhd Rho would be to bound the percent of
Stotal variation represented by Rho squared. For example,
&'uppose the analyst feels that Rho is approximately .7,
which equates to a Rho squared of .49,, This figure indi-
cates that the relationship between the two variables ac-
.ounta for 49% of the total variation which would occur if
the variables were completely independent. If the analyst
:C felt thatl the correct total variation sh Iould not vq'ry by
more than plus or minus five percent, the upper and lower
bounds on Rho squarod would correspond to 54%.and 45%,
respectively. Taking the square roots of these percentages
results in upper and lower bounds on Rho.
Rho squared error is introduced into the simulation
using the parameter 'o'. Two values will be assigned, .I
and .2. The .1 value was the basis of the above example,
where the bound on Rho squared for n possible 10% error
5%) in total. explained variation. The simulation model
'splits' the 'p' value equally around the Rho squared for
the 'normal' case. Any bounds which fall outside of the 0
and I nrterval for Rho squared are truncated to 0 or 1.
56
m
'The In;r and upper bounds on Rho can be used to calcu-
late a total system variance which is consistently higher or
lower than the total system variance for- the 'normal' case.
Recall that the total system variance is calculated by
summing all system variance and covariance terms There-
fore, given constant variances, the cotal system variance is
affected by the changing only values of the covariance
terms. The covariance was defined in Eq (3.19) as
CoV(XlXj) = R xi [ Var(xi) Var(X ) . From this equa-
4tion, the covariance for each component pair is maximized
when positive Rhos assume their highest magnitude and nega-
tive Rhos assume their lovest. Similarly, covariance is
minimized when positive and negative Rhos assume their Jow-
est and highest values, respectively. If the up.,,: and
lower Rho values for each component pair are used accord-
ingly in the compuLation of the total system variance, the
resultant output distributions reveal the highest and lowest
possible deviations of total system variance from the 'nor-
mal' case.
Ldrge overlaps between the 'normal' and 'low' and be-
tween the 'normal' and 'high' distributions indcicate only
minor sensitivity to Rho eatimating error. If the model
Sproduces 'low' and 'high' distributions that are close
enough to the 'normal case', then the methodology may be
applicable to real systema with predominantly similar char-
acteristics. However, the model may reveal ,.hat predictions
of total system cost variance deviate unacceptably when Rhos
57
..,.* ,. , . .", .. ,.,.,.-. .• r -. .:'.,•, ,. , j ,. .,.: .:.,. ,.. ." ..:. . . " . , ,, ,••,,,
are slighity Lnaccurete. This tells us that the methodology
requires furthvr refinement, or that it should be selec-
tively applied to only the most 'forgiving' systems.
Overlap of the 'low' and 'normal' distributions will be
measured by counting the n~umber of observations on the 'lo~w'
distribution t-hat appear above the 5, 10, and-15 percentiles
of the 'normal' cases Overlap of the ':hi.gh' and 'normal'.
distributions will be measured by, counting th'e n~umber of
observations on the 'hi~gh' distribution that fall below the
95, 90, and 85 percentiles of the 'nrml case. This-
technique provides the researchers with a standard measure
of overlap for the 54 system-s with-out making any assumptions
about the shape of their threeý variance distributions. It
also provides a 'well-behaved' measureme~nt thet is not sen-..
sitive to the location of 'rare events'* in the 'normal'
cese.
A decision rule is needed to determine what constitutes
'significant' overlap within the six percentiles uii the
'normat' distribution. The authors feel that the standards
in Table 2:j.. are reasonable. Our intention is to determine
howt well each system, on the whole, overlaps the 'normal'
case. The above criteria yields six standards for each of
tlic 54 systems, however, the authors do not want a prolifer-
ation of standnrds to 'mask' the results. For this reason,
we c hoose three criteria to measure each system 's per form-r
e~nc. apainst the standards.
58
-b- \*44J - ~ 4 ~ -4*4~* ~ h. ~. *4*, ~ ~ *.~ $
- - - - ---- - - ----- - -- - - - - - - - -- - - - - - - - -
Table :3.3
Acceptotice Standards
rl Z of 'low' % of 'high'noermal' observations observa!-ions
*percentile above %tile below %ileS.05 7t
.10 60%
.15 50%4 .95 70%
.90 60%".85 5,0%
"Criteria .#I is Lha most str"ingent of the three. It
describes only those systems which meet all six of the
standards in Table 3.3. This criteria identifies those
systems for' whLich the proposed risk assessment methodology
has strong predictive ability. Criteria #2 describes those
systems with both 'high' and 'low' distributions that meet
at least two of their three standards, In other words,
neither the 'high' nor 'low' distributions can fail to meet
more than one standard. The risk assessment methodology is
moderately able to predict total system cost variance for
these systems. Those systems which met Criteria #1 are
exclud(-d from this category. Criteria #3 describes all
other systems. For these systems, the proposed methodology
exhibitS its weakest application potential.
Because the selection of standards is subjective, the
researchers feel. that a second set of standards will
SstLengithen the rul iability of the findings. The second set
59
.x1"-'• , - ,. ", r. . " " ,, " • . '" • " • "" ,¢'"," ' "" " : ".';?".•
of standards will be similar to those in 'CAble 3.3, except
that the percentage of overlap for each percentile will be
'loosened' by 20 percent (50%, 40%, 30%). Criterion #1, #2
and #3 will also bo appl'Ld to this set of standards, Hope-
Sfully, this approchwlll .end better visibility toany Itra n•s that might not emerge with a wori, rigid standard.
Conc~lueion
This chapter ha.soutlined the methodology that will be
used to incorporate compbient dependency into an assessment
of total system cost variance'. The underlying statistical
framewcrk was developed as well as a technique to apply
thone con,:epts. A validation mathorology of the risk as-
sopsment technique wa4 proposed. The following chapter
analyz&,. the results of this methodolgy.
60
IV. Analysis
This chapter will apply the research methodology
developed in the previous chapter to assess the results of
the simulation. The data that is generated will be used to
assess the need for the incorporation of component depend-
ence in cost-risk assessment, The final portion of the
chapter is devoted tu a discussion of the internal validity'
of the model, and its implications for the cost analyst.
The simulation was performed for each of the possible
system configurations in Chapter 3, Table 2. The output
files of the Fortran simulation (OUTi'- OUT54) were input
into the BMDP statistical package which sorted the total
system cost variances for the 'low', 'normal', and 'high'
distributions, plotted histograms of these distributions,
and provided pertinent statistical information (6:74-78,112-
113, 124-132). The BMDP filenames (RUN1 - RUN54) are listed
in Table 4.1, below, with a list of the parameters that were
modeled. The BMDP code is found in Appendix C. The histo-
grains are provided in Appendix D.
Ratio of Variances
As mentioned in Chapter 3, the ratio of variances was
Sused as a measure of the significance of component dependence
in cosL-risk assessment. Table 4,2 gives the total cost
variance assuming independence for each system and summarizes
61
the calculations for the ratio of variances.
Table 4.1
System Configurations by Run Number-- - - - - - -- - - - - - - m -- -- -- -- -
"e .2 n -5 e a .1
RON# I IU.
1 .500 1.0 10 .500 1.0
2 .500 .75 11 .500 .75
3 .500 .50 12 .500 .50
4 .707 1.0 13 .707 1.0
5 .707 .75 14 .707 .75,
6 .707 .50 15 .707 .50
7 .866 1.0 16 .866 1.0
8 .866 .75 17 .866 .75
9 .866 .50 18 .866 .50
e w .2 n - 50 e .1
RUN# m RUN#
19 .500 1.0 28 .500 1.0
20 .500 .75 29 .500 .75
21 .500 .50 30 .500 .50
22 .707 1.0 31 .707 1.0
23 .707 .75 32 .707 .75
24 .707 .50 33 .707 .50
25 .866 1.0 34 .866 1.0
26 .866 .75 35 .866 .75
27 .866 .50 36 .866 .50
62
Table 4.1 (continued)
System Configurations by Run Number
e = .2 n = 100 e - .1
RU# m RUN# m p
37 .500 1.0 46 .500 1.0
38 .500 .75 47 .500 .75
39 .500 .50 48 .500 .50
40 .70"7 1.0 49 .707 1.0
41 .707 .75 50 .707 .75
42 .707 .50 51 .707 .50
43 .866 1.0 52 .866 1.0
44 .866 .75 53 .866 .75
45 .866 .50 54 .866 .50
Based on the acceptance criteria described in
Chapter 3, component dependence is considered significant
when the ratio of variances is less than .64 or greater than
1.44. Table 4.2 reveals that component dependence is sig-
nificant in a majority of the systems that were modeled,
Closer examination of the table shows several trends. The
first trend has to do with the number of components in the
systoi1I, 'n'. The rows of Table 4.2 show identical system
configurations except for the number of components. The
ratios across each row tend to be clustered near the same
value. These results indicate that system size does not
significantly affect the movement of total system cost var-
63
z I N
iance away from the independent cas~e. This implies that the
number of system components should not affect the decision
to model component dependence in cost-risk assessments.
Tabl1e 4. 2
Ratio of Variances'*
VAR w 336 VAR 2339 VAR .4616
RUN Rat N btC RUN jRo g t-i
1&10 1 .71 19&28 1.81 37&46 1 .85
2&11 1,38 ý20&29 1.40 38&47 1.*43
3&12 1.03 21&30 1.00 39&48 1.00
4&13 1.82 22&31 1.92 40&49. 1.97
5&14 1.50 23A&3.2 1.46 41&50 1.49
6&15 1.03 24&33 1.00 42&51 1.00
M&6 1.90 25&34 2.01 43&52 2,06
8&17 1.54 26&35 1.50 44&53 1.53
9&18 1.03 27&36 1.00 45&54 1.00
*Because the calculations for this table were based on theInormnul'cuse, Rho Ost~imation error does not affect the ratiosfor these runs.
C.aution should be taken, however, in generalizing this
ccrIc.'u;_dIoii to aysctems with a WBS hierarchy. The conclusion
* that system size is insignificant seems to be more limited
Lo Lhe simplified systems that this research addressed. To
illustrate, consider a 30 component system whose depend-
64
encies have been simpilfied by the linear ,dependence as-
sumption. The ratio of the dependent and independent vari-
ances for this system is shown in Eq (4.1).
30SRatio - [ Var(Xi)
iIN 30 30+ 2 ICov(Xiixi) ] / 1Var(Xi) (4.1)
iw2 1.1
Suppose that, the system size is inct-eased by one component.
The new ratio of the cost variances is shown in Eq (4.2).
30 .30
Ratio *[Var(X~ 2 ECov(X i Xj) + Var(X 3 1 )
"30+ 2 Cov(X 3 0 ,x 3 1 ) J / [ }Var(Xj)
i.l
"+ Var(X 3 1 ) (4,2)
The numerator of the ratio is increassd by the variance of
the new component plus twice the coveriance between the new
component and its immediate predecessor. The denominator of
the raL~io is increased by the variance for the new comipo-
nent:.
System size has a greater affect on the ratio of the
two cost variences when the syotem is defined by a WBS
hierarchy. ."q (4.3) shows the ratio for the 30 component
syuilurn which has been enlarged by the additional component.
Comparing Eqs (4.3) Lnd (4.2) shows that adding another
component may have a significantly greater affect when there
are no constrainLt placed on the covariance terms.
"65
9 O. -kaW o "4
30 30 30Ratio = [XVar(Xi) + =: E Cov(xi.xj) + Var(X 3 1 )
30 30+ Cov(Xi X3i) + F'Cov(X 31 Xx )
30/ [ 1Var(Xi) + Var(X 3 1 ) ] (4.3)
II
The denominator of both equations increased by the same
amount, however, the numerator of Eq (4.3) may increase
significantly more than the numerator of Eq (4.2). The
numeraLor of Eq (4.3) includes not only another variance
term, but also the sum of 60 covarianca terms, as opposed to
2 additional covariance terms in Eq (4.2). This could make
a significant difference.
Since the results in Table 4.2 are not sensitive to
system size, only Column 1 of the table will be used in
order to simplify the remaining discussion. Although only
the n-5 system will be illustrated, the following trends
also apply to the systems with 50 and 100 components.
The first: trend deals with the relationship between the
proportion of positive rhos, 'p', and the mode of the tri-
angular distribution, in which produced the Rho values.
Table 4.2, Column 1 is reproduced below along with the
corresponding values of 'in' and 'p'. High proportions of
posiLive Rhos consistently produced higher deviations of
total system cost variance from the independent case. This
is evident by the high ratios for RUNL, RUN4, and RUN7 when
66
SM i \,l •
Table 4,3
m and p Values for RUNI-RUN9
Run Rar .
1 1,71 .500 1.000
2 1.38 .500 .7b
3 1.03 .500 .50
4 1.82 .707 1,00
5 1.90 .707 .75
6 1.03 .707 .50
7 1.90 .866 .1.00
8 1.54 .866 .75
9 1.03 866 .,50
As this proportion decreases, so does the ratio of vari-
ances. A decrease in 'p' from 1.0 to .75 changes the ratio
for RtIN4 from 1.8 to 1.5. The ratios for RUNs 1 and 7 also
changed by approximately .3. A reduction in the proportion
from .75 to .50 changes the ratio for RUN 5 from 1.5 to 1,0.
A similar decrease is evldent in RUNs 2 and 8,
These results can be explained by recalling that the
o only d Ifference br.tweon the d.pundent anid independent total
system cost variances i.,i the inclusion of the covariance
trms. If the sum of the covariance terms significantly
deviaLes from zero, a difference in the ratio of the two
67
variances will result. The sum of the covariance terms is
very sensitive to the proportion of those covariances which
are positive. When the pruportion is large, there are
relatively few negative covariance terms co offset the in-
crease in the total system cost variance caused by the
p ositive covariances. As shown in Table 4.3, when half of
the Rhos were positive, the results closely approximate the
independent system for all values ýof the mode used to sen-
erate ,.he Rhos,
Comparison cf RUNs (i-3), RUNs (4-6), and RUNs (7-9)
demonstrates the influence of the moda of the triangular'
distribution on, the ratio o'f-varIanc,•s. Higher value'a of
Rho conoistently produce greater deviations from "the inde-
pendent total. system cost variance. These results are
explained by t'ecatling :he relstionship3 inEq (3,19). For
component pairs and their associated variances
Cov(XiX) R RXi x Var(Xi) Var(Xj) (3.19)
The magnitude of Rho, as modeled by the mode, determines the
magnitude of the covariance terms. With other model parame-
ters hold constant, large magnitudes of Rho (+ or -) drive
the maLgnitude of the sum of the covariances. As mentioned
above, no the sum of the covariances deviates from zero, the
ratio of total system cost variances will devIate from 1,
Stepwise chnnges -in 'in' frnm .5 to .707 and from .707
to .866 increase the rutto of the varLances bet.ween runs byI 681616V ., . ?Ný,
approximately j.. For example, 'ýhe ratio for RUNs 1, 4, and
7 are 1.7, 1.8, and 1.9, re-spectively. Each of these step-
wise changes in the mode represent arp increase in total
explained variation of 25% (Reference Chapter 3, Table 1.).
For the relevant ranges of the..prco,portion of positi~ve
*Rhos and modal values chosen in th~is**.sirnultion4 total
16 'system cost variance is more sensitive to the-p~roporti~on of
positive Rhos, 'p', than to the range of nodei values.., Table.......
4.3 shows that varying 'p' fr6ni its lowes-ý to h igsh'sat v~alue
(in increments of 25%) increases the variance ratio by
approximately .8. Varying Win from it&, lowest' to highest
values (in increments of 25% of explained aito)
increases the ratio by only .2. This .result has a favorable.
imolication for the cost-risk assessment method. In-the
troa:l world', determination of the direction of 'compo .nent
dependency should be easier to-subjecti~v'ely asseoas than th'tŽ
exnct magnitude of the dependence relationship.
Internal Validity
The high and Jow overlap percentages generated by the
validation model are displayed in Appendix E. These vialues
were compared to the standards and acceptance criteria that
were rpresentad in the previous chapter. The results of the
First standard (70-60-50) are displayed in Table 4.4.
'[he p)Urpose of this section IS to determine the ability
to predict total system cost variance whenL subjective esti-
miateds for Rho squanred are in error. This parameter, 'e
69
was given two values, .2 and .1. As expected, more runs had
acceptable overlap when e-.1 than when e-.2 . This
makes sense intuitively, since one would expect a better
estimate of the total system cost variance when the value
for Rho was estimated with greateer accuracy., In Table 4.4,
Matrices 2, 4, and 6 were Senetrated when the Rho squared
error was..1 and Matrices 1, 3, and 5 were generated with
ff the error at .2. There are 18 of 27 runs that met Criteria
"#1 when e=.1 . Only, 8 of 27 runs have met this criteria
when p=.2 , As the program personnel's ability to subjec-
tively aitimate component dependence' increases,, the ability
of the cost-risk assessment method to accurately estimate
total syst'pm cost variance increases significantly.
While R2 error is the parameter used to assess internal
vnli'dity, the other model parameters influence the applica-
bility of the method. Looking again at Table 4.4, the first
column in each matrix,.except Matrix 2, shows a consistent
weakness in predictIng total system cost variance. Whether
R2 error is .1 or .2, the cost-risk assessment method loses
predirtLve capability as the proportion of positive depend-
encies approaches 1.0. The number of components in the
system also appears to affect the accuracy of the method-
ology, When 'n' is low, the predictive capabiliLy of the
rn(mt~hod is at its best. T'h1s is shown by the large number of
systenms In Matrices I and 2 that meet Criteria #1.
70
Table 4.4
Internal Validity Result Matrix
Standard: 70-60-50
---------------------------------------------------------------------
MATRIX I MATRIX 2
n-5 p nm5 p
e=.2 1.0 .75 .50 em.1 1.0 .75 .50
.500 1 2* 3 * .500 1ODe 11* 12*. o ... I- -..-oo.. .- ,
m .707 4 5-9 6* mn .707 I13* 14* 15*
.866 _ 9* .866 16* 17* 18*
K ATR.X 3 MATRIX 4 ._
n=50 p jit50 pIie=.2 1.0 .75 .50 e-.1 1.0 .75 .50
.500 19 20 .500 28 .29* 30*
in .707 2 2 3 2 11 .707 31 32* 33*
.866 25 26 "ro 27* 866 3, 35* 36/
MATRHIX 5 MA'IRIX b
11 ~rzIU 10 1)100) p
.2 1. ./5 .50 U.=. 1 .0 .75 .50
~ 7 38 1.0 4 47 487 4 0 4 4 m .707 :4J 50* 51*
.80 I'i 44 I45 . 860 5 2 53? 54*1
v-!: mt, Cr i ,, r i , !
Iit ev' t s C r i t r i a 2niot no entry Ili('dIl; the system in t, cr ic t it" ia #3
"71
However, as system size increases, the number of runs that
meet the criteria decreases in Matrices 3 and 4, and again
in ilatrices 5 and 6. Finally, the strength of the component
dependencies, as described by the mode, appears to have a
limited efiect on predicting total system cost variance.' As
'mi increases fror' .5 to .866, in all six matrices, slightly
more runs meet Criteria #1.
Table *4.4 reveals that onily, a limited number of systems
met Criteiia #2. So few, in fact, that li.ttle information
regarding trends in the performance of the method can be
ascertained.
Criteria #3 shows the systems for which the method has
limited ability to accurately predict total system cost
variance. Since so few systems met the second criteria,
systems described by Criteria #3 are basically the reverse
of those described by Criteria #1. The same trends pre-
sented in the previous discussion hold true and will not be
reiterated.
The results of the comparison of the 54 systems to th-
s e,(cond standard are shown in rable 4.5. Notice that the
trends mentioned in the previous two paragraphs hold true.
hHs consistency adds to the reliability of the conclusions.
One notewor thy occurrence is the drast i cnc rease i n the
.Utnumber of systems that meet Criteria #1 for Matrices 3 and
0. A lowering of the ,standard tripled the number of systems
thaLt 11,1L the first criL riena,
S... • -i ....... r - - • • u n n u n
Table 4.5
Internal Validity Result Matrix
Standard: 50-40-30
MATRIX 1 MATRIX 2'
n-5" p n-5 p
em.2 1.0 .75 .50 em.1 1.0 .75 .50
.500 j* 2* 3* .500 1.0* 11*. 12.*
, .707 4** 6* m .707 [.3 14* 15*
.866 7* 8* 9. .866 16* 17* 18*
MATRIX 3 MATRIX 4
n=50 p n-50 p
c=.2 1.0 .75 .50 e-.1 1.0 .75 .50
.500 19 20* 21o*f .500 2 8 29* 3Q *S.. .... .... . .. .. ..in .707 22 23* 24* in .707 31*4 32* 33*
.866 25 2 7 .866 34 1 35 36*
MATRIX 5 MATRIX 6
n -1 0(. p n-100 p
1.0 .75 .50 u-.1 1.0 .75 .50
.500 37 B 39 .500 46 47* 48*
v 7[7 40 I41 4 2* "'i .707 49 50* 51*
IX.866 43~ 44* 4* .866 532 53* 54*- * mf(ets Cri teoriu #1
*.* m,,ct Cr i t c-r Ia #2wIiLc: no onlry im ins thu 8 syst.em meet.s criteria #3
7 '3
. ............ .... .... . . ............. -....................-............
ji sj) i t( o Ich r lie ncrensz? J n the number of syst ems frieetIng
Ci iteria #1 in Matrices 3 and 6, no systems meet the cri-
teria for Column #1 in thebe mnatrices when the proportion of
positive Rhos was 1.0. This reinforces the earlier point
that the cost-risk assessment method has poor predictive
capabilities when p-.1
Consistent with Table 4.4, Table 4.5 has only two
systems meeting Criteria #2. The minimal number of systems
meeting this criteria is significant. Since so few systems
meet this criteria of moderate predictive ability, one con-
clusion is that the method tends to have either very strong
or very limited predictive capabilities. This is true even
when the two standards used to define acceptable overlap
were significantly different. This conclusion is over-
whelmingly supported by the data.
In general, the cost-risk assessment method, incorpo-.1
r-t ing component cost dependence, predicts total system cost
variance with relatively good accuracy when the error in Rho
squared is .1. fL does not do as well when the error is .2.
Values of 'e' grrater than .2 would indicate that R2 values
, rL mi'-estimatir.g more than 20% of the total system varia-
ionu. To estimate total system cost variance when R2 error
ji grenter than .2 would be accepting entirely too large an
error into the methedology. When the percentage of positive
cuvarilance values upprouches 1.0, the method consistently
* predlct.; poorly. However, systems with all positive cost
dv,pnd(,rln ,es are highly unlikely. In Lhe researcher's
74
opinion, a much more likely scenario would have 50% to 75%
positive cost dopendencies. With this situation, the pre-
dictive capabilities of the cost-risk assessment method are
much better. Another expected result is the effect of
system size on the predictive capabilities of the method.
As 'n' increases, the estimation error for R2 is compounded
* and estimates of total system cost variance are further
distorLed. The mode value influences total system cost
variance in small amounts. The higher the value, the higher
the error in the estimate of total system cost variance.
Finally, the lack of system configurations that fell into
the second criteria for both standards indicates that thereI, are clearly situations in which the method has strong pre-dictive abilities and situations in which the method has
weak predictive capabilities. This result enhances the
value of the method because it defines the situations where
the method can be used with confidence. The pronounced
division in the number of systems meeting Criterion #1 and
#2 may be caused by the rather 'crude' incrementing of the
modet parameters chosen for the sensitivity analysis.
ModelLing smaller incremental. changes in these parameters
may identify more siLuaLions with moderate predictive cap-
ablity. it is interesting to note that the method may even
0 be iusa ble in situations where Jt has poor predictive caps-
hi t iit8es. As the proportion of posi tive covariance terms
%,• iJIpproa¢:ch,, 1.0, the 1nethod predicts total system cost vari-
ance wV . h less ac-curaccy. 'The reos l, Ls from the r tlo of the
75
IAbZ-
* variances indicaLe Lh•it the incorporation of component de-
pedency becomes most important when the proportion of pos-
itive component dependencies approaches 1.0. The method is
needed the most when it is least effective. However, even
though the methods predictive abilities are not at t'heir
best, the method is still accounting for an increase in the
total system cost variance that is not accounted for when
independence is assumed.
76
V. Conclusions and Recommendations
Conclusions
The results from this research indicate that component
cost dependency should be an important consideration in the
* cost-risk assessment process. The simulation revealed that
for the majority of the system configurations, component
dependency caused the total system cost variance to signif-
icantly deviate from its value under conditions of component
independence. This finding indicates that weapon system
acquisitions that assume component independence are likely
to significantly misestimate the total system cost varience.
The impact is the potential misallocation of scarce
resources.
There were several trends that became evident when
comparing dependence against independence using the cost-
risk assessment method developed in this research. The
proportion of the positive cost dependencies in a system
clear.ly had the stongeust influence over the difference be-
tween the dependent and independent system configurations.
The greater the proportion of positive dopendencies in the
system, the greater the difference between the total. system
-.cost variances. As the strength of the cost dependencies
was Lticreased the, difrerence beLween the independent and
Sdependent: system variances increased. llowever, the incrense
was siial. i relative to that i.ncrease which occurred when the
*77
1% ýý41
proportion of positive cost dependencies was increased,
Changes in the size of the system did not appear to affect
the ratio of the dependent and independent variances. How-
ever, the researchers believe this result is due to the
limiting assumption of sequential component dependencies.
As this restriction is relaxed, the number of component
dependency pairings increases and could significa~ntly affect
the ratio of the variances, Sensitivity analysis was per-
formed on the method to evaluate its ability to predict
total system cost variance under conditions of Rho squared
estimation error. Results show that the method predicts
total system cost variance fairly accurately whe R2 estima-
tion error was limited to .1. As the R2 estimation error
grew to .2, the method did not predict as well. The
researchers consider a .1 error in estimating R2 to be very
good, yet even this small amount of error does not guarantee
prediction of the total system cost variance. When the
number of components in the system or the proportion of the
positive cost dependencies is large, the predictive ability
of the cost-risk assessment method is not very accurate.
However, these inaccuracies may be tolerable when the conse-
quence of ignoring component dependence is considered. The
researchers found that those situations where the variance
was difficult to predict were the situations with the
greatest need to incorporate dependence. The potential
impact-t of each error would hayve to be weighed.
78
Limitatlons
The research was limited in several ways. Although
subjective easessments of component dependence will always
include some degree of error, the technique ueed to quantify
these assessments will introduce additional error. This is
Sparticularly true for tho 'descriptive category' technique
described in Chapter 3. The analyst has wide latitude in
as.igning Rho values to'each category.
The simulation only test'ed a, limited number' of system
configurations. Many scenarios that occur in the real world
were left out. In particular, wider ranges and more para-
meter settings are needed to test the effects of negative
component dependence, a larger range of rho values, and the
more system sizes.
The manner in which the Rho values were generated is
also a weakness in the research. Both positive and negative
Rho values were generated from the same distribution. This
assumes that the magnitudes of both the positive and nega-
tive Rho values have an equal likelihood of occurrence. In
vi real system, this sLtuation is unlikely. Ruther, strong
positive relationships may be more likely to be accompanied
with weak negative ones in the system.
NeCC innen dat ions
TRien a nuthors fevl that further research uhould be con-
ducLte• with t1hi ' cost-risk assOssment methodology. The
fol lowing iireas are recommended:
V 79
S '• •"•''. '"• ...... .;" "' • '•"i-s'' 4- ý, - " 4i
1. Sensitivity analysis should be performed to examine
the scenarios that were not examined here. A clearer iden-
tification of those situations with moderate prediction
capability is needed.
2. Better techniques to solicit and convert subjective
assessments into measures of dependenc-y would improve the
methodology, The use of conditional probabilities is one
possible approach to consider.
3, Extension of this methodology should be applied to
a WBS hierarchy to compare the trends %hich were discussed
in this thesis.
80
". *"(.' . --4* '* '
Appendix A: Calculation of Sample Size
1Banks and Carson describe a procedure for determining
the required number of simulation repetitions. A t:wo-sided,
confidence interval for the mean of a distribution, l, based
on the Students t distribution is given by " "
S± [(t) (s / r• )3 (A. )' ,.: ..
where
t - critical value for t distributions - sample standard deviation of xr - number of simulation repetitions
The precision of the confidence interval is specified so
that it estimates the true mean within a tolerable error,
+t , and with a desired level of confidonce. Therefore,
from Eq (A.I)
( > ( s) ( / / r•1 (A.2)
The number of repetltions required to achieve an estimate of
the mean within +e accuracy is. determined by choosing an
initial number of repetiLions. rrhe required number of repe-
LitiolIs is then sol 'vvd by manipulating Eq (A.2) as follows:
"* > (t) (s / r() (A.2)SrA t s / (A.3)
A. I
- A,'U
r 2 [(u) ( / *)]2
The number of repetitions is calculated using the standard
deviation frrdin the inital sample, the critical t value, and
the desired accurac'y (1:427). Generally, a few iterations
o.f Eq (A.4) are required in order to adjust the t value as
.the degrees of freedom change with the number of repetitions
thdt iS calu~lat'ed (1:426-427).
the authors selected 50 initial vepetitions for the
five component system and 100 for the f.ifty and one hundred
,componentsystems. *A 95% confidence level seemed reasonable
and the-amount .of tolerable error was subjectively estab-
'ished.at 50, roughly half the'sample standard deviation for
the five component system (s-ll0). The mode for the tri-
angular distribution, the proportion of positive rhos, and
the rho squared estimation error were set to: m-.707
p=.75 , and em.1 . The 'normal' case was used as the
basis for the calculations.
For the 5 component system, solving Eq (A.4) revealed
that 20 repetitions were sufficient.
r e> [(t) (s / 2)] (A.4)
r >. [(2.02) (110 / 50))2
r > 20
Hlowevcr , 50 repetitions were selected fur the 5 component
S~Y-Luu1 in order to provide a smoother histogram. From Eq
A A.2
*1 , . l> %t• • ,. " % '•-•• ,• % •, ,•" ,,".• 4 - • ' • m •v,,*,,..•.• % • ,", .% .. N ., ., . ., .4• *• . . . . ."" "'"•**"••"• , '"•
(A.2), the error for the 5 component system is reduced from
50 to 31 when r-50 . The sample standard deviations for
the 50 and 100 comnonent systems were 373 and 545, respec-
tively. Applying the above procedure, the calculated sample
sizos were 213 and 456 for n-50 and n100 , respec-
tively.
As discussed in Banks and Carson, the model is ru.n with
this number of repetitions and adjustments are made for the
new ssmple standard deviation (1:427). The results of these
calculations yielded r=229 and r1 393 . These values
were rounded. The number of repetitions used in thi-s re-
search is shown in Table A.l.
Table A.l
Simulation Replications
number of number ofsystem simulation
components repetitions
5 50
50 230
100 400
A. 3
Appenndix B: Simulation Program
The following section of the Fortran code declares these
real and integer variables and arrays:
ARRAY PURPOSE
var Stores component variances
sign Stores ran4om numbers. (unifo-rmly distributed 0-1)used to determine the' sign of rho. values
numb Stores random numbers (uniformly distributed 0..1)
used to calculate magnit.ude.of rho-values
rl Storog low Rho for component pairs
".rn -....Stores normal-Rho for component pairs
rh Stores'high Rho for component pairs
covl Stores 9cqyariance computed with low Rho
covn .!'St-bres covariance computed with normal Rho.
covh Stores covariance computed with high Rho
seeds Stores random numbers' (uniformly distibuted 0-1)to be used .'as random number stream seeds
"VARIABLE IDENTITY
N covsl Sum of low covariances
covsn Sum of normal covariancos
covsh Sum of h .gh covariances
v vatrs Sum of component variances
Stsvl Total system vartanco-Iow
'V . tS V1l Total system varIlancc-normni:
L. ts vh Tota . system variance-high
Jix Initial random numbe r st reun, seed
H. I
iy Dummy variable for random number generation
yfl Random number generated by subroutine
ayfl Dummy variable used for arithmetic operations onrandom number
m Mode of triangular distribution of rho values
e Rho estimation error
p Proportion of positive rho values
n Number of system components
r Number of simulation repetitions (Appendix A)
rnsq Rho-normal squared
i Counter
k Counter
c program l.freal var(l00) ,sign(99),numb(99),rl(99), rn(99),rh(99),
*covl(99),corn(99),covh(99),seeds(400),covslcovsn,*covsh,vars,yfl,tsvl,tsvn,tsvh,m,ayfl,rnsq,e,pinteger ix,iy,n,i,k,r
The following segment opens a file where the output of the
simulation is stored and sets the system parameters as
described in Chapter 3. Files outl-out54 were used.
open (6,file='out37')rewind 6
Sp"l 1 .0e=,2
•'• -in= . 5
r=400
This nection calls the random number generator (subroutine
B. 2
*l -f ,l i I j ,. tl I I I I
RANDU) and initiulizes the random number stream seed.
iy-Ol x-836 47
call randu(ix,iy,yfl)
This loop calls the RANDU subroutbie to generate random
numbers that will be converted to a different 5 digit inte-
ger random number 6tream seed for each simulation repeti-
tion. This is to ensure that the simulation results are not
autocorrelated (1:391).
do 20 i..1,rix-iycall randu(ix,iy,yfl)ayflwint(yfl*100000)seeds(i)-ayfl
20 co nt inuite
This loop calls the subroutine to generate random numbers
that are used for component variance values (0-100). These
variances are held constant for all iterations of a given
system and are identical for all. simulations of equal
component size.
(10 30 i-1,ntx-iycall ranidu~ix ,iy,yfl)ayf1!-yf*1*0Ovar( i)-ayf 1
30 continue
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
B.3
I'l II
This section begins the loop for iterations of the defined
system and initializes the sum of variances, covariances, and
total system variance to 0.
do 1000 k-l,rvars-O.0covsl=0.0covOn-0.0covsnMO. 0tsvl-0.0tsvn-0O.Otsvh-0.0
This segment initializes the random number stream to a new
value for each repetition.
iy-Oix-seeds(k)yfl0O.0call randu(ix,iy,yfl)
This loop loads an array with random numbers for each compo-
nent pair, These numibers are also non-autocorrelated and
will determine which component pairs have negative correla-
tion,
do 40 i=l,(n-[)ix = i ycall randu(ix,iy,yfl)sign( i)-yfi
40 continue
T'hu f oL nwllag ) loopI loads an array with random numbcrs for
j 13.4
,,VA
cachi component pair. These numbers will be used to calcu-
late non-aut~ocorrelated Rho values based on a triangular
di st rib ut i.on.
do 50 i-l,(n-l)ixmiycall randu(ix*,iy,yfl)numb(i)-yfl
50 continue
This section transfornms the random numbers in array 'numb'
into a Rho value for each component pair. Banks and Carson
use the cumulative distribution function (c.d.f.) to
transform the random number into a random variate of the
triangular distribution. The c.d.f. for a triangular
di'stribution on the interval (0,1) is:
01 0< 0~F~xx 2 Mj [l(0)]/(-n, n < x <
where
m - mode of triangular distributionx - rho value on the distribution
hq(13.1) is then set equal to the random number, 'ub o
each interval of the random variate. For 0 < x m i:
[ILMb) X 2 M B (1.2)
For m < x < 1:
numb - [1- (1-x) 2 (1-m) (B.3)
Therefore, Rho values (x) in the interval 0 < x < m implies
S* that 0 < numb <i m. In this case, the Rho value for ,random
numbers between 0 and m are calculated by rearrang~ng
Eq (B.2) as follows:
x 2 - (m) (numb) (B.4)
x - [(m) (numb)ji (B.5)
Rho values (x) in the interval m < x 1 1 implies that
m < numb < 1. The Rho value for random numbers between m
and 1 are calculated from Eq (B.3) as follows:
(I-x) 2 / (1-in) 1 1- numb (B.6)
(0-x)2 m (1-m) (1-numb) (B.7)
(1-x) - [(1-m) (1-numb)] i (B.8)
-x =-1 + [(1-m) (1-numb)]i (B.9)
x - 1- [(I-rn) (1-numb)] l (B.10)
A
To sunmmarize the transformation:
S[(0) (numb)I r < ntub m< m
X " .- f(1-m) (1-numb)] , m < numb < 1 (B. ii)
(1: 158,299-300)
B.6
[ ~ wrd1T~xu'w'x~WUW1WW J IN3V 6rW~bU WUWUW!WIb. I1 A %.FAdN UWWN WýW .1 6W V ý Uy 4. ý
do 60 i-l,(n-1)if(numb(i) .ge. 0.0 .and. numb(i) .le. m) then
rn( i )=sqrt(m*numb(i))else
60 continue
This loop changes the Rho to a negative value for component
pairs when 'sign' random number is greater that the identi-
fied proportion of positive Rhos.
do 70 il,(n-1)if (sign(i) .gt. p) then
rn(i)--rn(i)end if
70 continue
The next loop checks the Rhq values and assigns upper and
lower Rho limits based on the sign of rho and the amount of
Rho squared estimation error. The Rho limit which produces
the lower total system variance is assigned to array 'rl',
the other limit is assigned to array 'rh'. Rho limits that
are outside the interval (0,1) are truncated 'to the appro-
priate limit, 0 or 1. An example of this procedure is given
in Chapter 3,
do 80 i-l,(n-1)if (rn(i) .ge. 0.0) then
rnsq-rn(:L )*2if (0.0 .gL. (rnsq-e/2)) then
r l (i ) -0.0rh (1 ).sqrt (rnsq+e/2)
else if ((rnsq+e/2) .gt. 1.0) thenrl(i)-sqrt(rnsq-c/2)rh( I I1 .0
B. 7
elserl(i)± sqrt(rnsq-e/2)rh( i )msqrt(rnsq~e/2)
end ifelse if (rn(i) .1t. 0.0) then
rnsq=rn( i)* *2if (0.0 .gt. (rnsq-e/2)) then
rh(I )-0.0rl(i)--sqr~t(rnsq+e/2)
* else if ((rnsq+e/2) .gt. 1.0) thenrh(i)--sqrt(rnsq-e/2)
elerl(i)u-1 .0elserh(i)--sqrt(rnsq-e/2)rl(i )--sqrt(rnsq+e/2)
end ifend if
80 continue
The following section uses the normal, low, and high rho
values for each component pair and their associated vari-
ances to calculate the normal, low, and high covariance
values.
do 90 i-l,(n-1)covl(i)-rl(i)isqrt(var(i)*var(i+l))covn(i)=rn(i)4*sqrt(var(i)*var(i+l))covh(i)-rh(i)*sqrt(var(i)*var(i+l))
90 continue
This loop sums the variances for all system components.
do 100 i-l,nvars-vars+var(i)
1.00 continue
This loop sums the normal, low, and high covariances for all
component pairs.
B.8
do 110 i-1,(n-1)
covsl-covsl+covl(i)covsn+covsn+covn(i)covsh-covsh+covh(i)
110 continue
a--
This segment calculates the normal, low, and high total
system variance. Covariance sums are multiplied by 2 as
discussed in Chapter 3. The output is written to the file
"identified in the OPEN statement at the beginning of the
program. Program execution loops back for another repeti-
tion of the same system until r repetitions have been per-
formed.
tsvlwvars+2*covsltavn-vars+2 *covsntsvh-vars+24*covsh
write(6,7)tsvl,tsvn,tsvh7 format(3(f6.0,x))
1000 continue
After the final repetition is complete, the output file is
closed and program execution halts.
close (6)ond
The RANDU subroutine generates all random numbers required by
Sthe simulation.
"B.9
subroutine randu(ix,iy,yfl)iymix*65539if (iy)5,6,6
5 iy=iy+21 4 748364 7 +l"6 yfl-iy
yfl-yfl*.4656613e-9returnend
1. 10
L k' '-U
Appendix C: BMDP Programs
The University of California at Los Angeles (UCLA) BMDP
statistical package was used to provide the histograms and
the sorting procedure that aided the data analysis (6:43,
74-78, 112-113, 124-132). The following programs were used
to perform their respective functions.
Program 1: Histograms
The variable names tsvl, tsvn, and tsvh correspond to
those used in the simulation program discussed in
Appendix B.
------------------------------------------- -------------------------/problem title is 'run54'.
The problem paragraph states the title of the BMDP
execution program. In this case the name of the program is
run54.
/input variables are 3.format is '(3(f6.O,x))'.
Thi. portion of the program, the input paragraph, states the
number and format of varia~bles to be read in as data.
M'----------------------------------------------------------------/vorI1bl' names are tsvltLsvn,tsvh.
blankti tire zero.
The varJu.ble paragraph names the variables so that they
C. I
'S. ,,• •,:,.. ,,•,..'.•, :, ,:• .,, > ,.,• •.',• .;: - . .'',''.,,..c"• :•',.>#,',. ''. •,' ' .: ...: ,
may be identified in the output. The second line of code
makes the package read all blank spaces in the data as
zeroes.
/group cutpoints (1) - 2300 to 5800 by 175.cutpoints (2) a 2650 to 6150 by 175.
* cutpoints (3) - 3175 to 6675 by 175.
The group paragraph sets the intervals for the
histograms. (1) represents tsvl, (2) represents tsvn, and
(3) represents tsvh. The high and low values were
determined by the sort program to be discussed below. A
value consistent within each run is used to specify the
width of each interval. In the case of run54, 175 is the
interval width.
/plot type is hist.scale is 0,3.
The plot paragraph specifies that a histogram will be
1l0 t t.d , The second line indicates that zero is the low
value on the histograms and that each 'X' on the histogram
represents three data points.
This ends the program. After this itatement, the data
is 1i.,•ted in the format described in the input. paragraph.
C .24,,'
Program 2: Sorting
This program is the same as the histogram program with some
changes. The problem and variable paragraphs remain
the same. The following code is added to the input
paragraph.
/input variables are 3.format is '(3(f6.0,x))'.sort is tsvn, tsvl, tsvh.
The sort statement has the tsvn variable name first to
give it precedence in the sort since it is the critical data
column.
Instead of a plot paragraph, a print paragraph was used
to display the sorted data.
/print data.order
C.3
%'
~~' ~~ a. .~Z oý 4i+';Y m:0 I.
Appeýndix D: Histograms for RUNI-RUN54
rTh's Appendix contains the BMDP-generated histograms
for RUN1-RUN54. The histograms show the 'low', 'normal
Sand 'high' distributions of total system cost variance for
each RUN number
n).
_.,***i * ~ * - '. ~~ '.7 . ' t ~ ~ '''"*ff.- t ;,'**f** 7.*f* • '~ t~: ~'"
&ZLL. )
IQ+
co I I
in .+ in
+ +
II
+ +
z la
dt ) I I L
+ I1I In
*4 % insI-> I
W I
zo mI I
in I I
CA n I I c
ca w II
o to MI +i
w I
m cI
Of I I4K I. I 1
> >- Ix xx IIA xxx+x
* I NXX XX4) =x x x x x x x I I
.. . . .. . . .. . . .
N > I ptýt 4 QNtQk Rt Q t
I If
w uCyW-. t (N-.a R - -W fn W -0 (
+ +
N I I IN* I
al + + fti I IA 4. 1%
InI
> I
in Ln Ift 0
In in
I, w
EN w IV)1 II
I I
IL I I eCC ZA I IU w I I
> n I I0 Y I
so ~ II
> II
n in +in
41 i In N
> I x xx
I xx x x x xI
- wx x Ix xx ICx .- Jxx N.x x x
to t1 IR -Rt R0I Dt 0I 00- a00 tII K o4 Rmt Ra Rt io6 ot i ka%"%a ot
Z. w XXXX -VXX -0 1 nI nI n4 oi n% D at I1.~~~ a a 0X X XX X 41
o I XX XX XXX XXX XXXD.3
UX %-Manw-oua~
I.-
at
z I
LaiI
K to
lu w
CLa en. Is.o aA Iza I I
P 4 in a
La II
W- x II~
I axx x x x
D. a
#A I N'Il90'Mff% '~
uwu
aa
WIt
I I
x IIx x
o - j~ INORs IRRQolt Qo olo;1.- 10, 1 I lp 4,1, 1 111
*D 5
A. A q , .;q j A1.
UK -----a O NNMMM VMN
wo
af z
L&a W
co I Ia4
+ +
In a +p.
wn $to1
*az
LaJD I
an + in
0 n ý- fi.0 a
a a,4 n *aW a
K'aoV
a 49a>
U w a+
(fl aC a I*
3~a~
0 NI INxxxxxxx
0 af
to NI N w Pt(4v tNv
w aD. 6
zm - - - - ~- N CY m wl -0 ni) in
co a
in + + In
I'D
* In
.0 In
Z4P3- .1
L W I
to +
10 , ,i
8- NccA
In I x Iw I x -
o I IN xx x
*#
lb ft. IV m4,10 1 . . 411 ',.am No I00 aCdsaf oode 4
10 %0 +10
v. I- ý vw(
% %
zI M
m a
+ In
"I z
> !o
CA a
-K +
>0 a aVWI-
mi a6a
Ln +,
a aZw-. a4 111 . a
x x Wa kaIe a
Nx a aS. *5 ;KX x
I- ax xx x ax xx x x
w
a) a. 9 4a a : ev
L.D.8
'U Z 4w0 --- N N"m m m 0 P,
CK Z
+ +.
N II CD
I %D
i*I IQ + t
I I 0
z kI.c 'n.i
z44 00 5
0 z mI I Cul I I
IAI
x at
~44no N
00 I>X
3 ao
x In
> X. W XI
4% M0 'U% 21ftt
tM >U fl I ?f ýw t4 taýM9 UQf %4 S4
I. x Ia C ,0ýa l ,0C,-
. 4 Ulf,
z of 0 off*
LI.I X XXX X~XXXXD.9
0*4
WI-In
Z to
'n P. .A W
z fn
* IL 1
ca IK l a>
aj
aa I
L" I
1 I. 01
ax x>~~ ax x~~LflI ifx a
or tQM II
GK .4 .n ... .
£~D F.
. ~ ~ ~ ~ i .. . .
N N NMN M M VV 0 " l I
aI z
*+
I-
MIAb
wova I
1 ,In
x IK
(A 1J IPL
SN I t'.
D. 11
SM
0 0
la z
Is I II
I L. . I
LA ~
0 LM S 0
0~ rn.
ulf xInx x
of1 4 4 *1e
-I I
"- NN qqllqr q U
Ij i
* In
in I I If
u wI(A- &h iwI
~+ L
w
00 NIII I
) Id~x
U I x I-xxxx
Im 44 in NIt K1 0I NI 0t ItI eaO o,44t QI Im
I*I 100444406w Km 44 a 1tot 0IIQolk I Q ot INV
wwm w w00
D. 13
4WV
"--NN~MMV'*w&Awqj
I P%
inI
laI
I) +
wI 14s in I in
INCII0 I
40 ~
wN I
win*jI mg
IL IVII
* at 11j0 In4+I
N I IN
ininIn IV
x I I
x x x 4in
al o o m It
0 9 . . . . . ..I* j w inIa000 0 uI n0
o NO NNI
bw
'ID 14I
WXl ft-.. -.•N N*WWN/ M f N V Oll•N ON• VMq . NOVAlI~
w U
0*
WI
..* - I/l
*" l I .I l
6• 1 1 1
*o + + I
sm •),• I I
"I I
I. .. ,I,
WE 1 1i
I It
de14+W 1,a
to I
w i0++
.I w
-tx IXI I I
•; ,x x xxxxx xxxxxxxx~x+÷+ • e + +÷ ++ ++ 4 + ...... * *+e
I %I t I" tR to I I
ar1 t. Q5
I.- W Li nI o I lpL Nw2L to1 rt ):L p0V
~D 15A
WI= ~- N 4 - -- NNMMVV
In 4
r~IN,
W) I
I I •:
ip
go I I
1 *0 I
W,÷ ,1 ÷,
In I !
I I
Ir
x m I,"
S I.
.
*NK I I
so I I
• NIaIt I
SA~
goA I.
I X I
WX X inx
Sx xx x xxx=++ ..............
l 44 0 I III 11 4I N to10 tI0114it i ii
l'-, D.L'14 n 1 6t n I Ni IoI *L
41.5
D 1
"I I-v,
w U
IA I
U) +
I II
IA4
Ix I
A.++ .. .. . ... .. .
oil 14 I k i lI I oI % ta o % k1vI
W W I- t WN Nr NN NO 4NN
ElD.17
UK i
wu
In 4OA I
In,
In I
I I 4
BA6
x Im
IN"me
u I
x Ix o: II~
to 14 IN I. nI N1 nI N4 t4 NI
. . . . . 4 . .
I. D.I18
go 4- B
~ux
0.
99 a
1.0
Ln I I i
WI
(A.14
c j 4I in
+ +
= I a
V, U4 4W ;iW;1 4 6.
D 19
1) WuM
In
iin -
inn +
M~ I
W In 4 +inly 1I IN'
uw I
I
u I
N I- I +
xi-In+
.. . .I
WW~
D. 20
W'-
atm
IIa
fanw I0
MI
x m 111
P. 4.
j 1W, *1
W OWI Nv
D. 2
5 w voNN NMM4 4.0 qNf
In inr'..
SF•*l W"Ni*=N•'t' 0I~ l l I
*qV)- ' I',
in 4
I4 #
I--. Iw I
"4-c W)I in
in , 0 0 In
N I N
UIISI
w w
:0 44 il m ,
o toF % %P
D.22'•----'10 410I t t 4= l 4 n I 1 t4 g • l i4 l
h 1. 04 - 4- 4 *N VMN W 4IMU
InI
o I
"I I
• N I IN
LU
0 In
do w I I00 C
in + + In
I jo 1
. , . . . . , *l * * * * *e, *s*11 . aII t
~M OU xxOC eeoc l ceecos *S I
2
• D. 23
UZ ~~C~aNq N~CJ MWqU In
IAt
41+1 d
Winl~II !i
*-s maV 4 x
IN it IN
0( a ;L 6 4 1a 4 ,N 0 nL ' k ati
0 0a 00 0 l 0 0 00 4 6 0 Ia
D.2
&*X. skýl 5st,ý
w u
WI-~ N M. M 4N0E M MW N ~NP. iM .
41
+ in
to I
II II. I
* I I
Ii 1
44 x
in In
.64- In~~1
Its4 W 110v lmI VVI 4 4 4004 oI %
D. 25
1". Il
#A I
In. 44
0415
N NX~oxg*4x xxxxx ,.
+' +M + .44+4 +
a'2 6
wu
at
* !TII
S 'I I
* I II I
SI I
I A t
I i
c I
L, + + In
~o II I
4 4 1 4
I III I I I I I
.. 2
.. ,.'
at ImmI
0 Ira
R)I
+K +
14
I inInI In
14 1
In I
IA. 4 I
x ad II
U) I I If
i nJ xI In
I guI
wI I x xIx I xxxwxxxxxoxx1
INt % ot69 4 NV o1 1 o1 1
. .I- I nb ,I nI nL 0L %i o0i
IIN N N N N N V N N O O
lM 0P 0" N M V 0 0WWWWP ,f0 I 10aa40a0aaa06004
D. 2I
wux
a*
In + 4.i
CA 0.
Bw I
InB
(A
o B
we L
B + B
WW
U to+B
to I-,. iBc B
B BxB Bx
InB .5 xA x CL Iox xNB
* BxxB x Bxxxx
Xx Bxxxxxx
to ta tot 4meI&P 4 qI &l ottt
a a B 0a104
- MiDB2B
*~ 10,e
WIIn
oI~'
I IN
mewI
x m
cI
0 M
WW
In +a oax x inw axxo
MJW ItIf
4 0 1a04
D. 3J
Mwu
m l + . ....
i n in•
I Ii • o f.
. .I I •.! 1
.I I
*l S•I,, I
,, ' I I
I I
r+ In
w I 14I I
f,~I
p; oc I#I I II I IIIIII II i I
I,3I
I +
Iin f
io fJ
I I0 4 * L n w n I 1 1 o W w w 1
I IV
W1ý14
in i
W) I
In.AC ( I
x 41
I..
x I
IA %
0. ~ ~ D 32I I
W-I .
slwb4,
S|,I
SB B
II......
I B.
, I ֥ x x Iti I
B I I I il I I I I I .l4D3
411
ww
#A i 41 +x inu
In 4
in x x ox in1& inn~UI I- &aIFN Ilfi qxxIxx i N O I~~U
K~ ~ NAM N M 4UU'*
D. 33
wu
iftKB I
SIn
Sm 1Sfin
Lo I
Sin
C" It
Si,
1 50gNBx IN
*xoxx No Sxx.SJ 2 5x
%No% %NS N4lt
"Iwall
60 1 -N f m f4 t I ,M w a
3c S
D. 34
a LI In
*n I k
N 1Pm.
II
Wi in
x IN
In +
IN
o4j x I
x -ox Wx I
j IR toI RI *0 I ot m1
D. 35
,.lR '' IM
wK h O NOP
iin
""4 I
to
m 1 IM
in
I II
lN I
I In
x 1IIx
D. 36
-'N,11110~'S
'Iu
a.M
61J 11
in.
Is In
WII
II I
I.I
(A1 ~ D 37 4
O NI IN*ý
U--------------
MIu
OWI VOP
I #AI ., I
I
I I I
I II
" •II" i
i #"III ,P-4
I I
No M IN M
hD 38B
-U * seeutlitlt sotill t tisess
D. 38
IIuwu
N N
4 I1
Ln I Mg Ow M pN t0 p 29 llvA
~~D ¶31
wu Zý
t a
I I I
I InI I
we
V) M-N1 Iw . %Nt
D. 40
=4*Ot t 211
LU NN wqqmwuv NM.wow
ito
A In
I I
inc + i
InIhO II
I II#A 4 In
ly
4 Ix
* I.
AN~~jNNNN%41; -0 * 00 n2N, tN
CCC
D.41
fU- , mý
0*
L a II
OD InI
4D 0
x 0
p 0 a I
Din W + +It
0 11) m
I II
-j +n + in
x I I '
+ +*4
I I..
+~i a x ixu~ in x x
0:0 a tof 4
o ~ ~ D 42I
wu
ob I I fI
W I)U) I L
Ni IN
I AnI
Lh + + Inl
w~n
*.in r In 4 in~t
I- w
U+ I
x VI II
o o
U
WW IN+4 I
Oxx NIx IN
w Ilk 411
4D.4
LLZ
I-p
4.-
to II
5 WI I in.me I
Ij II In
C4 1Im +
in W + 4 i
0 z MI IM
Nk IN
'A N
Sin ++i
I xs
oo z01 1 II W I
II xSW I x L
>I Ix
lux xxxxxixxxx I
x OIM%4 - MM *tW~A -
01 004 149
ID.4
'V V 0 i6A L.
gem~tNu u~lf
W r NNV tN NI
L6..
*+
WNW
In Inin
w
I II
x I:
to~r I
I tI
.4 1i .4
C a. Mx xLU, I x
ol I X X X X X X K X .;
q( "'"' I It tt k k,11
fa
D.4 5
U. "4
+ In
In4.
Ln II 1
LO + + ln
IIs
C A -J In+ 00 ly WI
*1 mm
i- I* +A ca I
N. it II
x IVZN t +Ix x+
Ix I
VIxx IVx
CA Bo BOW M %W B
04 aB tIIaa 04
U~ ~~D 46 A.*U
wu
le it
I+IQ0' o f I
w I
+ In
I I
+ inWII
W I It
x IV#A W 4 i
+.u
Ln'
%0 fl- Ii
wx xx x I
me a 4b o4
F, t I 0N NW.0 I* IwoWWWONK~
D. 47
+ in
I4 I
In *
NisI
LA + 4 IA
.j l- IN +
I Ix ad
KW.- I411
c -1
C I INoxox
~I I-XIX M xx 1
++ + + + ++ . . . . .I zI4 tox ONI ar 4t R14t Ra ot
I ~ tM X .. . .. . . .. .Cw Im c ~ ~ XX~ ImL XK oL I ntL NL 0t n L o(
D. 48
Ito
UK -uwww~
•+ +•
cm +
MJ 0
WI I.
w I I
*VI IN
I
+ +
I IXXX X
in11 1O
w-
D.4
w
x09
l-II I I I II I I I I I
I I1
1* x
xa xI I i
YI~9 04 04 +1
11 IxI -x x
w in 011 W)W W W in Nn k to W N 1% K cc IW -j
D. 49
A~ wu
a.i
ka I
InoI It
P.+ IA
Ed Iw II
w104 +U0 W)I In
Ng I
in P- I
rn+
M 1 IM
0 NI
x me I
LI IIN
1%0- 'I+
Cn + 1 i
+U Im J
xX~C x ICn I + L"~X~( ~ XX XX++++444+ +4+4 444+44 1
U~ ~~ l~ *.* I i.*.
x flS-' xoI0 i N x OXXXX5
+* +4.. . ..D1A5O'c++ +
0 44NNNMMWVWM0W-
0+
Inin
kfI 4"IA I
I3 N.. 4 in1 1
ORO I
Wp I I
K" w
(A I
(Aj I
V9 btx %w ,.4
x .. d
In + L
0N
In 4+ 44 +'4 In+ 4
W
c D. 51
'1u
WI.
U.
ml IS
I iP% II I
I II I
in
(I• IIw NI U|
PA 5 , .1 1
In÷ ÷÷+ + i
n I I
Wi.5I
z m IIw Uj II
w II
x IN I
# In
x xK I
A~~ ~ I .~±)
"Ur i
wo
ia ISII %
11%
IIN
wII
VII In
I% ~
*0 a
I+ I
U 1*1w
10 x
W x
.1j II
'4 40I oI 1 00 Imp I ot oII a41 IR
-D 54 3
Vo j % va-P
wtJ
1 40
I a %
to a
fLn e0m I
x 19a'
I in + ai
mx +
4t4 1 42 1
a en P% NN(4mMm tv 0kND. 54
x m P-4 Nfl N r'm M V i
wu
K I
I II
lee
4n I I L
I~f
69 11
NU I I
NI I
m II
4* w
Ww w
go I sIn
LoiI
N I N
g~go
c +
x I
~~4V 44+4+4x +x 44 444+4+4x Ix xg x x
1 x x x xx x x x x..... ....
D. 55
-4 P- 0 0 %t- - -NNNNNMN
ONM*IMWOW%- WNM0~NNtMWM*N-4NN"%Ut 0-6 1"W4 0- p P" .44 4
ko I 4III
In
ID f
* in I
4I IN
@a I I I
Ln + i
'0 I
W j U l I
W x
4J O i 1
N I X x x I
cI IN+xx x x x x x 1I- to W XIX X X
Lt itx X X WXx xxN FMI x IXxxx
f.Jxx Ul I hxU, Ixx*1I
Ix o Ix xx x-i U oxW+xxx
OS. xMIxx x listxxxxW ox Ix.
Ut ...Jý U14xx x xx o~ o
'. .*InI'aI41 41tt414%tItIjtII 20
I.~ ~ ~ ~~D 56 KKKX~~K~
W - 4 .4... MN t, -e 0-- N nNM:)
+ +
+ +
I I
Ln I I I
w I I
I a ;n II 1!00 -4 t1) .1
Nu "tM IM
z Nl, I +INU
u ww
X CL
0 N I xI N
4.+I Z IxIL) I xx xx X
N '( I X X xXin 11 x4 X x XX x x 411
x W I X Xx(X I
-. bQ + x X xx'xxxxxxx IIx x XXXXXXIXXXXXI
o I '~xxxxxxxxxxxxxxxxxxXX I
+*++4++4+ ... *+4++4+++++t.4++444444
4K t NI Qt Rt Rt ot %0t 4% QI oI 0I uWt q0%to -IR0 ~qt~~ mWN O 0 0 4I~R IO
D. 57
LL,~
01 IN
to I
4c oI I i
.44 .41 A
u Iw I
w. I I
1A i U) + x4 %I- I. I
x Ix0 V)Is, + x 11
0 1xI I C
EWxx I I
W Ixxx I-
Ix xx x I
w W Ixx xx Iin IM+x x x x x I
dc I Ixx xx x x
I xxx xxxxx I.C~ I x xx xxx x x x I
LI) 4 xx x xxxx x x*i
Kxx x x x x x x xX I.xx xKxK K Ixxx
xxx xxxxx x x x Ixxxxxxxxxxxxx Vx+ . .. .. . . ..
Ws Op IQI 04 I Q QtKK t III W q 4kqt ttad44tR6 I 1 1 4 a 4 QtoIM44%isisto4 %toIQ%
.K X X K K ........Cf lp LnX XX X X Li 4i AW 4I Qi IL)t n14I lV
W WK XXXXXXXNOIINILIN NO IN OI VXX K X KX X
Z54 K X X K X X X 1'
I~ ~ D 58K K X XK ~ X I
I 4XKXXX XK
zz "NtMWWýWQNNNNMMWu Liý-4- N N NN (-INN
i--
+ km
1 to I I %
:.44
M lI I i
4A I
N I
In +
~ +
=Wr W)I +1ONXi Mi Im
(A4- I xP' WI 1W
0I-L
Wxx I
0 N1Ix V
go I Ixx
UJ Ix I'.
XX I Ix xxxx Ix
(#2 .J Wx xxx 4WxC NI )C xxx id
xxxxxxx~ xxxxcx
OK I xxxxKxx Ix
bw x XXXXXXXMXXX
U)4 i.xxxxxxxxxx
*I xxxxxxxxxKxxxxxx I.. . . . . +X ...... ++++
to to tX X X X XXXQX bQ ok ow RI o% l0t o im
D. 59
JX U
WLJ ~ vwN MN.CMNNN~JI
ka I
w.ri
+.1Ul I 1
tU c
Ln +
u w
0 C I I I
L '( I o ý x
c I Kx xx I
I 'Ixx 0 UI- WLxxfri I-. ~ h I%~x .. .V U1W-.Qx-N
R( x NNNM xxMMx gw'*..c -3 % *** x*ea xxx
m n -1xxjx xdcIo Ix x64 x XXXXD.60
4 kI
fn II+,
1 In
AWE Iat~ I I
0aýi in ++I
I I
x Iwx
sI.j It
a I I xad I Ix x
I Kx IU 4 I Kx I
.C f I xKK xbj v)Kx
t In X x xxx 1ISIXXx x XX
I xxxKxx I%X w I K KKxxx I
co I xKxxKxKx
I gxKxxxxxxxW+. XXXXXXXXXXX I
( I xKxxxxxKxxx.C*4 I xCKxKxxxKKx I
I xKKKxxxxKKx I
11 x -. x x xx xIn .. . . . . . . .
j tt mt ot I nI 11 ot 1t oI ott
D.b61
A*ui
VWMM NM M-MNO ON N.' 4N N M N d.
Is
I I
.II
In I
fn WI In
N I
09- In 1L
w I xIagA ~+ +
N I9
~~x x
aN N MI
W I XK-l xx0~ ~ L Ixx II
XX Ix
d NI xxxxxx INN XXX IIXxxxx
L 4 I ooK xxo XXf41 I * K xxKx I
I Kx xx X x x I
- I K xxxxx Ix
I4 Kxx x x XXKxx g x 1.I Xxxxxxxxxxx XX x I
N~~~~W W XXXXXIA XXKXXXXXXK m
D.62
ini-r + + InNNM
+* +,WI I.ko I
9' M IIa Inl 4. In
I II
CAP U, I
w I0 9 inI Ino w I I
x. od'IA n WI xi
4.0 IxW~o I
c ,u.I
in +, *0x x x4L
w tIxxxxx
Nt I Ix xx x
Ixxxxxxx II-x x xx x x x I
W11 I'l tal IM
Uk I I %Al4 Nt Ko mt t4'Ri%(
Il 4Q tm 0 itotO NM NIIO KK IN~MM NM
N, mmfnm I
D.K 6.
...........
WL) .4-o -4NNNNNNNN
"4.4 POM ~)NM (' '4s"
L6 lf
In 11
I IwLne L ni
Nt I
:w .
'L(A i LI+ o *k
W I I
Sxxxgin PI It's
K I B
0 NI KK) IN
x X.. xxx x
01Po01~x x Is"
xxoxxxxxxx xxx
B1 K XXXXXKKKXXXXKX I
K++ .... ... 4 +4+NM MSM M M..
. .. . . . . . . . . . . . . .5 5 5. . . . . .ORI m4 4i n% o6 o4toVD.64 m *I
IN1 o o O
*U--
040
P.~
In I I I
U) 4 i
L A. MI xuj I
N " I xxcL1 01 XX XXX
j n+x x*iWNNW0 NMMV flIN
= .g= *g g.~66 66 666 66 66
D. 6
+ +
ka in
a+na(0 A I Iw
YI- In + IW)I inWIIa
(A o ~IIaw II
FA U I
in I IL
+ in
14 1
I ~ goxx~o' I
... x ailul ii Ao ~
01 CA , ,, t
OS-rnX
D.66x
S'S x. x
0.
ad v mN -4IL.
In
+ +II P%
N U) 4A II4I0 I0.,4 1
.1 I I inI:lxc
W n+i
0 I I x I
XX~x ý X 4
~xxxx I.In , 10 xxIx 6 inaNI xx I
4 t+~~+~ +44+44
m~~P 67xx
I Ix
a .m
In 4
+ La
I inIts
tot I
In i
N 6UM
x 0.
+ 4-
~xaa I
Uin +.. xh N xm.*V4 NI INox
xUxxxxx I
t h Ih CxXO95 X55gg
.... .... ... ..it. x
11+44+l t CCl I +~ +CCC+II11142+tl. . . . . . . . . . .
41a
D. 68
I. v Vl
w Z NNmwNw 0 M %
FvN MNNN"
la It
I0 1 1111111+X X X +KKX In
L i.
x C 1 x KXKXKXKKKIW4444 4 44+ +i in++ + In4+
0- N I I NMM M MM M W~.
go 1 0 09r
%WtUW ,.NWM --.-- r-NNNNN
inI in
las~ +1
On II
II In~.
In I
in +
w WI 1W
CMI U4 0 Ix~ Uý xIm
U: to +Ix4U
x Ixlvx
Sxx
oxx Ad x
xx xKU: -~h+xx K 4i
+ I XX +i
xxxgx xMA J KXX9 wN w xxxxxxxxx
II'IIN + -dKxxxKx KKx I11 go I axxx x
c ~~ K xxxxx
K ~ ~~ 444 ** +4+ 44 444 ++ ++
Kg xxxixIn -K xx xx
.a .l~ .* ... . .* . .,l .- . h~. .4 .A .- .I- .
.1.
%K +
Oto
I I
4I I
III
cy I+ +0
MY)f wl 10
wE I
?A " ~+go n go k
r-
w qt I I
Ku+ InCA I
wN MI IM& 441 I m
00 I x0 N4 I I
cc ~x
3 Ni xxx IN
(.i ~4xxxxxNI x CCxxx I
4K I xxxxx
x IxxxN x lt ~x~xxxx
-i ftj +) X xxxxxx +tco .- lx x X X X X X I
4. 4 Ixxxxxxxxxxx XI
of u, xxxxxxxxxxxxxxx +x xxxcxx..X~xxxxxx
0 ~ ~ x IX XX x XXXexxx xx XX XX
+4 + *4 4 4444 4.. .. 4 444+ +4 4.. * 4# + +4+44
4cItll t D W ROMAh0 ,t ot4w at ItRV k00 Qi 4I 4t ot owI ba00t
> to . . . . . . . . . . .
cc to1 fQ0t 41 QM% mt l %a4 4t O4ms bt
D. 71
wu -A"4"-NNNNNNNNNN
IA. b4
iN +
BI W
B IW N 0.4W
W-in Bin
LO I
U)I~ 4 In
u w I
FA I
Bj I
meL B4
0 ILl B I B
Ifl .J 111 14
0 NI x x INeca B X x I
c > I xxx x I+~X Kx
XX XXXBKXXXXx
u I XX xx IX KN Ily Bx X XX xX
C Jx x x4 XX XX XX X~~~x Ux x BX XX XXXX KI
ryw I KXKXKxxxxKKxx xto +Kx x xxxxKxxKx I
>- IAJ xx x x xxxx* 111 I-~ x x Mx LAP-W-M"4Ux i I- E.( "-NNNNNNNNNNM M xiM1P1
-g x xe4040*a94u*4x0x4W0..t x I
U.#72
14a 40I Qt 4 641 ot o1 ot.t 4loaI 1%
*..~-*.>*I .- . IA ~. . . . . * p . I .. ..-- - .. .. .... ** - .
UK
+ +
B I
I0 w
m(, WI kW
40 I I I k
4, I I
1c I
in I + i
Z N B~i +BI
.Ci me + +
do w I B
0 . MI xx
Kx xc I2. Bx x 4
1/2 -~ txexx Nuu NI xx IN
m J I xx xx Ix xx x +I
qI xxxxxx IN
N W In xx4xxxxx x uc -A lp + xxxxxxxx x +I
'. L 2 21XXXXXXXXXX 1N Wc I xxxxxxxxxx I
-v.I ?x KKKKxx xxxxxIme( I xxxxxxxxxxxxx I
I XXXXXXXXKXXX,)XXX*I xlxxxxxxxxKxxxxKI x KXXXXXXXXXXXsXKtx
fx x xxxxxx~xxxxx~xxxxxxx+ ....... 4+4
14 mm tp~ m 1m % 4ImI p0 mta m %I m at l m %Im tkmm0 b Qt t nto0 Q1 %I RI a Q0I m" QUt0 t oM ka ta stLo . m. . . . . . . . .. .. .. . -
a.~~~ se too IR %k Immmmmmmu~
D. 73
ty .J N NN N NN
j+
wiI
to In+
IW Io o l 1in
14 1w I
0 nI I
In 0 1
m W
uw- I
wi I
ad w I x
0N MYI x fx I
1. x Ix x I(A +.. m xxxx x4n NI XX INxx
S I IXK
11 xxxxxxx
N w I x xxxxxxx IUIx xxxxxxx I~ X XAXXXXX IIg xxx x x x 1I xxxxxxxxxx I
w -, xxxxxxxxxxx ~I XXXXXX(XXIXXXX IX
I~l XX XIQXX OR INin.~~~ XXXXXXXX
vi ý- z NOWC%@MIR NNMMw~mtkinkWN-K..Z *z W nMm vvvv .
D. 74
xwL -4--4NMN-NNN
LOI I O
14.0%
CY I 19
In 11 i
w II
CA~ IQ
0 in
W+ It Iw
0. m n. +1x4 f
u w I xIt
U W I xw Ix
~ xx
xxxxxw .j in.xxxxx 4
c )0I xxxxx IN
In+ xxxxxx 4
~xxxxxxi41. xxxxxxx + L
N UJ I xxxxxxxxxC .I ~+ xx)xxxxxxx 1
: m-exxxxxxxxx f-I.. I xxxxxxxxxx
- In xxxxxxxxxxx L
vXXX)CXXXXXXXXX Ix x xx xxxx xxxxx I
4 ++ + ++ + + . .,
IA I-Ke.MM**OOWW CIMM214~--(
D). 75
UZWL
WBJ -4N NM*M N
In, + If%DI I kil
B +
ko W to I
0-6 on4 i
En I I ic
(Im
Wn +
Z N
Min
N Z BIo0 w11111Enx
NB Bx
PB B KXXXHtA -j n +1 xxxeL
m Ni I Bx x1 -Jxx
: U ~B KxKxKxKL C Bx
Bn + xKxxKx)( Wme KKxKKxxX U
B xxxKxKxxCx KXKXC BIn XKxxxKCxx B
N w 9xKKxKx B0oo B K Bxx
do4 *++++4+4+*+II -Jx
x
...............................................................
Bi'I B-xx xx IW~ ~ S. N
+ + +. .- .. . .. .I4~ . 44 ++
W9 to W.. om N N V Mb
InI
I0 1
a ) w I I I
w tsON InIL
Wlad I IWw II
.9 ~+LoU WI + In
CNZ. mi u
0 ILi I
I I
x n W.W i .91 x+a- N x
0 Ix Ix
Ux x x I-
x 9Uxx x I
c 4 to KKxxx x + O
Ln xKKK *xx x xxx x4inI' xKKKKx KKxxx x
WI K KKxxxKKxxxx
. . . . . . . I . . . . . . . ..Kcc ~ INK XK K X Ikt RI QI Rt QI oi Rb 4I 0t 4tmwINIOR4 40aa tot 40001
it( NNNNNNNNNMMMMMMMMM*-a a~ 41 0 0 0 6 a 0 0 0 a 0 6
D.7 7
0.
inI + to
W) + +II IA
to4 + ,in
I Iz
In Iin04, In I
I) I
in* 44
In I
I 04( WI M
u Nt IIul0 Iwa- I
mIL.. m If 400~ w I 1W
x AK I11 I
in + x 4InIN I I
in I xI I
5+ x 5(A x x xIt.
I IXXIas inx
HU)xxx I
C I x Kxxxx INS I x xKxxxxx
=xxxxxxxx ILi I xxxxxxxxx I
N x( I xxxxxxxxxxxI.. .+ ..... X ++ + . . .. .+
0OS40 -,~tn %XKX XXXX K 1-It N N NX X X K X NI )C V- t-
I~~~ X 8XXXXX
uI
in+
to I
in, + iWI 1
r% IQ+
>4f WI 1P. *M I
I* I
(A (X Iw I(A 144
94 i I In
*1 -0 1 1 o
+1 + In0 N mI lW
w Iad IQ+
ILw Ix I
MA . L11 x x I0 N I xxx x I Nx xxxC x I
3.. IA tX. +X Ix x
u X~X XX INx~' x xxx Ixxxxxxx I"
¶ flxxxxxxxxxxIN W Ixxxxxxxxxx I
1. o XXXXXXXXXX II9 X XXXXXXXXXXI
S xxxxxxxxx.xxxxxxx I0 Ix XXX XXXXXXXXXXXXXX XX I
IN 4Q 4 Q 04 OR to 4 to IN 4 %R to 4R 4x +s v 40 Q at IQ 4to
ww to) -K atU to -NM W1fl bn a-m %o-Ntoooin i
al -,K e s 4 a e ee...e,
D. 79
wu
owý
in + Ln
+II
Ln in
It 11+ +10
10, It I
WN" IW in + in
In I inI14 Qw
L4 I*+
0 Ln I
in *1V Ico I x
x
xin + x 4 W)m I x IM
x
x-j eg kk + xx +40. 'L m : xx I cn03 w Axx I
x mi xxxIf xxxul j in + xxx + ut
N: xxx INxxx
z xxx)I. x xxxV) + X xxx + IN
oxxxx :N
xxx wu I xxxxx x 1
xxxxx x04, xxxxxxx *1 Lnxxxxxxxxxxxxxxxxxxxxxxxx
N w I xxxxxxxxc -A I** xxxxxxxxx IN2 0 xxxxxxxxxxxL xxxxxxxxxxx
xxxxxxxxxxXICXXXXXXXXXXn wxxxxxxxxxx + inxxxxxxxxxxxx
x xxxxxxxxxxxxxXXXXXXXXXXAXXXX
xxxxx xxxxxxxxxxxxx
. . . . . . . . . . . . . . . .
Z4z a 0 0 4 0 0 a 0 a 0 4 a 0
V,
D. 80
up -fNVWMWr-%NNN(MMlaW -.- NNN~NNNNN
+
w to
w o@ 11 I 1
LoI in + 4 In
in In I IL
W
I IINM- I
I a ) I,0N 9mI in
M-4 adIN+ x *In
0 0 C I Ixtow I xx
i ~xx 1
xxxxxx
c X KX. x
w O~d"MOMO 4O MNNWIMNNN
A..
W IN . I
g * In: sIn
0 to
O MI x
uw I x
~xVW I pnI 1Wxto w I !x I:
f ItIN 1
IA S j L +*2I I-'
I x)icN W I )CxxCxKx~~ U xI
L ~ xxxxXX
0 I XXXXKXxxxxxx (x~*45* 54, 5.x '5, 5, 5
NJ Oxxxx
ly w x xxxxx.c axxxuxIww 0*xxxxxL A C - NMxlWN x-Ixm. t.. %
I ~ ixx4.4 NN NNM MM M(x.z~ -= *eesews...egex
14~ ~~~D 82+xxx xxx
I*JL NNNNNNN
a +
P% I
II
eI I
CO I
II in
&LI II
I I
in + I
ONM m I Inu U W I IV) I I
3X i In 1+
X SK
IA InI x x *KMKCNI x x IN
2. W K Kx xxx+
0 I Kxxxx xx CI = xKKxx K
Q xKxKxxKx Ipw I KxxKxKxxK I
I Kxx~xxxxxx II KXXxxxxxxxx I
*i I~ xxxxKxxxKKxxX Ij t44+ 44 444 +4+4+ +IQ
an -1 IW NWx xx x Iat ( I x x xx x x
K .K *eeueax4eeeeees9.e I
*1x x x x x x xI).83 xxxxx +i
I~~~ ~ ~ 2xxxxxxx x
Wu ----NNN NNNN NN
111+ + i
'a +1
g "4 WI to(A ad
w
U, ~+ZI m U
w IINL If
*4.
0 ad IP,
0 Ni xx IN
xxxxxx
X XXXX XX
I -I xxjx 1
x xxxx I
4 4 + 4 4 4 4 + 4 + 4 4 4 4 + 4 4 + ORKxxxx
-J xxxxx
I xxxxxxx69 Wx xxxiJx I W ~ W W jW~ WW N'
LA Mx xxxexx +ix-.zxxxx I
I+. . ... +.. .. . . .
j ~5
I u ý4NrNNNNNNN
In 4
-- 4 4c
g~cI I
z CV)- W)4 I
ac m I I IC
0 N I XX(xxx In.4' w I x'I x ad Ixxui
>. - I ~ X()xxx x.
D. L4 k xxxD.* I
N1px x xIo*-rx x
ur =
41 " N .-. N NNN N
co I
in, 4 I
i %D I I 4
0 1
InI f I .I+ in
kI +in
IMP WI in&NWZ *l i IM
0 4L m- I B Ii
01 7. W I Iii7
>- I x B
0 NWI 1Wxxf
>- I Ix x
ON N iix
P4 1 xHxx
Id. I (XXX)XCXXXXXI~N I xx ,x(x
to 'Axx x+Olo -4x xx x
L. a( -wx xx
I xxxD.8b x
L+ xxxxxx
-IN 0to O NwNNNNNMMMWuO pi --. NNNNNNNNzK
1 I %
I
I+ In
en ( IA+ I1
w w+
L)W &
~+ La i
u V) I x
I x xX
(A J LO x xxx +1X
xxxKX XxX
:0 A tk +X U~)XXKXX
XXXXXXXXXXI
NXXXXXXXI xxxxxx Iw 61 ++4+44+4 +* + +4+i
x xxx-Z M N NxL xx -. N1
x xxxx
N~ ~~D W87xxxx
-4NMOWMO aNNNNMt'MMIJJL .I4ý-4-ýNNNNNfNNN
od 4-NMNNM
CD I
LA 4 in
IOC
Ln~ U+ +eI
I., 0 In
V) Iw x
99t x , I Igo w x x
an I KX xxXxx I~We ~ X X XxxX K I
m X xxxxx xxxxc
N- LJx xx u
CLJ -j t+x x x x x+tD- etaNM W-NMxLxfl
I L xxxxxxx-axxxx
I xxxxD.88 x
0.
In. + I
N I
In. In
w Ia
2w-0
UJN.- I
9m In, IIin I
w IIIN +
in ICY- 0
I4 x
OM - In + 4WiONE m I xgIu Wj I 24c
U, IL m i0w
CA -j In4 +xCxxx 4O N1 xxxxxx INIl I Kxxxxx
xxxxxxXXXXXXXXXXXX IN
I xxx xxx
P-4 (x IX Xcxx 1In: aaa!!II'I!II+ It.14 Ew WN.-...."4.....-NN NN MM M I
c j q+4x x x x+t
n to 1 xx xxxD.81.
wu -MwM~NWý.NN c t t 1
W)-
I I
'IInin
kaI +
NII
I in
I xIa!U Win1
UJ I
w~ xxv i i vI-. I IIII IIn x
4J X CX X
I ~ ~ C)XX (XX I
x(xxxxx I
.j xxxxxxxxxNxxMMMx rXwx-gx
D.90 xxx
ME 4.-4NNNI'NNNN
10
Lo I.
M . I I In
WI IW
IA W WI 1W
clo In6
4K l in + W)
YI I x
I~x I
x x4MWI + 9 6
w x
0~ ILMIx
C9 xxxxx 4
0 N I xxxxx I N
so I Kxxxxx
u ~ w xxxxxxx
S~xx x I X XX X
L-K I x x x xI ~ qI.-- NNN NN IMM.xIzxxxxx
in+9.,x x x 1IV.x xx x
xx x, D.91
*xx.Ax *xx
II.9
I II I
I I,.
I~ I In
IS, 111at I
w I(A to I
00 In I
(qC -4 1 hty I
. +In+ Inu Il I0 I.
Um wUIX
N dI ~ x xx~c>- I x x
CA *( In 4 Ixx+
W 1n4 *LA
-I ~x)XCxxxcxx INiI xx x x I
wh . K V)xxxx+6V.xx x .
..........................................................xxAJxW
w x34xJ 943 + xxxx
2~ ~~~D 92-xxxxxx
K 0-0
tCA W I I #
04mgo I In
I InINIn In
I1 I v
(A x
so w IXX
ua- I Ix x
0 N I IxxN
0 NI xxxxIN
N XXXXXCXXAL xxxxxxx II-x xx I-I 1
C I ~xxxxxxxxx~xx
> x. 'xx''x x
0*w l- wNN10W'.
09 I
+ 61
r'Jt w1) Ii
M0 1
UI1 1 i
w II
in~ In I
I, I
I- I x
ONI. m NxI
( AI I xx I
w I xxxx
0 NLAI xx1% w I Ixx
z w I xx I>~~ xxx~xcN XxxxX
D. uA14+ Xxxxxxx+1Ix xx IN
""I XXXX~cxx+6
M w I xxxxxxxC -j +xx~x)xxxx
L x4 IxxI )~)xxxxCxxxI
0 I ~xxx4x~xxxxxxxxz +4+4.44 + + 44.4..4.. ..+.4 . .4 .4
. 4 A mIMkto0IQtotot t, I t t 0t t 4 t % o S
'0 4 *S SI I
D. 94
1. 44
V.%
M + I L
me 10CI
ulI I I
N I I
&nc -Vl +1 1WO
"I.I It
N I
=0 M 13 + + iONZ MI
I IxI- I
xJxad + ux
0 I I AINx
soX XXXXXXX X XXx ad x
9X 11 x Cx X(AX XX X -j W Ix4L
'CX X xxxx "x
Z(~~ ~ W P N NN N MI e
D.95 x
N, N YN
WIA
+ +
I+ I
.NN
ON m e
I) Iw I
I w I I
Ni i nxKx + I
&4 z +xxxxxCa I KxxxxxKxIe
0 q xxxKKxxI
CI 4x KKKxKxx I-
x xxKx~xx .
N1 I x x x xm oxxx
cZ N NNoN N N MMxxxxx + k
z go xxxxxD.96
L 41C x:vxx:x4xx
of m --- N N N N
cc( oI Il&
* I
$W IkaW IN
1 , I I I
3C 1W
I IN
ac W I IO Z MI x a
U I 1 W*(1 1 I
c.ow I xI3~h- I x xI
w. I Ix xz I x I
91 NI 1W x xx*I I x x I
mWP I x x IM t I x x I
ayIx x x x x I
m I x x x x x I
1. I 417 4 IAxxx6. 11 Ixxx x x I
ix I x x x x x x IKX + Ix x x xx x x +
I xxxxxxxx('I .. J 1174x xx 4117
L& NI xXX xx INx:x I x x x x x x x x XX X
X tR XX IQ IQt QI Qm0t t QW4 pf !
I) X WeX XXX ol-N wO N o - m* W
17 aa 4I 0 a 1 1ita 1 1414 4 4 0 0 It
D. XXX97X
1 4 NX X X X XI
I P%
I% I
IAI
4'. I
z (A
w I x
in +x I
I Ix x
WU) 401xx +.11 Ix xx
N .. I x x x x IU) i Im+x x x x x
W I x x x x I
* 'I x gUxx x xIxx x x xxOx x I*xxx x x x x I a
U.n !I xx x xxx xx
b'I I 0
D. 98
tot Q v * % k 0r nN nI NI
sea.
OD I
W) +
ka I IA
In I
W I +co 0
it,
t I x
atow I xION MIx4I
xI I
01 UJ 11 . x
0 C I Ixt
In xxx xI9 > Mt xx pew
41 m I x xIKu I xxx IINI x x x1
K I xxxx x4 C ~ I Xx cx xxx
I i x xxxx xw I xxxxcxx
N -j I xx xxxxx2 xxxx xxx x J-Ixxxxxxx II xxxxxxxx
Ini I XXU XX inIi x I xxxxxxxxx I
xxxxxcxxxxxx I
in(oloon~~ LtL NN N N M fin 0 o
D. 99
MY loll
P, v~ N P, 0 M Nm N vi-
IQI
Ina
+
C."4 + Ino-~- u'.in
mini in
4*11
WA ICA I
Ix I
0M. IAw+ X 4.111+ AC)N N! Nx x
.C I XXXX I0 1xxxxxC 14I I Ix
m 11 9XNNXXXK 4
u. 4- N K Kxxx KX InoI xI x
L. ~ ~ ~ D 100 ýgxxxx
NCO~
a)
* 11
in + IN
a I
L I wi
1.1
a e W)P4 I I11
W V
u wI
Iý I
Ifl I
(A j L 11 x
U, W4 XNXXX
u I g~x
XX X LM+ Kx +1 1n-'41 xxgxxx 1XX XI.4
I )xxX xxx x II ~ X XXXxxx I
3 XýXxxxxxxxxL .09 xxxxxxxxxX
I. (IxxxxxxxxxxxI
*I xxoxxxxxx~xxX Ie~ IXXXXXXXXXXXXXX I
W I K xxxxxxxxxxxxxxX I4 +4 *4++ 4 ....4'+ .. +... 4 4.4..4..+..4
ww00w0w W WWm W nWnu U,
D.101
1. V,.
WUX k%4.-.-4NNN WMNNM
I IN
g*%I
cI I
9mU I In"4 In I I i
w
XI."o I
I I
X- n44in
ON 9 I I fw
14 19 *4a. NI tCXX Ix
I xg~xxxII xxx x I-
xxxxxxx
-1 gVxxg
to V 1M4xx .
it I- KrJxxI axxxxI -xxx xxIn+JRx xx x x+I
I oxxP.102x
I Ix x x x
I.xxxxxxxxxxxxx
U.0"
ift +N N* -
at:
*n I inr*I
w o ICh I I
*I x
WcN.- I
w mc I I
.j IV0 6 I mw 1 Ix
xix I x II Kgx x
wN -jLnI Kx x * kht, N xxxI
to Ni 0 M I.C I
£~ )eIN* I +
.4 KxxKKxx Im wI KKKxxK)x I
LU~xx I KK
-I xxxKKxKxx I-
Ix x x x x Io Ix xxx+4+4 4+4++4x+ + 4 4+4
C -Jxxxxxx...... ......
Z
In- . . . . . . . .
D. 103
UK Nmi-.M
W ~ V N N?%4U
Ito,
n +9i
Im +ei
in.
In I In
Wu II
w
'mMU L I m
b ft I x IW I in.I+k
o N I x I l
30 (A Io I ,
IA I 5X X 5 Iini x Ix +XXX In
c -j 14+ emxgo U -OXX5XX XX Ko
p4 9 xxx xxxxxxxll0 OOXXXXgxxxxxxX
i x Cxxxxxxxxxxxxx IKxxxxxxxxxxxx
9K XX Kx XXXXX xgxxxx 1,o IXxxX K~xxx XX xxxx XXX I+ 4++ +I 44+4............
it miltitmm u~ itmm tit I...................-
w mkm4 INN NN 10 to m to t ~ q IQV w1
D, 104
a m P4 MWtWSIA-M , "NNNMMM
W-
29 a NN
NE I
ON In
uJ I
In I.
in +-Jt
in +. in
an ". I
w I
j n + 60 N I xI N
0w N) I xKKxXK I
~xxx I
In IKK RXKXKX X IM~ w I ONI XKCXKX C
r. j 4 ox~xxo( II 80KxKx Kx(x~l I
I KxxxxxxxoxxI~xKxx x)CKXX X
t444 4x. 44.4+444+44 444 I++K+ *
OK j-I al okl N ~k %t
D. 105
i~ w
a= ý -ýNmv IQNNN NNNNNm0'
IN: z
+1 InI I
+ +
m z I
in~ 111 61
w I
II~min I11
N m
(A 5 x 4 i
N x4 15
30 V) 4
0 1 X0 (~cXXXXM X4.0 x~x xxxxK '
~KxgXX~XOX~XIc XKKXXXX.SCXX x +1
0~x x x x I~X X X ~ K xX I-4+ 4 + 4 4 4 x 4 + 4+4 x *4+IJ x xxxxxIk xxxxxx
In, x 4 x x xx x x 4 4 4 44 . .4 .
:01 x xx x x x x
ww
D. 106
mU = 4lEWNom0%ao1 NNNNNN.w- NN M
I I
In + I
SW I
it WnI in8
No I
zM in I
cu~S- m*W4( iS sit
ON Z M
IA Iw I ox
.j xx0 C6AI t
I xx
LIt I xxxw Ix xxI
I 4x ICA -A Ln xxx 0(fl g. 111 gX t
(Ain' xxx +X
LM UJx xx + W)~XXc -~ ~ . xx~xxxxxc xe~~~~~ U - xxxxxx I-
I. IN xxxxxxxK x I%
a IxxxxxKxxxxxx
Ln KXxxxKxxxxxx +L
x~JU xx Its inuxx IIn I-E xxxxInx
Z( WNO ++*+++++NNNNN +++m ++
LL
usON- fNW" mo -M PM MOO-K P4NVWNMNNNNNMMwLJ -4.44NNNNNNNN
r% I
Ire
mf+ +9WII %9
z I
i.En
I~- I19 11 I n 8 +1 i
0IL m I X.
0 NI xxc x
FA NI ~ x x~x'xxXCm I x Xx
UU I x4 4444 +4+
N LflWUx ex-N
D.ý 108 xxx
LL.
I CZ
k" I I L
In*
z N 1 1in I
M I BB BB Bx
SCb 7 If BWIA B Bn+x x+W
3. I- B BCx~BhI- LB4 *xf
lABxxx ILi Xx x +c -j1 4flBx xx4t
Nx x xx x x IB Bxx xx x
U. L Bx x x x x x x
. .. .B . . . . . . .
bm Ik
o. "4XX~ "X "4XX BNN N M M M
D . 10 9
-- m movM44VNO r oNK M 0r rC ~
Li.'
N 1
N I I N
1-4 1 "'4I
NZ I 1 .4I..
~w I
II'+ :1De +4
x I IXKmK X K XX X X
D. 110 I P%
(A kQ 4 +10.A
+ +
I I y
W In
s in I I i
N m I I
v1 No II
SWI
. w.
xn I
I.- + In
L xxxxx I
O L n Ixxxxx an
I9 xzxxxxxxxx I
++ + +.. . ++ + + ++ *. .
o 4z NQt II Qt at Q4 q p0 4I lt la ot44+ 4+I i o 4 L+ t +4n 44 +n +4 +4 4+ in kq44+ oI t ot i(
-J kpI Mt. tL nL 4 ' IC DC 4k Wo 0 Da, DC cC)WWC NM-I- WW a
W-
im x
N II
NI NN NI IN
I I
cNI I
In IC 4
cpmI- Le 4 kIva" I low*
V) 1* e
L I0
Z 14
f I IV
00 1~z I I->mU I
ka I Im I I
CL SI jg.
U, x..a *ei
0 ml Ix x x Im
41 Ixxx IOf I x xxxx x I
"I XXCXXXXXXX .1%I xxxxxxxx I
I' W I xxxxxxxI
+4 .........
44 I 4 XX0c ) IQt R00 k Q1me XX X XX X 0 00t flR00
-. . . . . . . . . . . . . . .I.-tQX XX X X %kI4 Qt 4I uI mi Rt QI
It X X cX X X 14I NL OLtI n40w %L 4L 1wItmlq XXXXXXXXX iniwwC cod(I
D.eeee 112 eee~e
.... . .. .
0*iMI =
-NMNMMMNMNt'M-4
k I In
IN,
N to
>I% -0
Wim
in* 14 CIa
Inn
XIn I I%D~
u W II
0A ILSMx a II
t4 IQ 4I
Il I
O 11
t) d i x x x Im
Ln eKmxxxxWI 1Wxxxx
I Ixxx xLn U,, ICIAxxx +L
1)It 14 12,X~ X XXR XXX)XXX
Ilw I1,ItIn XXXCtX~(X(XX wX~;N "Ins.wnQ N q~~p
D 113
*uan0*
~.NmNmW'mmmNNN
4~IN
In +NI1 IN
it +
'.4 1ly II
in l-. I
cI. in.i
w ta
w
*W In01 IW
m. IIin +4i
in' .41
i;n I
w II
in +P. I
x I
w In+ *inxx
x XXXXXXCkzxxxx
U.I Ixx x x x x x x x
to : 1. I. IItttztIt
0 -4 K t..q n a~NU'
D. 114
00 N W NPl NM 0toI
gY I
am In4 I 4
-Y +
in 1 I 4inI
I ,
0W= U7+
IL cm I I
I-
x. x xxx
( w x xx Ix
I t XXxKxxxxxx)( ~ I
o i xxxxxxxxxxxxxxxxxxx44x xx x x xx x x 4444 +*4444444+ 4+zxxxxxxxxx
... ... ..
ORt oI f0MI Ir ot R 4t 4t oI 4I oo mt8s to %I to I* *I okt 04 mkR1 041u 4b ow4 w#
D. 115
Wa W 4_NNMMMMMW~v
N II
NIn
No I INt
WY t"I IS
a.1
0 ka I I l* ( 1I 1W
Nx I11 I e
w n45o xi
"41xI, I* I
A IUxx x
W i Ix IN0 xoxx
L O 4c SIx ISb" I Ixxx
WI 1Wxxxxxxx
IAI
~~.D 116
I. ... A.
wL) ýNNNMMcMMMM**
WI-~~~ NN .NM--
In + ~ ~ ie~ in-
cc I I
In.In I
N I
.44 1II
II
I~d If
I I
(8 I
V) IN
II x x% I
kn + 41in
AI I. I qtX)(X
xxxs xo)X xXCC(XX(XC)0xxxxI-+,, S +44*+4+ 4+
c S S S S S
b~d *5~gt46,x4@4g54
L xD. 117
N I
N I
go +
Ln I*+
ml
LAII
.j 19 +SA. Vo I
="Mx 9
4~~L 41
111xIM -K I
w4 in X0W Iow
-Ju
13 . 1144
ka~ 14 +~ K 41oqIxgg xlx 1.4I Ixx~x xMW Ix x xCx -xox xxxn
+M 44+44 1444I,*4 +14 4*
itt *I 10 *1 loll Winnin . Sw w
D.11
.jQit
wuZ-S7N W~mmqSS-M~mM*
Im m -- 4 N N m mm Nq gWN
Li. 0.
NI1
Ih In
in.
m WI
U WIt
,IA
0 %I I NON I
14 %a ~LS + i
w Ix -x
.j x XXX0 ,
I ~~ XXXXXXxxxx II.&.zx xx xx I X ( X X X X X X X0 I XKXX~xxoxxxxxxKXXXX
N. 4. 4 . 4 4 4 4 + .4 . 4 .4 4 4 .4 . .
Wl. IflxxxioUx *lU fUuhlhl L,X.4 UWx xx WxNN MU U IO~
3~ 43 x *xixgx ** ****
D. 119xxxxxx
uU wu-NNNMMMfMMfM*
N 1
411141 N
(A AN
w I
SNE
a tm I I
IQI
W 1
to1144IL In
x be +V) j In+I
1* *
Wi1*
c j 1A + XXXXX X+t
* I xxCxxxxxxx IL. '( IKxgXXXXXKxx
66 XXOXXOXXXXXXXXXXXKxx KxXXXXXXXX
knwwp% ~ N 1.too air .4 N M mlow n
D) 120
in WI. + LnlV N
N I IN
*l + InNIn
w ODI Im
ko I II- IINk I
I
w I
- +
x~- ImjI Win+1i
u I x xIn1 wX+L40 1 mlei~~ -xI H-
ul I Ix3IflC'x I
44+x x x3 en oI-x
I Ixx
W I~xxUxx xx+
In 2Ixx x x x 1W4
Lk IIt Qt mt Rt Qt m1 R1 1
.fl . . I o% 1 1tIIt Im kA tR1 oO 0tk II ,t n4
ty I I f
+ +I
N( oI I o6N " I 1I9
4A 090%I
W 411W II
w WI Io
Wwe II0 J in 4- 1 Ln
*r. K I i Pgo I
kW + I+L
I WI 1WIx x x I
w&l' I x x xI
0.4 .41x xxI Ixxx
Ln+ Ix x x x x 1
-. 1x xxI-g xx x x I
xEx x I Ixxxxxx xx xxxIx
th z. W4 % 0r ML 4111
O ~D. 122
0 X N P, 0N N N 0 P 0 ý M
lNN I fly
0+4
~I
IQ +
510 cc I 1
kI-
m I
em
o.0.4
w
x i I Io I IN
U~ IQX
41 10u I xI
4 I x~x2XI I
xx I-
L. x ~X22XX X X XXX X 1xxxox I
I . . S SI S S * S04x xxxx L
I- ~u~uix mmmmmuxL1/2 iflI-W M WIxx Nx wwaxNxu
o U. 123xxxx
xxx x>xxx
tu 0 . ,u
Lft + + knNi
In ..
N*%DI wN
Sul 1110
LM. Wi M
xSpo I49 I
W L
W
ko I I
u) .. Mi x *I
Sxx I
xUx I
cXX i xxxe
- I XXX XXI1 qW Ixxxxxxxx
I x XXXXXXXXXXX x................. ...
ad XX X X XX Xo w m 0 m oX nmXmXwoXX 0m "t %a P, MOOMM444.4ONO -4
D. 12 4
WO -*ýNNMMMM~MMM'V
(I.ad a NNMMUMVNNN4-"-
OD I l) - It I "
NI INI
g I lo InN t
x 4 In-.n
NINI
W(1 m IIL.. I
ins- +
41)1 ~
""t w4 II.-.
-1 Is-sXXX
E4B ad IIxx x x
IA Ix x xx x I
Ix x xxxx x x In .. .. .. .. . 4+1++++++
. I
DI 125
% .... .... X
aK N-NMWWOM..4p-4m4 %
I~I
NI I f
+ +N I I l
I IIlo it
Ic I ILW
ty P4 1 0
L0 I
OMIW .1+41
10 la II
LMM + x
xgxx xx+11)1* Ix x
4xx Ix
L&A I XXXXXXXXII Ut+xxxxxxxx++4 .. . .. . .*+ * +
j a -CA I. . . . . .
la Ix 4 vtRIt ivm l omk ot 4tlww V n i oa u oL)InLa nt nI
!Z- ONa paIaIi 4 11,'1
o ~ D 126 Ip
-IA A
WT-
*r *I,," A o. P %0
I Il
~q'~I~ me in
W. I
am I
z4 wt I 1w
w II
.1 W I-J~~ L 4
w .. n '4 xi0 x-
x Ix
Ux IS10 m I XXX
xxxxxx
I ~x xxxx xxxxxxaxxxxxxxxxx
.~ .~ . .. .. + + +
04)+ 1, o 4.0 + 44N44+4+ V+44K i7*4v*"*L 00000wwý
D. 12 7
-,~~~~~~~~~ .-' ,vW ý. - ~ -W V. 61 ,.Knv~.~ldi~. ~ U ~ kl W ~ ~
w Q N~% W NI N N mm m m t0 % 0 %M*-
"p.4
I II
04 4
x 09
01~ I 10
x I.I0 w II
2~ I W* m ClxxI m-C 4 WI o1Wxx
YI n I x x x x x I I
OI x x x x x x x w I..fi I Ixx x x x x x x x
z N
Snj
uK IR V O .* a ýNaML 4a
IMU~~ WNMAp~mMe q
m "N"ON 0 NoC.Jmom
.In, 4 I
40~
U,. I Ia3ceP" - 4 11 t
0 WA 1
.j I %A+
al w I-
A0 IA
Y, I AW A InU x4o A. Oi
UX W Iw 4thxxc iIN + A x +O
U A Axxxxxxxxxx x4
A XXX XXXXX
*+C+ ...... ...... ......4K tXixXlttlil t 0 o t Nt Mk
.- . . . . .t
UD . 12
"1w
wI. "-4-qwNNNCwo% wm mmm"
1 9
IN I lN
It A
cc1 ItItI
I mcw un4 +6*
w IA
z a%
x ad
Rexx I
-gx xx Ie9- x9 9x
I ul Ix x x x x+L
144 -op 019 a,'v ai
a LI
2 ~ ~ D 130U
""I I I
NII IN
111
oiin
0i
in.
W +
m II
co II
W- I3 L(d I
W I
I I
W + I
ox I VIIIhI,, I
40 U) xxxxxc -j A., Sxxxx*
2 0 xD. 131
6 4 ,xxx
99II
to I I
14 1
co I
-d I Iw
so mI M1
+ -
u w I
NI I P%
It
LU +
N x Ix x~ I
sM U24 m: g1*,I I.x xx
I x Kx xx x 11X
I x Kx x I
In4x xxxlx.K XX ..... 4-++++
I
acwI44J in I +%4 44 4 4 44.
D.13 2
.4NMWKOMVNOMM MM%wO
D...
Co
inI IN
f-Iml Im
an ka In,I*U~i~W4
in ;.
.1 ce '2
40 NI
-l ~xx
2n a, m94Wxx
-I Ix x x x
D. 133
ýz ?~M
WO ýýNNMcMIMMMM"*
WIn
I N
NIN
*I IN
M (A II
W +
*+ M
MI *tI
w L.xg xGux lo140
xxxx
~x 4 I xx11 IdHOX IXXX
C ..........
+I.4.~e 13 I4*,+C+*+
II IN
In + in
InIn IN
in,+
ke I
ww
(A IAI
N l In
in I -XX4
c i + xKxo A
I Kxxxx
ulI KX Kxxc KKx KX i
&0 I KKKKKKx)xKKxx~x
t* . *... +4+ + .... s.,. +++*++
i14 toK toU to OP .@ 14I e 10Im . . .
34NN Mi M~ U W~
D. 135
J4.I,, N ,'I
.14
+ in
W~- 4
I W
w I
No I I
in +91 1
in,
Ix I
4 1l WI
:1 to IX4.11
44 -gx omxc Ixxx
gU W .41xxC,, Ix x x
w I
O~D 0.36
In In
NII
IlI-
NmI IN
~IA It
th + In9 w I Vol
I-
x IVI4A -J W I
ax
U, I
V. II
to -I xII4. x111
6 o I x x I1 KCx x x II Kxx~x I
LJ 4. Ixx K K x In
MIxxxxxxx IM
.I . .X K I. I. . . . . .iv Ww KLA KO INNO W W
ID. 137
ut
w z- O m M Roa."~mmm*v
WI- I
+ + Inqa~
N
NI
I IV
I I
a . 1M - 6 1 In
41
U- wW
IA
I Ix
*. (A M+I0 xn Ixxxx+u Ixx
I Ix x
I' -I Ifl+
in ~+ xx Ix'a' oxxx 1LXX a xx
N ~xxxxxa
C .. J XXXXXXXx U MI XXxXXX IM+ 4 . .X .XX a
ada
WW NONOWNrn
D. 138
wI.. PNNeMMMM***V**
lyI I N
N I
InI
I I
I I I
- I
cn0 P1I I .-
itn AI I I
~ 41
0 * z WI i
(A IW Ito w I I1
toi' I x I
uI W ~ xxx~xx
ka Ix x IAA KU U,, 4
I 4 x x x IMIA. I4 I l o
LL WLLJ x x
ce In u L o n ta L N C W ~I W ) t L
D. 139
kU 0 U ~ W N W~
In.
NI IN
I CD
IDIa
NI % I k
III
NI
Z I
CII
P% I
x I
to +1 '.41 %
t jV' IN Ix x
2I on M .4x xL Itx ýO x~
* I Ix
'.41 1.4xK xxUI I
U W Ixxxe
o~D 140m
w w
InCNI IN
lyI N
N?%I I N
u INI In
%(n wI CD
IDW
g~ +
w' Ix J cse I~ I
LA wl 1I1
i WI II*x
I IX II Ixxx
I Ixx x
xxxx IIn- I x x xx+1U
t I I InXXX XX
.n i . . ... 44L....
04 Wal II o0iwu j QL AL I Lon, 16 IV-
N In XX K*C W 1m*wN0xXX w LnwL1w. 001t % t-Nff
'.~D 14 1 X
vA IXX~X
* I IN
Iy N
NImI
In + 941
W&U M I IWE .41 L".4i
w I
0% w
IL M
MI XXwme em
I.'. I I
5 4 31( m W W xwx w I r)
L. ~ ~ D 142xx~e
wo -- NNNMMPMMfM1M
*11 N
I I
in +
141
In I
ka I I #
to in
am I
w III
u I x
xxx I
w I 1WxxL .Oc I Ixxx x
Ow I xx x x x xI
WI xxxx Iv
WI KKxxxxxxC
LA. I x x x x x
1111104XX XX 10,I, o I 1 Il,2
r IA4
V, .. . *1 f* n- ~~-t.I. a nM~tlSR5%a iSI.
wb.
UKW-nN-J-w~t
if'~ I.-
In4 fyW
N I IN
W)I I M
C o I I D
+ In%011
gN I-W, +k
in In
I I
16 ingo I f
F x 69
('1 . +10 410
go I Ixxx
OXXXgX
I ~~xxKxxx
de 1~~D 144xxxx
Pci iA Axx xx x I A
UK 14 M NW %%t N- Ai NM O41
w1w
f I I
f I INNI IN
I I
441NI
IInI
inn
P41 INk-wW WI IW
InI
N I4
W I* I10 in II
m I I
C) I
* I
I I
+ In
Wxx IM W Ix I
6 3 I xx x xI
I x x x kx x I
1 xa Ixxxxxxxx xxxxxxCxx x I
a Ic
in.~D 145xx
m~X ~ X X 41
wu
Nw mm*mo.
In +
i )I in
~i
Nm P4 I
*N' WI% 1
CI N I
W + Ix
wt I I*
W I I e
I I11 +. A* 4W
LA 4 .5xxx
- xxW* 4117qIl
5w UIN 10W Nl .U 1NOIUN w
D. 146
66W
I In
UI iN *1
! In
~ *O I
x 09 1-4
j f l k"
N It
0 xO
w Ix
41 II x O
u i.~ 001XX
* MI XXX XIb4.xx-ex xx x x I
wI N I- N0N0N0NinNV %IIz XX XX X Ih0 ,0,a i
U.. 147
uK mm - f oks4
IN Iin. +
LmI I
Ih I
I $
cmI 10 i
Pm R
~14 imp
cc II
3 0 (A +x I
40 1 ~ xI
U)I I H
-j U~ + 4WI so I x x x1
6 4 1 Ix x
Ln 4 XKXXXXX I
CI Ixt x x
4 + + + + 4 +4 +4 + + 4 + 4. 4 4. 4 4 +4
D.4 148m m41I
i1.. 6'4 Ln 0
I.tl UKLn 6' W't U'J A~ 41L ) P-0N i
M=1 NMONNNw tmwrmm
C)
IN I
I w
(A
LLI-
In I
Wx x I i
1 Ixx xAn wI Ixx x
Al el Iv¶x I
Wf+ KKxxx +0p "I XXxX x x x I P
I xxxxxxxxxxx
+ + +X ++.. . .. .4 . .. .
~~~ to IQXKKim Wt 1 otol %t o,%b4fýIR 21
4 a wn in K0XC~X
x I n K0 Kn V) W kXXXXKNX N
g4 9
IlkM u2u f
L I U
NI InN I I C
f41
INII
'a ko Ln+
w II
ua II
11 )01+i
InCx +11-0
uw -, I-x
in .4 xn emxg x +
U1.4 to
I1 0
0*
NI'
N II
I. IA9NI
&o I
I InII IA
in I
W) +
*W LI 1W
W Il 0 16toWIUO GeIwI01 I In+L0 P-4
+J +
x I
I 11
0 IJx4 x Kxx 41 U-41 x xxxxx xxx K
I xxxxx x~It')L6xxxxxxx
. D .. . 151 .....
NwV)Niioc g inswq)~ ~ ~
N 0I yv ac IN(44 1cc N t IN
Iw w
N I I t
IIN IN
th i Im
In I*
IN a
'a m .
ID I-j
0 ILe2 9
0 K I' IV%
KPcXXxXx
in w *o 4e u~ ti aog* ees
c j t
2 ~ ~ D 152 xx+
aUi
U.
NII
kI In
NI
N I
ItI
W c I I
Im- ) n + + uW Iko I
10I
) 0 1 1W
Z CA~W I IW
.1 - + +Iu
I IIad~ IQ++6
0 I 1I1
go I
zIC >
414~'WI to
xJ x x4. I x xxc
Ny 4( I x ~xL*J Lfl+ x 41.+11
x .xIx)xxx
- xxxx01 W I wwxxxc 4j ta +x.xxxx(
M luX~ Ixxx xmL. 4C Ixxxx.x~xx
-xxx Ix
4 ~xxx~xxx~xxx> xxxxxxxxxe x I"
I xxxxxxxx~xxx II~. I xxxxxxxxCxCxxx I
4
fQ 0eI~ IRt %I Q0t RI R Aq ttQt pmtR t4 o N Q m ftt'.W o a m 0 ? m 00 ft0
. . . . . . . . . . . . .
w~~D 153mLa n Iot
WW\N NI ~~Q~~%NIN P,)X OC lJ Dt W0P
*~*~.-. X~Z~' ~ ~'~ ~*::;' ~ . **Z<~-
a -IJZ
4I f
NI IN'
I
N I 44.
-I
.v to I
am 041
JA go Ia,- In
dc .Mb in + Iti
z~ in I
Ln +
In +0- m InM I
4A JL M I in
CA I R +
(A ..J inCCa
L C w I x xh
0 o nI 1Wxx
24 I c I Ix xxow xx xx x
1A. I Ixx xx xo I )C xx xx x Ix
++C ......... 4+++++++++CC+
toz ~mNt om 4mli mRt4 aVRI
m mf" m&J V -0 V I in k"-.fl Ia lN bl w to %D %D
D. 154
NyI I
NyI IN c1tyI II lII I
II
14 1 1 IN
0N WI 1 c
I wo
acI I
I CA
owmI0 I I0 W IxJ U U9
oA ki +I'
). I l xU. L
N 5.JXXXXX;X
i xx xx x I
xxx +xUxxxxz -zxxx
D.xxxxx 15
I Sxxxx
1~~~~ XXXXXXXXX Ii
wu~
IN I
N II IN
NI IN
P4 I
Ii I
+ I
W 1
InI
ev4 WI 1W
+ IO
P% I
IN
.. 0 16,4
P% I
O W I
0 I o
IN x
IN XýXx
LI w xxX X Ix
0x xxx IIY
xxxxxxxx I
XXXXXXXXXXX4 xU. I xxxxxxxxxxxx<
+ + + 14+++
WW ý. LhN N 61 N P 6 mNN f- n N m PU) V
z~~~~. 56 mmmI tc ni
IC 00400000a0
ow
Li. -4W ~ w N
L& "
N! INNI IN
I I
NII
Iwl,
Mg I I
1*0 In
1ý I I
(A IIw
cc w.
x I N
x xm0 tm + .,
KU ~I IxdU Ix
W wexxxxo lo+ S'IIx Iii
-C UUxI.,xxx I.
N Ixxxxxxxx I.n . .
> ..X XX X .4W i nk nt nt
*~D 157
* I a :L
LIZ I
wI
I II N
I I
-Imi ION
Lo I I
1A IU'w I
'a InmP.W 01
+E +I I
IA I I
w I
w WI I111O 1II
5x x~' + In
xxx I wt
$11 I Iin IW Ix
.1 41 CIAxo
U I xxxC xx I-1x x x x x x I iXX xxxxxxx x em
W MI %NONXfXXX NOION N0(A 1, m %X XX X I
*11
D. 158XX
U.
I~I
+ 'i
IQ M NoI IN
IA +
I
at 1 IW
m k i P-in+
U W P4
Ig +
CA jIn in
IA -
in I xWxx In
In .J 1194 402 g f4%g 1 11%x
Sxxx I
in xxx'4xxi
to to 11111 Wm IV%%i 4N st t mMt %fqi -J 0 s t
S l ONIONI6L M I O
z in4 I n wo w wo PXPXK = 0
D. 15
M 0 -MN M o L M-nNO w 0 m~
N) IIN
NIINII
L I I
IN a
Ic +
*U) W Iw -
>I It
inn +4 + o
U) I
wx x I
In w J MI+4-u
0xxx I
K Ix x xx x ILL x x x x x x I
tR IIU
WD. 160
I KX)CX)t
0.
w tI
9~ I I
NyI INNI
In +
1411
IQI
asp LACC
0 &J WI I Ch
go w I
1(4 .4 pp,
N Ix
c A I+ Cmxx 4I0 o f xOx I mL c WX XX X I-
'4 IxxxUxxxxx 1
it) + UCx xx x W)o a. ml Imx
(VX X XX X X
v4 -J IflC CliixoXXXXXXXXX 4-
D. 161
4K pq.*m V IA N NN.4
In. 4 imNI wi
cc Is
20 o I IN b4 I IN
in II
uiin
ke IN ~+
M P4 IB
u UK P141
x Bt I
Il Bin, in,+ in
u I Bx
c - I Ix
I Lx I B
r.. _j V 4ixx
Bc IIxxx
IJ ox x x x x x Bx x xx x ILA I ~ x x x x x x
in in toI ot ol 4a ot & oI R0t ok :1
WW U) NI NI N N o
#n * %* n w n in %aVC toyX X XX X ON4I0M 4 1 0D.16
UK j WPINS M M M MM MV
0.
IN:4
N I I v
1+ h
t4
WLn ka I
~. In
Mi' II
u 1W
IN.
IA k I 4
x "t
i In -J 14 +1M~ I x x x Ien
> AIX x xx x x x I
41 x 4XXXu~xx
4It tI tI1 1t".tItt1
MW M NWM
I i IiI1
Appendix E: Overlap Percentages for Variance Distributions
The percentages in the table represent the percent of
the observations that were above the percentile for the
'low' distribution and the percent of the observations that
4 were below the percentile for the 'high' distribution.
LOW OBS HI OBSRUN# RUN#
RUN 1 2 3 4 5' 6 7 8 9 RUN 1 2 3 4 5 6 7 8 9
05 52 88 90 58 88 92 64 90 92 95 50 74 88 48 72,88 58 80 9010. 52 84 78 52 84 so 60 86 82.. 90 44 66-88 48 72 88 52 76 8815,32 74 66.,46 76,64 50 78 72 ,85 32 64 74' 46 70 80 48 72 82
------------------------------------ -----------------------------
RUNIO 1il 12 13 14 15 16 17' 18 RUNIO 11 12 13 14 15 16 17 18
05 74 92 92 76 92 92 82 92 92 95 78 84 90 72 82 92 78 84 9410 66 88 88 72 88 88 74 88 86 90 66 78 88 72 82 88 76-80 8815 58 84 78 62 82 80 66 82 80 85 66 72 82 64 78 82 72 80 82
RUN19 20 21 22 23 24 25 26 27 RUN19 20 21 22 23 24 25 26 27
05 3 61 69 7 73 76 16 79 79 95 3 57 72 10 65 77 17 71 8010 2 43 56 4 54 66 7 53 73 90 0 47 53 4 57 63 10 60 7015 2 30 43 3 44 59 6 50 63 85 0 33 41 1 51 54 4 53 62
E. 1
LOW OBS HI OBSRUN# RUN#
RUN28 29 30 31 32 33 34 35 36 RUN28 29 30 31 32 33 34 35 36
05 38 84 *83 51 87 88 60 88 88 95 47 80 85 51 86 90 58 88 9010 31 .70 76 43 77 78 49 80 80 90 32 69 76 41 73 77 46 78 7915 23 60 68 35 66 75 43 68 76 85 25 58 69 31 62 74 36 67 76
RUN37 38 39 40 41 42 43 44 45 RUN37 38 39 40'41 42 43 44 45
05 0 31 36 0 51 55 2 61 65 95 0 28 42 0 41 60 2 51 6810 0 19 24 0 39 38 1 45 48 90. 0 20 28 0 33 43 0 42 5315 0 7 18 0 22 28 0 31 37 85. 0 3 18 0 ,27 35 0 34 41
------------------------------------- ------------------------------
RUN46 47 48 49 50 51 52 53 54' RUN46 47 48 49 50 51 52 53 54
05 14 69 71 25 80 81 37 86 85 95 14 61 77 30 69 83 41 75 86.10 6 57 59 14 65 67 22 72 72 90 8 54 63 16 63 72 25 66 7615 4 42 48 9 52 59 17 57 63 85 5 47 55 10 55 62 17 59 64
E. 2
,11
Bibliography
1. Banks, Jerry and John S. Carson, II. Discrete-EventSystem Simulation, Englewood Cliffs: Prentice-Hall,Inc.,, 1984.
2. Beverly, John G., Frank, J. Bonello and William I.Davisson, "Economic and, 'Financial Perspectives onUncertainty in Aerospace Contracting," Procee'dihls of 4
Manasvment,'of Rii_ and Uncertainty in the Acquisitionof Major Programs. 264-269. United States Air ForceAcademy 00, Feb 1981.
3. Bryan, Noreen and Rolf Clark, 'Is Cost Growth BeingReinforced?" Journal of Defense Systems Af:quisitionManaRement, 4: T105-11r. (Spring 1981).
4. Defense Systems Management College. Risk AssessmentTechniques, Washington DCi Government PrintingOffice, Jul •983. ...
5. Devote, Jay L. Probability and Statistics forPunbineering and "he Sciences. M'onterey:l Brooks/ColePublishing Company, 1982.
6. Dixon, W.J. et al (eds). AMDý Statistical SoftwareManual. Berkeley: University of California Press,1983.
7. Fatkin, A. "Cost Uncertainty/Risk Analysis,"Directorate of Cost and Management Analysis, DCSComptroller, HQ Air Force Systems Command, Andrews AFBMD, Jun 1981.
8. Howard, Truman W., III, "Methodology for DevelopingTotal Risk Assessing Cost Estimate (TRACE)," UnitedStates Army Research and Development Command, DRDMI-DC,Redstone Arsenal AL.
9. Kazanowski, A.D. "A antitative Methodology forE~stimating Total System Cost Risk," Proceedings of the1983 Defense Risk and Uncertainty Workshop. 135-163.Defense Systems Management College, Fort Belvoir VA,Jul 1983.
[0. Ingalls, Edward G. and Peter R. Stoeffel. "RiskAssessment for Defense Acquisition Management,i"
Proceedings of the 1983 Defense Risk and UncertaintyWorkshop. 55-64. Defense Systems Management College,Port Belvoir VA, Jul 1983.
262
-12
11. McNichols, Gerald R. "A Procodure for Cost-riskAssessment," Proceedings of Management of Risk andUncertainty in the Acquisition Of Major Programs. 249-257. United States Air Force Academy CO, Feb 1981.
12n-.... O the Treatment of Uncertainty in ParametricCosting, DS Dissertation, School of Engineering andApplied Science, The George Washington University,Washington DC, Feb 1976.
13. Rowe A.J. and I.A. Sominers. "History of Risk andUncertainty Research in DoD," Proceedings of the 1983Defense Risk and Uncertai'.ty Workshop. 6-19. DefenseSystems Management College, Fort Belvoir VA, Jul 1983.
14. "Methods to Evaluate, Measure and Predict CostOverruns," Proceedings of Management Of Risk andUncertainty in the Acquisition of Major Progzrams.26-71. United States Air Force Academy CO, Feb 1981.
15. "Taxonomic Concepts Panel Summary," Proceedings ofMana emeit of Risk and Uncertainty in the Acquisitionof Major Programs. 175.1-175,2. United States AirForce Academy CO, Feb 1981.
16, Wilder, J.J. and R.L. Black. "Approximately BoundedRisk Regions," Proceedings of the 1983 Defense Riskard Uncertainty Workshop. Defense Systems Management
College, Fort Belvoir VA, Jul 1983.
17.--------. "Determining Cost Uncertainty In Bottoms-UpE•stimating," Proceedings of the 1982 FederalAcquistion Research Symposium. A-49 - A-57. GeorgeWashington Utniversity, Washington DC, May 1982.
t8. -----. "Using Momemts in Cost Risk Analysis,"Ilroceedins1(i of Manaspeient of Risk and Uncertaintyin the Acquisition of Major Programs. 193-202.United States Air Force Academy CO, Feb 1981.
11). Winkler, Robert L. and William H. Hays. Statatistics:Probability, Inference and Decision. New York: Holt,Rinehart and Winston, Inc., 1975.
20. Worm, George N. Standardized Factors fur RiskA An I . lysis. Contiact F3361 -.83-K-5075. Air Force111 tsi ness Research Management L(.enter, Wr ght -Patterson.I13" OH.t, Jan 1984.
.21. - . .. . _ d Risk Analysis with 1)- iiJen ei( Anon! oC(wo t C,)i)nent . Contrnct "336 15-81-C-- 3H8. AirI- orr. e Busi ness Re:ea rch Managem(,nt Ceni er, Wright-K't terL;ton AFH 011, Nov 1981
. 2 0 -3- -
822. Viewgraph from AFIT/LS Sys 227 Class, Jun 83.
I
.1)
i
V
J
'2"
• 264
Vita
Lieutenant Robert E. Devaney was born on 28 January
1960 in Brighton, Massachusetts. He graduated from high
school in Chelmsford, Massachusetts, in 1978, and attended
,* the United States Air Force Academy from which he received
the degree of Bachelor of Science in Economics in June 1982.
Upon graduation, Lieutenant Devaney was assigned as a
Financial Manager to the Aeronautical Equipment System
Program Office, Aeronautical Systems Division, Wright-
Patterson AFB, OH, until entering the School of Systems and
Logistics, Air Force Institute of Technology, in May 1984.
Lieutenant Devaney is married to the former Cathleen Ann
Sadlak and has three daughters: Christine, Carolyn and
Mary.
Permanent Address: 19846 Burleigh
Yorba Linda, CA 92686
2
•' 265~
'm ~ * ~ * * * * ~*A *. ~ * -*-~* '' ~ p* t A. ~
Vita
Captain Philip T. Popovich was born on 16 October 1956
in Wamhington D.C. He graduated from high school in
Brecksville, Ohio in 1974 and attended the United States Air
Force Academy from which he received the degree of Bachelor
of Science in May 1978. Upon graduation, he was assigned to
FE, Warren AFB, WY as a missile launch officer. He
completed a Masters of Business Administration degree
through the Minuteman Education Program and the Univers:
of Wyoming. In October 1982, Captain Popovich was
reassigned as a cost analyst/financial manager to the
Aeronautical Equipment System Program Office, Aeronautical
Systems Division, Wright-Patterson AFB, OH, where he worked
until entering the School of Systems and Logistics, Air
Force Institute of Technology in May 1984.
Permanent Address: 8637 Hollis Lane
Brecksville, OH 44141.
266
%
UNIASIIE -Best AvailableCoISCURITY CLASSIFICATION OF THIS PAGE AA%04k:1 -
REPORT DOCUMENTATION PAGEI& P!PORT S UITY CLASSIFICATION lb. RESTRICTIVE MARKINGS
2&. S9CUA.fY CLASSIFICATION AUTHORITY 3. DISTRISUTION/AVAI LABILITY OF REPORT____________________________ Approved for public release;
2b. OECLASSIPICATION/DOWNGRAOING SCHEDULE distribution unlimited.
4. PIERFORMI1NG ORGANIZATION REPORT NUMBER(S) 5. MONITORING ORGANiZATION REPOR I NUMBER(S)
AFIT/GSM/LSQ/85S8-17Sc& NAME OF PERFORMING ORGANIZATION b. OFFICE SYMBO0L 7*. NAME OF MONITORING ORGANIZATION
School of Systems (if applicable)
adid Logistics AFIT LSR _________________
Sc. ADDRESS (City, State and ZIP Code) -7b. ADDRESS (City, State and ZIP Code)
Air Force Institute of TechnologyWright-Patterson~ AFB. Ohio 45433 __________________
Ue NAME OF FUINDINGISPONSORING Sb. OFFICE SYMBOL 9. PROCUREMENT INSTRUMENT IDENTIFICATION NUMBERORGANIZATION (if applicable)
11c. ADDRESS (City. State and ZIP Code) 10. SOURCE OF FUNDING NOS.
PROGRAM PROJECT TASK WORK UNITELEMENT NO . NO. NO, NO.
11. TITLE (include Security Classification)
12. PERSONAL AUTHOR(S)Robert E. Devaney, B. S. f, iLt 0 USAFPhililD T. Popovich. m.B .A CnUA
134L TYPE 1 Q RfPORT* 13b. TIME COVERED 14. DATg OF REPORT (Yr., Mo., Day) 15. PAGE COUNTMS Thesis IF ROM _____ TO t___9__epeme_7 7 7
IS. SUPPLEMENTARY NOTATION
17. COSATI CODES 18. SUBJECT TERMS (Contimnue on re verse it necessary and identify by block number)FIELD GROUP SUR. GR. AcustoCtAalss CotE im e,
o 01 Covariance,.Risk
19. ABSTRACT (Continue on reverse If necessary and identify by block number)
* Titles AN EVALUATION OF COMPONENT DEPENDENCE IN COST-RISK ASSESSMENT
* Thesis Chairmans Richard L. MurphyAssistant Professor of Quantitative Management
Jý3"d Z~PWI rloae:LAW APRfl 1
D*= fat ftevoarch and Profeweml" 2.topohniC,- ~Air Powe. Iantituie at Technoloy jAI,-
Wrlghi-Pattenesa A'B ON1 4s4M
20. DISTRIBUTION/AVAILASILITY OF ABSTRACT 21. ABSTRACT SECURITY CLASSIFICATION
UNCLASSIPIED/UNLIMITED 11SAME AS RPT, 0 DTIC USERS 0UNCLASSIFIED
22a NAME OP RESPONSIB~LE INDIVIDUAL 22b. TELEPHONE NUMBER 22c. OFFICE SYMBOLRichard L. Murphy 513-255-8410 AFIT/LSQ
00 FORM 1473,83 APR EDI1TION OF I JAN 73 IS OBSOLETE. NL8SFESECURITY CLASSIFICATION OF THIS PAGE
Best Available CopyUNCLASSIFIED
SECURITY CLASSIFICATION OF THIS PAGE
Abstract
This r project developed a cost-risk assessment methodthat incorporated the effects of cost dependency between componentsin a system. The method uses program personnel's subjective assess-ments of component dependency as inputs. A simulation model wasdeveloped and employed to test the method under various levels ofcomponent dependence strength and direction, estimation error, andsystem size.
The analysis was accomplished by performing sensitivity analysison the predictive capabilities of the cost-risk assessing method.Results indicate that the model has strong predictive capabilitywhen component size is small and when the direction of the compo-nent dependencies is mixed. It was also determined that the useof component dependency assessments produced more realistic totalsystem cost variances than those produced under the assumption ofcomponent cost independence.
U,S.Oovevnment PVIMtIng Office, 198-- 646.047/20357 UNCLASSIFIEDSECURITY CLASSIFICATION OF THIS PACE
