Benchmarking Options: Frontier and Component Cost Analysis
26 July 2012
1 Introduction 1
1.1 Background 1
1.2 Benchmarking Approaches 1
1.3 Conclusions 2
2 Data Analysis 5
2.1 Available Data 5
2.2 Quality Adjustments 6
3 Benchmarking Data 7
3.1 Data Presentation 7
3.2 Component Cost Benchmarking 7
3.3 Frontier Analysis 8
4 Discussion and Recommendations 10
4.1 Limitations of the Analysis 10
4.2 Approach to Defining Lower Bound 10
4.3 Recommended Approach 11
Annex 1 Identifying Outliers and the Implications for Benchmarking 13
Annex 2 Calculating Percentiles 29
1
1 Introduction
1.1 Background This note examines a number of implementation issues associated with public transport
tender price benchmarking. The purpose of benchmarking is to obtain an estimate of
what would be an achievable and competitive market price1 for a unit that is procured
via direct negotiation rather than competitive tender. It would also inform cost re‐sets
during the term of contracts. The benchmark price, or price range, is an estimate of the
price that would result if the unit was competitively tendered. Using this estimate to
inform price negotiation is consistent with NZTA’s statutory requirement to ensure it
obtains the best value for the money spent.
The concern expressed by service providers is that, if the benchmark method produces a
single estimate of a competitive market price, this might give little freedom of
movement to negotiations, particularly as the negotiated tender will not be the same in
all respects as the routes used for obtaining a benchmark price. In this context the PTOM
Working Group recommended initially that the benchmark approach produce a range
of market prices between upper and lower bounds, rather than a single price.
This provides the context for this paper of how to (1) provide information to inform
negotiations that pushes towards a price representing best value for the money spent
while (2) retaining some flexibility to take account of route‐specific circumstances
affecting costs.
The other issue raised has been that of the extent to which data outliers will affect the
benchmarking exercise, leading to an estimate of the competitive market price that is too
low or too high.
This note examines:
what set of benchmarking information to provide to decision makers;
whether data outliers are a problem and if so, how to deal with them.
1.2 Benchmarking Approaches The PTOM Working Group proposed that two benchmarking methodologies are used
in combination. These would be used by an independent analyst working with data
from competitive bids to produce a benchmarking report that would inform NZTA and
councils. The methodologies are:
• Component Cost Benchmarking. This approach involves bidders for
competitive tenders providing a breakdown of their bid price into dollars per
kilometre, dollars per hour, dollars per bus (component costs) and total cost
(annual gross cost). Benchmarking could include analysis of averages, extremes
1 We define a “market price” as any conforming tender offered as a bid in the tender market, but a
“competitive market price” as a winning bid price; other bids in the same tender round are, by
definition, not competitive.
2
(eg lowest) and percentiles within the range of component costs.
• Lowest Practically Achievable Cost Benchmarking (also known as “Frontier”
Analysis or Data Envelopment Analysis). This approach uses bid cost data and
the defined “outputs” (kilometres, in‐service hours, bus numbers) to assess the
estimated maximum of each output that can be produced for a given level of
input. Bids can then be compared in terms of their relative efficiency against this
benchmark.
The benchmarking options were analysed in the light of assumed constraints to the
availability of data, which would apply particularly immediately after initial
implementation when there will be few data with which to develop benchmarks.
Analysis included consideration of the use of larger sets of competitive bid data by
including all compliant bids in the analysis rather than winning bids only.
The Working Group initially proposed that the benchmarking report would construct a
“lower bound” and an “upper bound” that would be used to define acceptable contract
costs. In this report we examine approaches to setting bounds but find that, if using a
dataset of bid data with more than just winning bids, it is difficult to constrain upper
bounds in a way that is consistent with value for money for funders.
Data outliers will not generally be a problem and, in most circumstances, can be
identified in initial data screening. The issue is likely to arise largely if:
Some data used for benchmarking are from sites with very different
characteristics from that being tendered; or
Non‐winning bid data are included and are used to construct an upper bound of
acceptable bids. However, we discuss below how the focus on achieving value
for money is likely to render this issue irrelevant.
1.3 Conclusions We recommend that frontier (data envelopment) analysis is the primary analytical tool
but that additional information is provided as context. We recommend that:
1. A dataset is compiled of bid data from competitive tenders for routes (and units)
that are deemed relevant to the route being negotiated. This should include all
compliant bids. It should include:
a. the route information—kilometres, in‐service hours and peak vehicle
requirement;
b. total bid price;
c. components costs for the bid—$/km, $/hour and $/bus.
2. The dataset of bid data should be screened initially to identify any outliers. This
would be done through a “visual” scanning of the data followed by an analysis
of specific data points identified as potential outliers, to better understand
3
whether they should be included or not.
3. Frontier analysis is used to create a lower bound of costs for a contract
negotiation. It should be based on a dataset of:
a. winning bids; and
b. complying bids that are lower cost than the winning bids;
Group tender bids should be included only in aggregate, ie as though they are
representing a single bid for a very large route.2
4. Additional information is provided to funders that allows them to better
understand the range of winning bid data used in frontier analysis. This might
include information on the estimated costs at different percentiles in the results
of frontier analysis of winning bids. For example this would show the price at
which, say, 25% of winning bids would be more efficient (the 75th percentile in
Figure 1 below).
Figure 1 Possible data from frontier analysis
Frontier price Frontier price
75th percentile 75th percentile
Highest price Highest price
$0
$500,000
$1,000,000
$1,500,000
$2,000,000
$2,500,000
$3,000,000
$3,500,000
$4,000,000
Winning bids only Winning bids plus lower‐priced compliant bids
$3,368,056
$3,459,836
3,768,984 $3,749,556
$3,479,294
$3,350,694
5. Information on component costs for all compliant bids should be supplied also,
including averages for component costs and the average mix of costs for all
compliant bids, eg hour‐related costs as a percentage of the total; kilometre‐
related costs as a percentage of the total etc. An example of data presentation
from the component cost analysis is given in Figure 2. It shows the $/km data at
different percentiles and represents the percentage of bids that would have costs
below the amount on the y‐axis. For any new negotiation, it would provide
information on where the component costs might fit within the range of costs
analysed.
2 If a group tender bid won for 4 units (routes) the data (bid price, kilometres, hours and bus numbers)
for all four should be added together as though it was one large route
4
Figure 2 Component Cost Analysis ‐ Percentiles
1.75
1.80
1.85
1.90
1.95
2.00
2.05
2.10
2.15
0 10 20 30 40 50 60 70 80 90 100
$/km
Percentile
5
2 Data Analysis
2.1 Available Data The PTOM Working Group has recommended two benchmarking approaches are used
in combination: (1) Component Cost Benchmarking and (2) Lowest Practically
Achievable Cost Benchmarking, also known as frontier analysis. The recommendations
of the Working Group have been made in the absence of analysis of the implications of
the different approaches using actual tender bid data.
We have used the limited data sets that have been identified to examine the implications
of using winning bids only versus a wider set of data with all compliant bids, and the
associated issue of outliers. The analysis of data is included in Annex 1. It includes:
A limited set of data for commercial bus tender rounds provided by councils.
These data have information on bid prices only and no accompanying data on
kilometre length, in‐service hours or numbers of buses; and
A larger dataset on school bus bids from the Ministry of Education that include
price and kilometres.
Neither set allows an analysis of the full methodologies as they are missing data on in‐
service hours or peak bus requirements. Nevertheless some conclusions can be drawn
on the implications of outliers.
Some tenders are won by bids that do not have the lowest costs because they
include quality premiums – we explore this issue further below (Section 2.2);
Outliers in data can generally be identified relatively rapidly through visual
analysis and excluded through more specific analysis of the circumstances.
Those relating to non‐conforming bids can be identified easily, for example;
Group bids, in which a single bidder has offered a uniform price (eg in
$/kilometre) across several separate units or routes, can result in misleading
benchmark information if the results for some routes only are included in the
benchmark analysis. They may imply that lower costs can be achieved for some
routes than is achievable in practice. Group bids should only be included to the
extent that all the routes included in the group bid are suitable for
benchmarking and they should be included as an aggregate;3
Using percentiles as a way to construct an (upper) bound for possible prices can
be misleading. Analysis of the implications of a 25th percentile approach (25 of
bid prices would be expected to be lower cost) led to results, even using winning
bids only in analysis, in which the suggested prices were 56‐97% above the
lowest price bids.
3 If a group tender bid won for 4 units (routes) the data (bid price, kilometres, hours and bus numbers)
for all four should be added together as though it was one large route
6
Overall the analysis suggests that outliers are not a significant problem and that it is not
possible to define a percentile that might be used to represent a suitable range for bids
prices.
2.2 Quality Adjustments Evaluation of bids takes account of quality as well as price. In achieving best value for
money, funders take account of the trade‐off between the two and will sometimes
choose higher cost bids that offer more quality than is required. It is important to take
this into account in any benchmark analysis. Ideally this is done through stripping out
the quality adjustments so that the benchmark data represent prices for bids that offer
no quality premium. Otherwise, the inclusion of data from winning bids that
incorporate a quality adjustment may suggest that the competitive market price is
higher than it would be to run a service that only just met the quality requirements.
There are two possible approaches to take account of quality adjustments:
To adjust downwards the bid prices that include a quality premium; or
To include data from compliant bids that are lower cost than the winning bids.
Councils can use a supplier quality premium (SQP) in their supplier selection process.
The SQP is the additional amount that the organisation is prepared to pay for a higher‐
quality supplier. But it is a willingness to pay estimate rather than an estimate of the
costs of supply of quality. If a supplier can provide a quality service for a low cost,
subtracting the SQP from the bid price may over‐estimate its cost of supply and result in
an adjusted bid price that is lower than achievable. For example, say a bidder with a
quality premium that the council values at $100,000 bids for a route with a price of $1
million. It may have only cost the bidder $50,000 to achieve this quality premium and
another bidder may have submitted a bid without a premium at $940,000. Using an
adjusted winning bid price ($1 million minus $100,000) results in a lower price estimate
than a competitive market price would be if no bidders had included a quality
premium.
A preferable approach is to use actual bid prices in the form of compliant bid data that
are lower than the winning bids.
The issue of quality adjustments is more problematic early in the process where there
are few data to use. As a larger dataset develops there will be more data without quality
adjustments and quality‐adjusted bid prices are unlikely to be setting the benchmark
price.
7
3 Benchmarking Data
3.1 Data Presentation The data presentation issue is that funders of public transport are seeking to identify a
low benchmark price that is an estimate of the competitive market price for the route, or
group of routes. However, the benchmark methodologies may produce a single price
estimate. Funders are likely to require some room for movement around such a price if a
price is to be agreed with a service provider. This takes account particularly of the fact
that the benchmarking exercise will be using data pertaining to routes that, while
similar, will not be exactly the same as that being negotiated. We discuss these issues
relating to the two benchmarking approaches below.
3.2 Component Cost Benchmarking Component cost benchmarking uses data provided by competitive bidders on the costs
of individual components: $/km, $/bus and $/hour. The key problem with this approach
is that of how to combine these data, ie it is unlikely that any route could achieve the
lowest value in all cost categories.
Given this, information might be provided on the range of component costs and their
averages. Figure 3 is an example which uses the data in Table 5 (excluding outliers) to
shows the $/km data at different percentiles. This represents the percentage of bids that
would have costs below the amount on the y‐axis. For any new negotiation, it would
provide information on where the component costs might fit within the range of costs
analysed. Simple averages might be useful also.
Figure 3 Component Cost Analysis ‐ Percentiles
1.75
1.80
1.85
1.90
1.95
2.00
2.05
2.10
2.15
0 10 20 30 40 50 60 70 80 90 100
$/km
Percentile
Developing relationships between the component costs, eg identifying the way in which
$/km changed with changes in $/bus is likely to require a large dataset to yield any
8
statistically significant relationships. This may become possible as the benchmarking
exercise develops over time but there will not be sufficient data in the short term.
In addition to the question of how to present the data there is an additional question of
the data to include in the analysis. Excluding outliers is an obvious first step, however
the discussion in earlier sections also raised questions relating to group tender bids.
Group tender bids involve averaging over a number of different routes and the data are
only meaningful at this aggregate level, ie by using the multi‐route average component
costs.
3.3 Frontier Analysis Frontier analysis has advantages over component cost analysis because it uses the data
in a way that takes account of trade‐offs between the cost components. The resulting
frontier is expected to be an achievable competitive market price for a route grouping.
However, the result is provided as a single expected price rather than a range.
Ways in which a range might be presented can include changing which data are
included in the analysis plus presenting the results in terms of percentiles. The set of
information might include:
a frontier price using winning bids only;
a frontier price using winning bids plus lower‐priced compliant bids (where
winning bids included quality premiums);
a price based on the winning bid that was furthest from the frontier;
a price based on some percentile of all winning bids.
These are explained further below. We start with a set of hypothetical data as shown in
Table 1.
Table 1 Hypothetical bid data
Route Km Hr PVR Total Cost Efficiency(a) Efficiency (b)
1 1,750,000 100,000 50 8,500,000 97.4% 96.5% 1a 1,750,000 100,000 50 8,200,000 100.0% 2 300,000 17,500 8 1,400,000 99.5% 98.7% 3 700,000 45,000 13 3,000,000 100.0% 100.0% 4 1,500,000 115,000 34 8,000,000 96.4% 95.9% 5 1,800,000 135,000 47 9,500,000 100.0% 97.4% 5a 1,800,000 135,000 47 9,250,000 100.0% 6 725,000 30,000 9 2,800,000 100.0% 100.0% 7 600,000 30,000 15 2,600,000 100.0% 100.0% 8 1,650,000 100,000 60 8,750,000 100.0% 100.0% 9 1,200,000 35,000 20 5,400,000 89.4% 89.4% 10 950,000 30,000 23 4,200,000 97.3% 97.3%
Negotiated 750,000 50,000 15 3,368,056 3,350,694
Notes: (a) Winning bids only; (b) Including lower‐priced, compliant, non‐winning bids
It shows data for 10 routes. The un‐shaded entries are for winning bids; the shaded
entries are the lower‐priced, compliant, but non‐winning bids.
9
The first efficiency column shows the efficiency of the winning bids only. Using
these data only, the frontier (100% efficiency) is set from routes 3, 5, 6, 7 and 8;
the market price for the negotiated route is estimated to be $3,368,056.
The second efficiency column includes the lower‐priced compliant bids. Here
the frontier is set by the new data for routes 1 and 5 plus the original winning
bids for routes 3, 6, 7 and 8. The market price for the negotiated route is now
estimated to be slightly lower at $3,350,694.
The next pieces of information that can be obtained are at the top end of the range.
Using the first efficiency column, the least efficient winning bid has an efficiency of
89.4%. This means that, to be on the frontier, its bid price would need to have been
89.4% of what it was. We can use this value to estimate what price the negotiated bid
could be to be similarly inefficient. This is estimated as $3,368,056/0.894 = $3,768,984.
The value is slightly different if we use the alternative frontier specification including
the lower‐priced compliant bids. Here the least efficient route is 89.4% efficient also and
the similarly inefficient negotiated price would be $3,749,556.
Values can be estimated also using a percentile amongst all the bids. The 75th percentile
of efficiencies (25% of bids have a higher efficiency), using winning bids only is 97.3%
and results in a negotiated upper price of $3,459,836. The same analysis using the low
price bids also is 96.3% and $3,479,294.
Data for Negotiations
Using this kind of analysis, data could be presented as an input to negotiations in the
following form (Figure 4). This provides a range of data as input. Data could be
provided for a range of percentiles, as in Figure 3 for component costs.
Figure 4 Possible data from frontier analysis
Frontier price Frontier price
75th percentile 75th percentile
Highest price Highest price
$0
$500,000
$1,000,000
$1,500,000
$2,000,000
$2,500,000
$3,000,000
$3,500,000
$4,000,000
Winning bids only Winning bids plus lower‐priced compliant bids
$3,368,056
$3,459,836
3,768,984 $3,749,556
$3,479,294
$3,350,694
10
4 Discussion and Recommendations
4.1 Limitations of the Analysis The analysis of options in this report is limited by the availability of data. Ideally we
would have been able to test the different approaches using actual data from a large
number of urban bus route bids. In the absence of such data we have experimented with
a limited set of bid price data, a large dataset pertaining to school bus routes but that
was limited to cost information in one dimension only: $/km, and hypothetical data.
If the approaches discussed and recommended here are used, it would be useful to
review their workability soon after commencement. This would enable adjustments to
be made, taking account of any practical difficulties that arose.
4.2 Approach to Defining Lower Bound The lower bound represents the estimate of the achievable and competitive market
price.
4.2.1 Frontier as a Lower Bound Despite the limitations of the analysis, of the options considered, we suggest that the
best method for setting a low benchmark is frontier analysis. It has the advantage that it
finds an achievable combination of costs as it uses actual outputs. For example, it
establishes whether low $/km routes tend to be high $/bus routes also. In contrast, a
methodology that uses percentiles of cost components uses a combination of costs from
different bids and does not take account of the trade‐offs between outputs that operators
face when developing their bid.
The frontier analysis could be undertaken using all bids, winning bids only or winning
bids and lower‐priced compliant bids. As winning bids would be expected to be low
cost, there may be no difference from including all bids. However, winning bids may
include those that have been accepted because of higher quality elements, ie a quality
premium is being paid (see discussion Section 2.2). Under these circumstances the
resulting frontier may be at a higher cost than a “competitive market price” based on a
just‐conforming operation. If the quality premium effect is not explicit, this frontier
might be used subsequently in a contract negotiation in which a quality premium is
then added to the frontier, effectively giving a double premium.
An alternative and arguably better option would be to use a dataset of competitive bids
that included all conforming bids, ie they (just) met all quality requirements. Where
there are compliant bids, ie they meet all quality requirements, with lower prices than
the winning bids, their inclusion can yield a lower estimated frontier price. A quality
premium could then be added to a contract price estimated from such a frontier.
Group tender bids can be included but only in aggregate, ie as though they are
representing the costs of a very large route.4
4 If a group tender bid won for 4 units (routes) the data (bid price, kilometres, hours and bus numbers)
for all four should be added together as though it was one large route
11
We recommend that the largest possible dataset is used that reflects all combinations of
conforming bids. The only time this could be a problem is where:
there is a low cost outlier that is not a genuine bid. However this is unlikely as
any low cost bid would have been likely to have been accepted;5 or
the route for which the benchmarking exercise is being conducted is very
different from those for which benchmark data have been collected.
Provided the second point is not a concern, and this can be addressed outside of the
benchmarking exercise – it is an issue of data classification ‒ the chances of a bid being
included in the frontier analysis that was not an achievable combination of costs, is very
low.
4.2.2 Additional Information Providing a lower bound estimate on its own may not be as useful as providing it
alongside additional information on a range of prices, particularly amongst winning
bids.
The frontier analysis can be used to examine the cost‐efficiency of each winning
bid relative to the least‐cost frontier. This provides information on the range of
potential efficiencies amongst winning bids and could be used to estimate what
prices would be for the current negotiations consistent with this range.
Component cost analysis can be used with the wider (all compliant bid) dataset
to demonstrate the range of costs for each component: $/hour etc.
This additional information provides greater context for negotiations.
4.3 Recommended Approach Our recommendations are that the frontier approach should be the primary analytical
tool but that additional information is provided as context.
We recommend that:
1. A dataset is compiled of bid data from competitive tenders for routes (and units)
that are deemed relevant to the route being negotiated. This should include all
compliant bids. It should include:
a. the route information—kilometres, in‐service hours and peak vehicle
requirement;
b. total bid price;
c. components costs for the bid—$/km, $/hour and $/bus.
5 Where this is because the low bid has low quality attributes, this could be eliminated by limiting
analysis to conforming bids
12
2. The dataset of bid data should be screened initially to identify any outliers. This
would be done through a “visual” scanning of the data followed by an analysis
of specific data points identified as potential outliers, to better understand
whether they should be included or not.
3. Frontier analysis is used to create a lower bound of costs for a contract
negotiation. It should be based on a dataset of:
a. winning bids; and
b. complying bids that are lower cost than the winning bids;
Group tender bids should be included only in aggregate, ie as though they are
representing a single bid for a very large route.6
4. Additional information is provided to funders that allows them to better
understand the range of bid data. This might include information on the
estimated costs at different percentiles within the results from frontier analysis
of winning bids. For example this would show the price at which, say, 25% of
winning bids would be more efficient. Examples of this are given in Figure 4
above.
5. Information on component costs for all compliant bids should be supplied also,
including averages for component costs and the average mix of costs for all
compliant bids, eg hour‐related costs as a percentage of the total, kilometre‐
related costs as a percentage of the total etc. An example of how the range of
component costs might be presented is given in Figure 3 above (page 7).
6 If a group tender bid won for 4 units (routes) the data (bid price, kilometres, hours and bus numbers)
for all four should be added together as though it was one large route
13
Annex 1 Identifying Outliers and the Implications for Benchmarking
Tender Price Data Some tender prices for urban bus contracts are available, however the data exclude the
required route characteristics such as hours, distance and peak vehicle requirement
(PVR). We use these data to consider the variability of tender prices. In Figure 5 we
compare bids for each route. The points represent the bid prices as a percentage of the
winning bid. For some routes tender prices vary greatly with the highest tender price at
376% of the winning bid in route 2.
Figure 5 Bus Contract Bid Prices as a Percentage of Winning Bids
0%
50%
100%
150%
200%
250%
300%
350%
400%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15* 16*
% of Winning Bid
Route Source: NZTA; Covec analysis
* Routes 15 and 16 were won by a group bid not shown here.
In routes 1‐14 the lowest bid is the winning bid. In all of these bids, the “quality” levels
are similar and the winning bid is chosen entirely on price (quality issues are discussed
below in Section 2.2). In routes 15 and 16 there are a number of bids that offer services at
quality levels both better and worse than specified. These two routes were won by a
group bid (a consolidated tender price for more than one route); and the percentages
shown represent the bid price relative to the lowest‐priced conforming bid, ie bids that
meet required service levels. Tender prices for non‐conforming bids can be excluded as
they are not directly comparable to winning bids.
Figure 6 shows the number of bids in 25% bands for the above dataset; it excludes the
winning bids but shows the number of bids that are within certain percentage bands of
the winning bids. The winning bids would be classified as 100%.
14
Figure 6 Number of bids in Percentage Bands Relative to Winning Bids
‐
2
4
6
8
10
12
14
16
0‐24
25‐49
50‐74
75‐99
100‐124
125‐149
150‐174
175‐199
200‐224
225‐249
250‐274
275‐299
300‐324
325‐349
350‐374
375‐399
Number of Bids
Tender Price (% of winning Bid)
Source: NZTA; Covec analysis
The low data points (below 100%, ie less than the winning bids) can easily be recognised
as outliers because they are non‐conforming. The three bids below 100% describe non‐
winning bids with lower service levels. Most bids (25 of 41) are higher than, and within
50% of, the winning bid (100‐149% of winning bid). Bid prices up to 225% of the
winning bid are not uncommon. There is only one bid which is a significantly greater
(376% of the winning bid) and might be considered an upper outlier.
The analysis of this small dataset suggests that:
There is a very wide range of bid prices and that there is a need for a clear focus
on prices that would be likely to result from a competitive market, for example
by having tighter criteria for what are comparable routes; and
low outliers can be identified by looking at accompanying information relating
to quality. This could easily be achieved by a specific analysis of the low data
points.
High outliers can be a problem if an upper bound is defined on a percentile basis, and if
there are a significant number of outliers such that they affect the percentile itself, eg a
75th percentile would be affected if more than 25% of bids were high outliers; this is
unlikely.
School Bus Data The tender price data above exclude route parameters such as hours, distance or peak
vehicle requirement. A large dataset was made available to us by the Ministry of
Education from school bus bids for a 2008 tender round; these included prices and
kilometres. The bids did not specify the number of buses and this information was not
collected; service hours data were not available either.
15
The school bus data include rural contracts used to collect pupils for school and urban
(technical) contracts used for transporting pupils within a town or city to alternative
facilities during the school day. We have limited our analysis to the technical routes. We
exclude fleet bids, where a bidder has averaged the per km price across multiple routes.
Figure 7 presents the data, showing how the bid price per km varies with route length.
The blue × is the current price per km (the cost of the incumbent) while the red dots are
the bid prices per km. The current prices were adjusted for inflation and are directly
comparable to the bid prices. Contracts include two rights of renewal for 3 years each
using existing prices adjusted for inflation. Therefore prices of incumbents can be
treated the same as new bids for benchmarking purposes as they are actual achievable
prices. There is a clear inverse relationship between route length and price per
kilometre. That is, as route length increases, the price declines per km; this implies a
significant fixed element to costs.
Figure 7 Bus Contract Bid Prices ($/Km) compared to Route Length
0
20
40
60
80
100
0 20 40 60 80 100 120 140 160
Price ($/Km)
Route Length (Km)
Current $/km
Bid $/km
Source: Ministry of Education; Covec analysis
Frontier Analysis with Outliers Included Frontier analysis (in the form of Data Envelopment Analysis, DEA) identifies the most
efficient price for a number of outputs7. The data here are only in one dimension (rather
than three that we expect to be available for bus contract benchmarking) but we can
consider which prices are the most efficient per km for routes of varying distances to
illustrate some aspects of DEA and the significance of outliers.
We focus on the low bids only which indicate the most efficient prices. Figure 8 shows
the same price information as above but for bids less than $6 per km only. The solid line
in Figure 8 describes the “market price” frontier by connecting the lowest price bids in
7 DEA analysis considers the most efficient price for more than one output (Kilometres, hours and peak
vehicle requirement).
16
the same way as under the DEA analysis: a vertical line is drawn from the highest unit
price bid on the frontier; it is recorded for a short‐distance route. We then connect the
low bids with the minimum number of straight lines that can enclose all bids; a
horizontal line is then drawn from the absolute lowest priced bid. This figure is used to
represent the outcome of using the frontier approach without adjustment for outliers.
Figure 8 Bus Contract Bid Prices ($/Km) compared to Route Length — Prices less than $6/km only
0
1
2
3
4
5
6
0 20 40 60 80 100 120 140 160
Price ($/Km)
Route Length (Km)
Current $/km
Bid $/km
Source: Ministry of Education data; Covec analysis
A high proportion of the less than $6/km values are the incumbents’ prices. This is to be
expected as it would be typical that the lowest price was accepted previously. And all
but one of the lowest prices (forming the frontier), are current prices. Therefore, in this
example, these should not be considered outliers as they are genuine prices for existing
routes. The one lowest price which is not a current price, was the winning bid chosen
for a new contract, and is again a genuine bid.
The outlier issue is illustrated in Figure 9, which is a section of Figure 8. It shows part of
the frontier; above this frontier, and between the points that define it, are bid prices and
the value of an incumbent contract. If DEA was used, the contracts represented by these
dots and Xs would be expected to achieve a price at the lower level of the frontier. If the
frontier is constructed using outlier data, achieving the frontier price may not be
possible.
17
Figure 9 Identifying Outliers
Excluding Outliers Next we consider how the frontier might be defined if low outliers are excluded. Figure
10 constructs the frontier using new bids only (the solid line) rather than current prices
(the dotted line is the original frontier). Being the lowest prices, these are most likely to
be winning bids. However if the option to renew existing contracts had been available, it
is likely that some of the new bids would not win over existing prices.
Figure 10 Most efficient price ($/Km) defined from tender bids only
0
1
2
3
4
5
6
0 20 40 60 80 100 120 140 160
Price ($/Km)
Route Length (Km)
Current $/km
Bid $/km
Source: Ministry of Education data; Covec analysis
We then exclude bids that appear to be outliers. This is done initially by visually
identifying data points that appear to be significantly less than the others. Figure 11
shows the revised frontier if we exclude four low bids that look like outliers from
visually scanning the data.
18
Figure 11 Excluding the lowest bids from tender bids ($/Km)
0
1
2
3
4
5
6
0 20 40 60 80 100 120 140 160
Price ($/Km)
Route Length (Km)
Current $/km
Bid $/km
The next approach is to construct a frontier from a dataset that removes values below
the 25th percentile (ie the 25% of bids that would be the least cost/most efficient). We
illustrate this in Figure 12 for the whole dataset and in Figure 13 for the under $6/km
bids only.
Figure 12 Frontier based on 25th percentile (whole dataset)
0
20
40
60
80
100
0 20 40 60 80 100 120 140 160 180
Price ($/Km)
Route Length (Km)
Current $/km
Bid $/km
25th percentile
19
Figure 13 Frontier based on 25th percentile (under $6/km bids only)
0
1
2
3
4
5
6
0 20 40 60 80 100 120 140 160
Price ($/Km)
Route Length (Km)
Current $/km
Bid $/km
25th Percentile
The 25th percentile has been estimated by grouping the data into 10km route length
bands and estimating the 25th percentile in each band. Those data were then used to
construct the frontier (ignoring the 25th percentile point in some bands if this would be
above the frontier). The resulting frontier is pushed further from the origin.
The Results of Excluding Outliers
The overall results of the different approaches are shown in Table 2; it shows the
percentage increase in the estimated price above the lowest price frontier as a result of
using the different approaches to removing outliers. The approaches used, which are the
separate lines in Figure 13, are:
Lowest price (including incumbents) as shown in Figure 8;
Lowest bid prices (excluding incumbents), as shown in Figure 10;
Frontier from bid prices with visually‐identified outliers removed (Figure 11);
Frontier based on 25th percentile (Figure 13).
Table 2 Difference between frontier and lowest contract price using different approaches to remove
outliers
Kilometres Lowest Lowest bid Manual outliers 25th percentile
10 0% 26% 65% 192%
20 0% 42% 66% 149%
40 0% 10% 35% 96%
60 0% 1% 28% 57%
80 0% 2% 22% 43%
100 0% 2% 19% 43%
For this dataset using the 25th percentile can result in price estimates that are very
considerably higher than that estimated from the lowest prices (including incumbents).
20
If these data are representative of other bid data, it suggests that using a 25th percentile
could lead to a very wide range of acceptable bid prices. This is unlikely to be useful as
an input to decision makers seeking information on a competitive market price.
Analysis Using Winning Bids Only Benchmarking using winning bids only is consistent with the achievement of the value
for money requirement. It is also a means to exclude outliers. If they are winning bids
we know that they were both acceptable to the funder and were achievable by the
operator. In the analysis here we exclude group bids as above, and existing contract
prices. Since we exclude group bids, using winning bids also excludes individual bids
on routes which were won by a group bid, even if the individual bid is lower than the
group bid. Figure 14 shows frontiers based on the lowest bids and with the exclusion of
visually‐obvious outliers.
Figure 14 Efficient Price from Winning Bids & Excluding Visually‐Obvious Outliers
0
1
2
3
4
5
6
0 20 40 60 80 100 120 140 160
Price ($/Km)
Route Length (Km)
Winning Bids
Lowest $/km
Lowest $/km excluding outliers
Figure 15 shows, in addition, the 25th and 50th percentiles using winning bids.
Percentiles have been included here as a way to examine the potential for using upper
and lower bounds in the analysis, as initially considered by the PTOM Working Group.
21
Figure 15 25th and 50th Percentile for Winning Individual Bids (under $6/km bids only)
0
1
2
3
4
5
6
0 20 40 60 80 100 120 140 160
Price ($/Km)
Route Length (Km)
Winning Bids
50th Percentile
25th Percentile
Lowest $/km
Lowest $/km excluding outliers
The Results of Excluding Outliers
Table 3 shows the percentage increase in the estimated price above the lowest price
frontier for the winning bids dataset as a result of using the different approaches to
removing outliers. The approaches used, which are the separate lines in Figure 15, are:
Lowest price (including incumbents) as shown by the dashed line in Figure 14;
Frontier from bid prices with visually‐identified outliers removed (solid line in
Figure 14);
Frontier based on 25th percentile.
Frontier based on 50th percentile.
Table 3 Difference between frontier and lowest contract price using different approaches to remove
outliers using a dataset of winning bids
Kilometres Lowest bids Manual outliers 25th percentile 50th percentile
10 0% 28% 97% 159%
20 0% 9% 62% 96%
40 0% 32% 60% 98%
60 0% 48% 72% 97%
80 0% 40% 62% 81%
100 0% 33% 56% 74%
We would expect that, using winning bids only would lower the position of the 25th
percentile frontier relative to the lowest bid frontier, in comparison with the result using
all bids (Table 2). This is only so for the three lowest distances shown; for the other three
distances the results using winning bids are higher than if using all bids.8 The reason for
this outcome using this dataset is that some winning bids were group tender bids. They
8 72%, 62% and 56% compared to 57%, 43% and 43%
22
competed with sets of individual bids that had low bids for some routes. In these cases
the winning bids dataset also excludes these individual (non‐winning) bids, even
though they are lower‐priced than the winning group tender bid. For example, Table 4
describes a route grouping where the group bid won the grouping but some individual
bids (in bold) are lower‐priced than the group bid.
Table 4 Bids on a Route grouping with a winning Group Bid
Route Km
Current $/Km
Individual Bid 1
Group Bid 2
Winning Group Bid 3
Individual Bid 4
Group Bid 5
Individual Bid 6
10 $6.15 $12.98 $10.30 $5.32 $8.20 $7.47 $11.11
40 $6.15 $8.58 $10.30 $5.32 $4.11 $7.47 $7.36
5 $12.47 $24.42 $10.30 $5.32 $14.11 $7.47 $20.64
12 $7.45 $11.88 $10.30 $5.32 $7.38 $7.47 $11.43
20 $6.15 $10.78 $10.30 $5.32 $5.49 $7.47 $7.36
35 $6.15 $8.58 $10.30 $5.32 $4.28 $7.47 $6.39
50 $6.06 $8.58 $10.30 $5.32 $3.82 $7.47 $7.04
12 $6.15 $11.88 $10.30 $5.32 $7.22 $7.47 $12.71
5 $15.28 $26.62 $10.30 $5.32 $16.68 $7.47 $24.10
20 $4.15 $10.78 $10.30 $5.32 $5.27 $7.47 $7.36
In the winning bids dataset, all of the bids here are excluded because a group tender bid
won the route grouping whereas in the previous dataset, individual bids 1, 4 and 6 were
included.
As with the analysis using all data, the results suggest that using the 25th percentile as a
lower limit may result in benchmark prices that are significantly higher than a
competitive market price. Manual removal of outliers can produce significant increases
in prices, depending on the location within the dataset, but it is likely to achieve results
that are closer to the objectives than is achieved using the 25th, or similarly high,
percentile.
Individual Bid Analysis As an alternative way to analyse the outlier issues, we examine the bids for individual
school bus routes in more detail, as shown in Figure 16. The bid prices for 19 different
routes can be seen in the columns.
Looking at individual routes it is possible to identify bids that appear to be outliers, eg
the very high bid for route 18. However, this bid price is not much higher than bids for
other routes (14, 15 and 16), and all four of these routes included fleet bids (see below).
To explore this further, Figure 17 presents the same data but excluding the fleet bids in
which companies have average‐priced and bid the same unit rate for several routes. The
high bid is still there for route 18 but there are now two bids only, such that it is not
possible to tell whether the high or low bid is the outlier. High outliers are of much less
concern than low outliers.
23
Figure 16 Bids for School Bus Routes
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
$/km
Route Source: Ministry of Education; Covec analysis
Figure 17 Bids for School Bus Routes – excluding group tender bids
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
$/km
Route Source: Ministry of Education; Covec analysis
Hypothetical Data In addition to these actual data, we have used hypothetical data (Table 5) to give a
clearer demonstration of the effects of including outliers. The data are for bids on 5
routes in order to assist in providing a benchmark cost for route 6. Route 3 includes a
clear high outlier and a clear low outlier (shaded in the table). The outliers are bids for
route 3 and have component costs that are either well below or well above the bids for
all other routes. We use these hypothetical outliers to demonstrate the effects of
including them in the dataset.
24
Table 5 Raw Data (Hypothetical)
Route Km hr PVR $/km $/hr $/bus Total Cost
1 1,764,693 102,648 50 1.87 24.41 52,886 8,454,791 1 1,764,693 102,648 50 1.89 25.79 53,665 8,666,161 1 1,764,693 102,648 50 1.92 26.87 54,732 8,882,815
2 1,011,362 57,699 31 1.87 23.46 54,568 4,931,423 2 1,011,362 57,699 31 2.03 25.55 51,929 5,137,073 2 1,011,362 57,699 31 1.86 26.93 52,581 5,064,978
3 296,121 16,090 8 1.89 25.56 53,080 1,395,569 3 296,121 16,090 8 2.36 31.95 66,351 1,744,047 3 296,121 16,090 8 1.42 19.17 39,811 1,046,428 3 296,121 16,090 8 1.88 24.41 45,850 1,316,264
4 232,449 19,129 4 2.1 25.38 45,169 1,154,313 4 232,449 19,129 4 2.13 5.85 50,950 810,821 4 232,449 19,129 4 1.94 25.3 53,936 1,150,659
5 1,709,004 79,738 47 1.84 25.43 46,306 7,348,687 5 1,709,004 79,738 47 1.82 27.18 52,405 7,740,701 5 1,709,004 79,738 47 1.81 23.13 46,100 7,104,337
6 750,000 50,000 15
Figure 18 shows the outliers relating to one component cost: $/km. All but two bids have
costs within the dotted lines; the outliers are outside these.
Figure 18 Outliers in $/km data
1
1.2
1.4
1.6
1.8
2
2.2
2.4
2.6
0 1 2 3 4 5 6
$/km
Route
Actual outliers may be less noticeable than those pictured here. We have provided
outlier examples in which all component costs are either high or low; in practice a route
may have a low cost for one component and a high cost for another. This is to be
expected as each component is, to some extent, a trade‐off. That is, some routes may
face less congestion and cover a greater distance but this would be reflected in a lower
cost per km and a higher cost per hour. These are not outliers as it is an efficient trade‐
off between factors of production. Frontier analysis identifies an efficient combination of
these trade‐offs. However, these individual component costs become a risk if
component costs are combined by choosing the lowest from any route.
25
Given this, the approach used to benchmarking will have a significant impact on the
importance of outliers to the analysis. We explore this in the next section through
examining the implications of outliers under the different methodologies.
Outliers in Component Cost Benchmarking To demonstrate the effects of an outlier we compare the component cost benchmark
including and excluding both clear low and high outliers. We use the data in Table 5
above. We also discuss the ways in which benchmarking data might be presented.
Calculating the 25th and 75th Percentile Component Cost Benchmarking uses the $/unit data only and finds the 25th and 75th
percentile as potential lower and upper bounds. In Annex 2 we describe how
percentiles are calculated. Depending on the total number of observations in the
dataset, the 25th or 75th percentile may not fall on a specific observation in the set, but
between two values. In this case we calculate the percentile as a weighted average of
the two values, where weightings depend on where the percentile falls between the two
figures. The location of a percentile changes when a dataset is made smaller. Therefore,
excluding outliers affects what numbers are included within a percentile, although with
larger datasets, the effect is negligible. For example, with 9 data points, as expected for
the first Auckland round, the 25th percentile would be half way between the prices of the
2nd and 3rd lowest bids.
Effects of Including or Excluding Outliers
All Values Included
When all values are included in calculating the benchmark, the total suggested cost of
Route 6 (Table 5) based on the 25th percentile and 75th percentile component costs is:
Table 6 Percentiles including all Values
$/km $/hr $/bus Total
25th Percentile Component Costs 1.85 24.24 46,222 3,288,878
75th Percentile Component Costs 2.01 26.60 53,868 3,643,649
High Outlier Excluded
When a high outlier is excluded from the series, the series is now smaller at one end.
The effect is a slight decrease in the rank at which each percentile falls, but the overall
effect is limited because the percentile is typically still a weighted average of the same
two values:
Table 7 Percentiles Excluding the High Outlier
$/km $/hr $/bus Total
25th Percentile Component Costs 1.84 24.21 46,208 3,283,446
75th Percentile Component Costs 1.94 26.02 53,723 3,561,918
26
Low Outlier Excluded
When a low outlier is excluded there is a decrease in the rank at which each percentile
falls, but each value also drops one rank so the net effect is a slight increase in value.
Table 8 Percentiles Excluding the Low Outlier
$/km $/hr $/bus Total
25th Percentile Component Costs 1.86 24.00 46,111 3,286,801
75th Percentile Component Costs 2.03 26.87 53,936 3,675,040
High and Low Outliers Excluded
When both high and low outliers are excluded, the rank at which each percentile falls
moves closer to the centre. This has the net effect of reducing the difference between the
25th and 75th percentiles.
Table 9 Percentiles Excluding the High and Low Outliers
$/km $/hr $/bus Total
25th Percentile Component Costs 1.86 24.17 46,255 3,293,693
75th Percentile Component Costs 1.96 26.06 53,733 3,580,866
To compare the effect of outliers, the cost data for each component are shown on a
number line below.
Figure 19 25th and 75th Percentiles for $/km
1.40 1.50 1.60 1.70 1.80 1.90 2.00 2.10 2.20 2.30 2.40
$/km No Outliers High Outlier Excluded Low Outlier Excluded All Values
Figure 20 25th and 75th Percentiles for $/hr
19 20 21 22 23 24 25 26 27 28 29 30 31 32
$/hr No Outliers High Outlier Excluded Low Outlier Excluded All Values
Figure 21 25th and 75th Percentiles for $/bus
39000 41000 43000 45000 47000 49000 51000 53000 55000 57000 59000 61000 63000 65000 67000
$/bus No Outliers High Outlier Excluded Low Outlier Excluded All Values
The raw data are shown by the small red points; the outliers are clearly outside the main
distribution.
The 25th and 75th percentiles for the data, including all values, are shown by the
larger red dots.
The yellow triangles show the slight increase in the 75th and 25th percentiles
when we exclude a low outlier.
The blue triangles show a slight decrease when we exclude a high outlier.
Lastly the gold square shows a narrowing of the range for the 25th and 75th
percentiles when we exclude both high and low outliers.
27
The larger change to the 75th percentile in Figure 19 and Figure 20 are a characteristic of
the data set; there is more variability in $/km values and $/hr values in the upper half of
the data set.
We can make some general conclusions on the implications for component cost
benchmarking.
Using a large dataset as a result of significant competitive bid experience
removes most of the potential risk of outliers. If outliers exist, they will have
little impact on the estimated percentiles. Outliers can have a significant impact
on the calculation of percentiles for component costs when a large portion of the
data are outliers, such that one of the datapoints used to calculate the 25th or 75th
percentile is an outlier itself. In this situation, determining which figures are
actually outliers is difficult.
Using a small dataset increases the potential impact of an outlier, makes it more
likely that an outlier would be used to calculate the percentile and makes it more
difficult to identify an outlier.
If any data are to be excluded, there must be a thorough understanding of the dataset,
causes of potential outliers and the trade‐offs between different costs. Data should only
be excluded if they are considered not to be a genuine reflection of cost components for
the bus route it represents, and therefore not a genuine bid, or if they are from a route
with characteristics that are so different from that being analysed as not to provide
useful information.
Outliers in Frontier Benchmarking Frontier (lowest potentially achievable cost) analysis uses the total cost data and the km,
hours and peak vehicle requirement to find an efficient cost to provide this combination
of outputs. To demonstrate the effects of using outliers in a Frontier Analysis, we
compare the benchmark from the hypothetical data (Table 5), including and excluding
both clear low and high outliers.
Table 10 Results of Frontier Analysis Including and Excluding Outliers
Exclusions Frontier Efficient Cost
All Values Included $ 2,844,291
High Outlier Excluded $ 2,844,291
Low Outlier Excluded $ 3,357,289
High and Low Outliers Excluded $ 3,357,289
Including low outliers is a problem for frontier analysis because the low outlier is
always on the efficient frontier. The exclusion (or inclusion) of a high outlier has no
effect because the point will never be on the frontier. Frontier analysis thus prevents
gaming ‒ high outliers have no effect on estimating the efficient frontier.
Determining whether a point on the frontier is an outlier might be undertaken by
assessing the efficiency of an individual point compared to a frontier that has been
28
derived using all data apart from the potential outlier bid, although this might need to
be in the context of an analysis of the variability amongst the other data. As with the
component cost approach, a larger dataset will make it easier to identify outliers.
High and Low Bounds As noted above, the PTOM Working Group recommends constructing a lower and
upper price bound (See Section 1). In this section we assess the implications of outliers
for constructing these bounds.
The upper bound is defined either using the 75th percentile of component costs or the
highest component costs from winning bids. The inclusion of outliers has little impact
on which approach is taken. In contrast, the inclusion of outliers can alter which
methodology defines the lower bound. To illustrate this, the impacts on lower bounds,
using the hypothetical data, are shown in Table 11.
Table 11 Impacts of Exclusions on Lower Bound
Exclusions Frontier 25th percentile Approach defining
Lower Bound
All Values Included $ 2,844,291 $ 3,288,878 25th Percentile High Outlier Excluded $ 2,844,291 $ 3,259,500 25th Percentile Low Outlier Excluded $ 3,357,289 $ 3,286,801 Frontier High and Low Outliers Excluded $ 3,357,289 $ 3,302,130 Frontier
High outliers have almost no effect on the lower bound because it has little effect on the
25th percentile and no effect on the frontier. Low outliers have a small effect on the
lower bound because, when low outliers are included, the bound switches from being
set by the frontier to being set by the 25th percentile.
The use of the component cost 25th percentile (including all outliers) as an alternative
lower bound goes some way to solving the problem with inclusion of low outliers in
Frontier Analysis. Any percentile that excludes the minimum value would achieve the
result of excluding the outlier. For example, if there were 9 values in the series, the 20th
percentile would select the 2nd lowest value, and therefore exclude the low outlier.
However, there are risks in both directions:
the 25th percentile might result in a combination of components that are not an
achievable combination and/or
it will result in a combination that is more expensive than the “efficient” cost, ie
the estimated competitive market price. This is a significant issue for funders
and could amount to several millions of dollars.
It is only in rare circumstances that a percentile of each component cost would achieve
the efficient combination. And this highlights the other advantage of frontier analysis: it
always chooses an efficient and achievable cost, provided all bids used to construct the
frontier are genuine bids. Building up a sufficiently large and relevant (ie data are for
similar routes to the rote being examined) dataset, analysed using the Frontier
approach, would be the preferred approach.
29
Annex 2 Calculating Percentiles A percentile is the value below which a certain percentage of observations fall. That is
the 25th percentile is the number where 25 percent of all observations are below this
number. To calculate a percentile, the list of observations is first sorted into ascending
order. We then calculate the rank of the percentage we are trying to find. Depending
on the total number in the dataset, this may not fall on a single observation in the set,
but between two observations. In this case we calculate the percentile as a weighted
average of the two values, where weightings depend on where the percentile falls
between the two figures.
Statistical Percentile
If we have N numbers in a series, after arranging the values in ascending order, the rank
of the pth percentile is calculated by:
The value is then calculated using this rank. If n ≥ N the value of the pth percentile is the
highest number while if n ≤ 1 the value of the pth percentile is the lowest number in the
series. Where the rank is between 1 and N, n is split into an integer k and decimal d (i.e.
1.25 is split into k = 1 and d = 0.25). The value of the pth percentile is the kth ranked
value, plus d multiplied by the difference between the kth variable and the kth + 1
variable. For example if there are 6 values in a series. The 50th percentile (median) seeks
the value ranked 3.5. It is the third lowest value plus 0.5 times the difference between
the third and fourth number in the series. The 25th percentile seeks the value ranked
1.75. It is the lowest number plus 0.75 times the difference between the lowest number
and the second lowest number.
Microsoft Excel
Excel calculates the ranking of the pth percentile in a slightly different way.9
This rank is larger than the statistically correct ranking. The effect is most pronounced
with either small datasets or with extreme percentiles.
The percentile entered in excel is a decimal between 0 and 1. i.e. p/100. The statistically
correct value of the percentile can be calculated in Excel by modifying the percentile
entered into the formula, p*/100, such that:
For example, to find the statistically correct 25th percentile in a sample with 17
observations, rather than using p = 0.25 in the formula in excel we should use p* =
0.21875. However, this formula may result in values of p* that are less than zero or
greater than 100. In these circumstances an adjustment needs to be made:
9 See the Engineering Statistics Handbook, available at
http://www.itl.nist.gov/div898/handbook/prc/section2/prc252.htm for further explanation regarding
how percentiles are calculated.
30
For low p, if the value of the pth percentile is the smallest number in the data
set.
For high p, if the value of the pth percentile is the largest number in the data
set.
