Quality audit of draft Nera water bills model
Report prepared for Defra
Final report
December 2014
2 Quality audit of draft Nera water bills model
Contents
1 Review process ...................................................................................... 4
2 Comments by area ................................................................................ 7
3 Quality audit of draft Nera water bills model
List of tables
Table 1. Record of model audit review activities and inputs .............................. 4
4 Quality audit of draft Nera water bills model
1 Review process The model has passed the quality audit and only a few minor
checks and revisions are recommended
1.1 Audit procedure
To answer the questions set out by the model review process, the audit was carried out at two levels.
The first level is to look at the aims, objectives, design and structure of the model. The second level is to dig
deep into the data, assumptions and formulae used in the model. The overall structure was evaluated on both
the coding of formulae and the logic behind the main calculations. The basic inputs and outputs of the model
were checked for clarity, consistency and replicability (usability of the model).
A detailed evaluation of the spreadsheet examined all the assumptions, scenarios, data and data
processing sheets in considerable detail, evaluating each formula or data field for consistency. Other
aspects of the model such as colour coding, error code handling and assumptions and data management were
also checked. Each formula was transcribed in a word form alongside the variables and arguments it uses and
each input\argument was given a name. Formulae were checked based on first principles and on whether
they made economic and \or accounting sense. Also where a formula was used in more than one place (for
different companies and\or elements of the value chain), its inputs were verified for consistency.
Nera responded to the comments made in this report and as a result some figures changed between Nera’s
draft final and final work. Figures taken from Nera’s draft final work are discussed here and have not been
updated with its final figures in this document, otherwise the audit comments attached to them would no
longer make sense. Hence there will be differences between the figures quoted in this report (particularly in
section 2.6) and the final Nera report.
Table 1. Record of model audit review activities and inputs
Task Percentage of
model covered
by review
Time
allocated to
task
Finding
Overall
4.5 h
Are the overall aims of the model clearly
specified
100 per cent 2 h Yes
Does the model solve the problem it was
intended to? 100 per cent 2 h Yes
Is there a colour key outlined? 100 per cent 10 min Yes, but it could be extended
5 Quality audit of draft Nera water bills model
Is there a table setting out the purpose of
each of the sheets in the model? 100 per cent 10 min Yes
Assumptions 2.5 h
Are all the assumptions specified in
assumptions sheets? If not, is it clear
where else they are?
100 per cent 20 min Most, but not all are specified in
the assumptions sheets
Are the units for all assumptions
identified?
100 per cent 30 min Some units are missing
Do all assumptions have dependents? 100 per cent 30 min Yes
Is it easy for the user to restore default
assumption values?
100 per cent 30 min There are a few hard-coded
numbers
Does the model store scenario
assumptions alongside stored outputs?
100 per cent 30 min Yes
Structure 20 h
Is the structure consistent and coherent
throughout the model?
80 per cent 2 h Yes
Are formulae used consistently? 65 per cent 2 h Yes
Is it flexible enough for input variables to
be changed easily without causing errors?
100 per cent 2 h Yes
Does the flow of calculations and the
assumptions used make logical sense?
65 per cent 6 h Yes
Is the model working correctly – i.e. does
each set of calculations (i) calculate the
right thing and (ii) calculate it correctly?
a. Data forecasting b. Data compiling c. Main calculations
65 per cent 8 h
1. There are a some issues with data forecasting
2. Data compiling works correctly
3. Main calculations work correctly, with small number of issues identified
Calculations 18 h
Is the purpose of each calculation clear
and accurately labelled?
65 per cent Formulae and tables are not
labelled. There is no detailed
calculation architecture or list of
word equations
Has the colour key been followed?
65 per cent Yes
Do the ranges in any formulae cover the
appropriate cells? Pay particular attention
to the data in the last row and last column
of any table which has repeating
formulae.
65 per cent Yes
Are there any ‘#ERR’, ‘#REF’ etc. cells?
65 per cent A few instances of #Value and
#N/A
6 Quality audit of draft Nera water bills model
Are there labelled ‘checks’ built in for
checking important calculations i.e. do
shares sum to 100 per cent?
65 per cent Yes
Are the arguments of functions such as
VLOOKUP, IF etc. correctly articulated?
65 per cent Yes
Do any calculations use numbers rather
than referring to other cells?
65 per cent A few instances of hardcoded
numbers
Are unique formulas
arithmetically/economically correct?
65 per cent Yes
Output
Are there separate sheets summarising
the outputs from the model?
100 per cent Yes
Are all units correctly identified? 100 per cent Yes
Does the model contain a convenient way
of labelling and storing output data?
100 per cent Yes
Source: Vivid Economics
7 Quality audit of draft Nera water bills model
2 Comments by area The model performs well but would benefit from narrowing of
assumption ranges
2.1 Documentation
The draft report is clear and well structured. The draft main report is clear, well written and well laid out.
There is evidence that issues have been recorded and addressed. Nera kept a log of comments received
during the model development process and these were shared with and read by Vivid.
The report introduces the reader to some of the main calculations and all the user interface sheets. The
report explains how to operate the model, sets out its functionality and displays example outputs. It is written
clearly and is easy to understand. The model explains data sources and states which principal calculations
take place.
There are a number of areas not covered by the report, which are candidates for appendices. The
report could go further in appraising the evidence for input assumptions. It could explain off-model
calculations and in some places, could more fully elaborate explanations of key steps in the calculations. The
report would greatly benefit from a table of abbreviations and this table should be included in the spreadsheet
as well.
The model provides particular value through its ability to highlight the relative importance of drivers
of bills. The model allows the user to explore the sensitivity of bills to a wide range of drivers and
assumptions. This is very useful, however, the user will find it very taxing to explain why bills have
particular sensitivity to drivers. Two additional pieces of model documentation could help: an influence
diagram and word equations. Nera responded by adding this material. An influence diagram shows how
important input drivers and assumptions feed through intermediate variables and calculations to output
variables. It is complemented by word equations which show how variables are combined at key stages of
the calculation. It is common but not universal practice in planning model construction to build influence
diagrams and to write out equations in words before encoding them. If such information is available it would
be very worthwhile including them in the report as appendices, or if too large, as e-documents.
While the report is generally very clean, there are some minor tidying edits needed. There are some
minor editing revisions to attend to in the finalisation of the report. Some tables are presented without units
and some figures are quoted without time periods, and there is ambiguity in some cases whether long term
growth rates are arithmetic or geometric averages. There are a few typographical errors. Vivid has marked
these up on the draft report except for the typographical errors which could be picked up with a spelling
check.
8 Quality audit of draft Nera water bills model
2.2 Structure
The model has a clear and organised structure, the logic of the model is sensible. The spreadsheet has
information on the structure of the model, the purpose of each sheet is documented and consistent; however
it can be improved by including a glossary of the abbreviations used. The model was split into raw data, data
forecasting, data processing and main calculations. The details in each block are comprehensible and the data
is constantly being summarised and consolidated for ease of use. Policy and driver shocks are treated as
deviations from a baseline scenario and, where there are many scenarios, ample care was taken in defining
and coding each scenario separately. Drivers and policy scenarios are laid out explicitly with a clear purpose
and organisation. The formulae used in the model can be characterised into:
– forecasting formulae;
– accounting identities;
– accounting for cost of growth and quality enhancement;
– accounting for own and cross price elasticity of demand;
– cost appropriation along the value chain; and
– data retrieval and consolidation.
The structure of data handling is coherent throughout the model. The main calculations repeat across
the companies and the elements of the water and sewerage value chain. Vivid Economics checked that
repeated calculations adopt the same formulae across companies and value chain elements and found that the
calculations are carried out in a consistent and coherent fashion across all these elements. The two main
panels for controlling the scenarios and data input make the model flexible enough to introduce assumptions
without altering any of the core calculations. Vivid Economics checked almost two thirds of the model
formulae meticulously by transcribing the formulae into words and naming each variable. The word
representation of entire sheets of calculations ensures a clear understanding of the logic of the formulae. For
each of these formula, Vivid verified that:
– it was coded correctly;
– it used the right inputs;
– it makes economic\accounting sense; and
– it was subsequently used properly.
There are a small number of instances where formulae might be considered for revision, as described in the
issues log.
2.3 Calculations
The formulae used are correct and there are very few coding and logical errors. There are no major
instances where formulae were deemed inappropriate, however there were few instances where the inputs
were deemed inappropriate. Nera adopted all suggestions from the log including minor suggestions such as
formatting.
The data in each row is carefully described, however the spreadsheet would be more accessible if it
included descriptions of some of the key calculations. Nera added responded by adding descriptions.
9 Quality audit of draft Nera water bills model
Given the complicated coding involved in retrieving data by constant reference to scenario and variable
codes, the logic of the calculation can be hard to follow. There is a basic colour coding that is followed in the
spreadsheet for data inputs. A remedy could be to introduce colour coding for the calculation sheets,
mapping colours to companies and value chain elements, to improve the navigation and usability of the
model. We found only three other minor issues with the calculations:
– there is a minor concern with the handling of error codes because #N/A and #Value appear frequently.
This can be addressed by explicitly indicating which data is not necessary and can be omitted from
subsequent calculations. Nera added this to the list of potential future upgrades;
– there are a few checks built in to verify the consistency of a string of calculations but the model could
benefit from the inclusion of more checks. Nera added this to the list of potential future upgrades; and
– there is a small amount of hard coding of values in the model (see issues log). Some calculations refer to
empty cells which may cause a problem if the empty cells are altered by a user. This problem can be
resolved by locking the empty cells for editing. Nera responded to this issue.
2.4 Assumptions and inputs sheets technical review
All the assumptions are specified in the assumptions sheet, with similar sheets laid out for economic
drivers and policy shocks. All the variables defined in the assumption sheets have dependents. There are
instances where assumption variables are not recorded in the assumption sheets. On occasion, assumption
variables are introduced directly into formulae and there are other instances where assumed values were
explicitly inserted in separate rows in the calculation sheet, rather than in the assumptions sheets. The
introduction of variables directly into calculations is best avoided altogether. It is preferable to gather all
assumptions in one place, rather than to have them spread out among the calculations. Nera responded by
creating an assumptions worksheet. There were some occasions where assumptions figures were adjusted for
inflation by hard coding the inflation number into the calculation. This may cause a problem of inconsistency
with the user’s choice of economic drivers. Nera addressed this comment.
Some units of measurement were missing from data rows, making the sheets harder to read.
Generally the sources of assumed variables are stated. The assumption variables are stored appropriately
alongside the outputs.
2.5 Assumption ranges
We have examined the ranges of assumptions and in most cases find them to be reasonable. We have
looked at the ranges of assumptions and based on our experience, or on the quality of evidence offered in
support of the ranges, identified those which merit further discussion. Those meriting further discussion are:
– RPI; and
– efficiency improvement rate.
Nera revisited and updated the assumptions.
10 Quality audit of draft Nera water bills model
The RPI range spans very low and very high figures and may not be consistent with other
assumptions. In Table 3.8, the central and high RPI rates are well above the Bank of England target rate and
the long term low figure is historically extremely low. Table 3.9 even suggests large negative RPI figures
compounded for 28 years, which is an extreme scenario. There is a question whether it is possible to choose
assumptions so that the RPI, CPI or other price indices and GDP projections are inconsistent, which could
introduce unintended relative price effects. It appears that inconsistent assumptions can be introduced. These
two issues combined could lead to some odd projections, perhaps including very low bills. We strongly
recommend the review of additional evidence on RPI rates and the adoption of a narrower range of estimates
for the long term. This review could usefully encompass the GDP and CPI rates which also appear to span
negative figures which are improbable in the long run.
The compound efficiency assumption is important but unsupported. The model uses a compound
efficiency assumption of 1 per cent a year in the baseline and offers a range of 0.5 per cent to 2 per cent. The
outputs are sensitive to this assumption. No evidence is given for the range of assumptions. The higher figure
would generate a reduction of 50 per cent over a 34 year period from 2015 to 2049 if it is applied to the draft
determination figures and beyond. This is a large real-terms decrease and, although significant efficiency
improvements are likely to be found in the sector, a 50 per cent reduction may be too large as a realistic
planning assumption. Nera added a sensitivity section to its report.
There is a sudden fall in enhancement capex in the baseline scenario in the mid-2020s. This is
presumably in part because the demand for quality enhancements becomes much more uncertain beyond that
date. It is possible that a combination of aging assets, population growth, climate change, stricter
environmental legislation, and demand growth will lead to higher levels of capex and that the sudden drop in
capex will not occur. The user should be made aware that alternative, higher capex scenarios are possible,
and the change in the mid-2020s is worth highlighting. It may not be possible to introduce simple
functionality to allow users to raise the level of capex in the second half of the projection period, but it is
worth considering a more ambitious Water Framework Directive programme from 2025 as an option in the
policy drivers settings. There is a lot of detail in the model on water resources but much less detail on waste
water and water quality, which may become a spending priority in the mid-2020s, and this action would help
to address that balance.
The basis of the maintenance projections could be made clearer. Maintenance is a key building block of
bills. The use of totex and pay as you go calculations make it harder to understand how the capital
maintenance projections have been prepared. It would be helpful to the user to describe in more detail how
maintenance is calculated and projected forward.
GDP is a driver of demand. Population and GDP are both drivers of demand and they are treated
exogenously. However, population growth is a factor in GDP growth so there may be the potential for double
counting of population effects on demand or inconsistency between GDP scenarios and the hard-coded
population forecast.
One of the effects of upstream and retail competition is on cost of finance. It is not clear whether the
transition to a contestable market is assumed to change the cost of capital for the whole service regulatory
11 Quality audit of draft Nera water bills model
capital value (RCV) or whether the increase is focused on the value chain elements which are contested. It
would seem most appropriate to focus it on the contested activities. Nera confirmed that this is the approach
they took and that they had assumed zero effect on the WACC from retail competition.
2.6 Outputs
The graphical and tabular outputs cover a wide range of useful and intuitive reports. They are easy to
interpret but it requires some effort of investigation to find out what is driving trends. Some of the figures
quoted below, for example, the projected average bill, capex and opex, will differ from those published in the
final Nera water bills report.
There are a number of small changes in presentation which could be considered:
– it would be helpful to include a breakdown of the capital enhancement programme by value chain
element because capital expenditure is an important driver and it will help to explain what is driving
capital enhancement to report it across the value chain. Nera added this to the list of potential future
upgrades;
– the charts of bills expressed as a proportion of household income are useful and should be included in the
report;
– Figure 4.45 shows absolute totex by value chain element. Complementary charts could be included
showing absolute and percentages changes in totex by value chain element over time, allowing the user to
check whether there are large changes in some value chain elements and whether these are plausible; and
– it would be of policy interest to report bill impacts along the value chain, perhaps using waterfall charts,
but it is not clear whether the model can accommodate this; this is something for Nera to consider. Nera
added this to the list of potential future upgrades.
The baseline model runs suggest a long term downward movement in bills. There are two principal
drivers of the trend:
– enhancement CAPEX falls by half between 2015 and 2049, shaving 25 per cent off totex, leaving the
RCV fairly constant over the period; and
– operating expenditure falls by 18 per cent, due to assumed efficiency improvements.
These are offset by a 24 per cent increase in capital maintenance, despite efficiency improvements, reflecting
a larger physical capital base. We discuss these in turn.
The declining enhancement expenditure is plausible. The arguments supporting a declining enhancement
capital expenditure over time are that the proportion of assets remaining to be upgraded to address
intermittent sewage discharges, to improve effluent treatment, to increase drinking water quality or to
enhance resilience of resources all decline over time. It may be that the standards which the sector has to
attain are raised over time, but as has been seen in other utility sectors, it is possible that an historic peak of
capital enhancement in the sector has now passed.
The increasing level of capital maintenance is justified. Although additions to the capital base may be
slowing and the RCV, which is an accounting record, may be stable, the physical asset base of the water
12 Quality audit of draft Nera water bills model
industry is projected to grow. The argument for this is that the population is growing, quality of service is
improving and the technology employed is fairly constant. This growing physical asset base demands
increases in expenditure to maintain it. The projection of capital maintenance requirements has always been
a difficult area of water industry science and there is quite a lot of uncertainty around maintenance
projections. Figure 4.41 shows baseline flat capital maintenance until 2035 followed by a gradual increase.
Nera could provide a fuller explanation of its projection of capital maintenance expenditure including why
capital maintenance remains flat before increasing when capital enhancement is on a declining trend. If the
capital maintenance is pegged to movements in the RCV or totex, then consideration should be given as to
whether these relationships will be stable over time.
The model projects a very substantial shift in the balance between operating expenditure and capital
maintenance. The figures in Table 4.29 can be used to derive input shares. In 2015 the ratio of operating
expenditure to capital maintenance is 1.9:1.0, which falls to 1.27:1.0 by 2049. This would be consistent with
an industry which has become more capital intensive. It seems reasonable to expect that the sector will
become more capital intensive and that capital maintenance costs will grow relative to operating expenditure,
as technology changes and as quality of service rises. However, this is quite a large change in input shares
and it merits some discussion and explanation of why it is reasonable.
One of the factors driving the 18 per cent fall in operating expenditure is the efficiency assumption.
The efficiency assumption, which is 1 per cent per annum in the baseline, applies to aggregate operating
expenditure, not to unit costs of activities. If underlying operating activities are increasing, it implies a
greater than 1 per cent per annum efficiency increase, and if they are shrinking, a figure lower than 1 per
cent. Given forecast population increases and further capital enhancement, it seems likely that the former
case persists. The assumption has a large effect on OPEX over the period to 2049 because it is compounded
for 34 years. A factor of 0.99 raised to the power 34 is 0.71. The baseline assumption on efficiency is driving
an underlying OPEX reduction of 1 – 0.71, or 29 per cent, which is offset to the extent of around 11 per cent
by drivers such as population and volume supplied, to give a net change in OPEX of 18 per cent. The model
outputs are sensitive to this assumption and Defra should consider whether 1 per cent is the best estimate to
use in the baseline, a it may be too high.
The model predicted a very wide range of possible future bills from £165 to £478/hh/yr. Nera
subsequently revised its assumptions, producing a much narrower range. Not all of this range appears
plausible. Although bills have moved dramatically downwards over time in some other utility sectors, such
as communications, the technology and market structure changes which have been the cause there will not be
replicated in the water sector. Very large real reductions in bills seem implausible though some real
reductions in bills over time are possible, especially for customers in regions where the capital expenditure
drivers are weaker or absent.
The variation in future bills is mostly on the upside (increasing bills). There is little difference between
the bill estimates under the low settings and the baseline settings, but there is a large difference on the
upside. There are reasons why reductions in bills may be truncated from the baseline position, for example,
because capital enhancement expenditure cannot be reduced below zero, while variation on the upside is
much less constrained, but the closeness of the baseline and low scenario estimates suggests that the baseline
13 Quality audit of draft Nera water bills model
has been set too low. A careful review of the baseline assumptions relative to the low and high scenarios is
worthwhile and may lead to some adjustments of the baseline.
Within the body of water customers, average measured and unmeasured bills may change relative to
each other as metering penetration increases. These changes are not presented in the report but they are
visible in the model output sheets. There is no discussion of the calculation of the measured-unmeasured bill
differential or these results within the report, but this is a policy-relevant issue and should be covered, if the
model is intended to cover it within its scope, or alternatively the outputs could be removed from the model.
The model currently predicts very large increases in average unmeasured bills over time. Nera added this to
its list of potential future upgrades.
The long run projections in the model rely on constant compound annual growth rates and highly
uncertain projections of capital expenditure. The model gives a much more accurate picture to 2025 than
it does thereafter. There is no solution to this problem, since the uncertainty is inherent in the problem, but it
does suggest that users should select ranges of assumptions of compounding drivers which are narrower than
the range used for short term projections. That is, users should assume some long term mean reversion of
drivers, and Defra may wish to add this advice to the report.
It would be helpful to present botex and enhancement expenditure as separate figures. Table 4.28
shows that the baseline projection states a 26 per cent reduction in totex in the sewerage network compared
with 10 per cent reduction in treated water distribution. In order to understand these relative trends, it would
be helpful to decompose the total figures into botex (totex less enhancement capex) and enhancement capex
and include these data in the table. The same argument applies to the absolute change in expenditure to 2049.
OPEX and capital maintenance are slowly declining and slowly increasing respectively over time and
the large decrease in totex is due to reduced capex enhancement. The botex and enhancement capex is
stated for the combined services value chain in Table 4.29 and shows that enhancement capex falls by 50 per
cent between 2015 and 2049. This seems plausible. The share of enhancement in totex halves from 53 per
cent to 27 per cent between 2015 and 2049 in the baseline scenario, which again seems reasonable.
The model shows decreases in regulatory gearing, debt divided by RCV, over time. The estimated
regulatory gearing falls from 60 per cent in 2015 to 25 per cent in 2049. The level of debt falls from £40 to
23 billion over the same period. The company financial gearing is not reported. It seems likely that the model
is underestimating the level of debt carried by companies in the late 2020s and beyond and will thus slightly
underestimate the value of the tax shield and therefore slightly overestimate the required return on capital.
The model only reports figures from 2015 onwards and there is no comparison with historic data or
trends. The model starts projections in 2015. The report states that trend analysis has been performed off
model but no details are given. It is not possible to see whether the model’s projections continue smoothly
from the historic data. Greater transparency on the trend analysis and consistency between past and future
would enhance the credibility of the trend assumptions and projections. Nera responded by adding a chart
with historic average household bills.
Contact us:
Vivid Economics Limited T: +44 (0)844 8000 254
The Media Village E: [email protected]
131-151 Great Titchfield Street
London W1W 5BB
United Kingdom
Company Profile
Vivid Economics is a leading strategic economics consultancy with global reach. We strive
to create lasting value for our clients, both in government and the private sector, and for
society at large.
We are a premier consultant in the policy-commerce interface and resource- and
environment-intensive sectors, where we advise on the most critical and complex policy
and commercial questions facing clients around the world. The success we bring to our
clients reflects a strong partnership culture, solid foundation of skills and analytical assets,
and close cooperation with a large network of contacts across key organisations.