Terms of Reference

1. Focus on illustrating how statistical methods are used to solve business problems and how statisticians interact with colleagues and clients to achieve this.

2. Descriptions of past and on-going case studies

3. Short introductions to their organisations and to the diverse roles of the organisation’s statisticians,

Models

HistoricBasics

WEATHERfrom Met Office

(Actual and forecast)

Reading UniversityRSS 15th June 2005

Shanti Majithia

Forecasting Development Manager

Wokingham, Berks

UK Transmission

Agenda

My Background

Company Background

Application of Statistical techniques within the Company

University and Project work

Conclusion

My Background

Further education in London Maths Stats and Computing

Market Electricity Load research, Manpower planning

Operational Forecasting (Short Time Scale)

Liaison with students and Uni. to assist in data and direction

Presentations: Research paper and Forecasting conferences

Wind Energy, Climate Change, Heating and Cooling Load ( Air Con)

Risk management

Short term Gas Demand and Supply Forecasting

Translating data, analysis and information into decision making tools

Argentina:27.6% Transener

Zambia:38.6% CEC(Copperbelt transmission)

Australia:Basslink (Interconnector to Tasmania)

UK:E&W transmissionGB Gas TransportationLNGGridCom

USA: NEESCom

National Grid Transco - principal activities in regulated electricity and gas industries

Over 21,000 Transmission Towers

Over 13,000 circuit kmof 400 & 275kvtransmission lines and cables

Fibre optics

National Grid - UK : Electricity

300 substations

Electricity

Balance generation and demand efficiently

Ensure quality and security

Non stop process

Keeping the lights onKeeping the lights on

Electricity Transmission Elements

96/29355 ISSUE A SH. 1 OF 1 30-04-99

Power Station

GeneratorTransformer

Transformer

33 kV

To Small Factories, Farms,

Residential Areas and Schools

Large Factories,Heavy Industry

Medium Factories,

Light Industry

11 kV 240 V

132kV

23kV 400kV

}

40 % of Distribution

TRANSCO & IPGTS

TRANSCO

Suppliers

• Producers

• DFO’S

• Storage Operators

• Shippers

• Traders

The UK Gas Industry Model

Competitive Monopoly

Energy Companies Regulated Systems

Gas supply Independent transmission

6300 km Pipelines

CompressorsRegulators

St. Fergus

Teesside

Easington

Rough

TheddlethorpeBacton

Barrow

Terminals

Burton Point

Gas: National Transmission System (NTS)

• 6,600km 450-1220mm diameter pipeline• High strength steel X65-X80 • Operating pressure design

70-94bar• 7 Transco terminals• 24 compressor stations• 400 above ground installations (AGI)

Key Stats• Max demand 02/03 205 GW• Peak Demand (1/20) 240 GW• Energy Supplied 1150 TWh/yr

Compressor

Salt cavitystorage

LNGstorage

LDZ Offtake

Regulator Station

Governor

Meter

CUBIC FEET

0 0 1 2 6 5

High pressure storage

Low pressurestorage

Industry

Terminal

Power Station

Gas: From Beach To Meter

Balance supply and demand efficientlyBalance supply and demand efficiently

Facilitate the marketFacilitate the market

Ensure quality and securityEnsure quality and security

Maximise system capacityMaximise system capacity

Non stop processesNon stop processes

BUTBUT

Gas can be stored => daily balancingGas can be stored => daily balancing

Electricity can’t => real-time balancingElectricity can’t => real-time balancing

Real Time System Operation in Gas and Electricity…..

Application of Statistical Techniques within NGT

Data collection - live metering, market intelligence and field measurement

Data mining e.g. Kohonen SOMs, Genetic Algorithms.

Forecasting Methods

Regression, Box-Jenkins, Bayesian, Neural Network (MLP & ALN), Curve fitting and Holts-Winters, Arch and Garch

Probability and Risk Management

Liaison to keep abreast of modern methods e.g. Statistical methods

Management Information System

Area of Application of Statistical Techniques

Forecasting Energy Demand

Trading advice

Minimising of volatility

Management of probability and risk

Calculating and calibrating climate sensitivity

Health of the assets in terms of the return period

Simple use of statistical methods in plant reliability

Responses on the efficiency of the equipment

Electricity Forecasting Techniques

Multiple linear regression

Last 3 years of historic data

Summer (BST) and winter (GMT)

Weekdays / Sat / Sun

Special days excluded

‘Conventional’ and ‘Trend’ models

~ 120 models per annum

Interpolation between cardinal points for half hourly resolution

Forecasting Tools

Oracle database Weather and demand feeds StatGraphics EViews SAS PREDICT & Forecaster Clementine NN and ALN Genetic Algorithm Library (MIT)

Weather Input

Historical Demand Input

Mathematical& Statistical

Models

The Forecast

The Forecasting Process

Demand - Influences

Seasons/ Weather

“Exceptional events”

TV

00:00 02:00 04:00 06:00 08:00 10:00 12:00 14:00 16:00 18:00 20:00 22:00 24:00

15

20

25

30

35

40

45

50

55

Typical Summer Day

Minimum Summer Day

Typical Winter Day

Maximum Winter Day

GW

NGC System demand during 3 minute silence on 14 September 2001 in memory of the tragedy in America

35000

35500

36000

36500

37000

37500

38000

38500

Dem

and

200

1

33000

33500

34000

34500

35000

35500

36000

36500

Dem

and

199

9

3 minute silence

Remembranceday 1999Eclipse 11-08-99

Previous Drop on the 11 August 1999 - 2200MW The Solar Eclipse

The start of the Drop, just before 11:00

2700MW Highest ever drop in Demand.

Remembrance Day 1999 gave a 750 MW drop

13 minute duration

The Effect of Temperature on Demand

0

1000

2000

3000

4000

5000

6000

7000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30Temperature

Dem

and

Eff

ect

(MW

)

COLDHigh Demand

Comfortable

HOTHigh Demand

Degrees Centigrade

The Effect of Illumination on Demand

0

500

1000

1500

2000

2500

3000

3500

4000

4500

0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 105 110 115 120 125 130 135 140 145 150 155 160

Logarithmic Function of Illumination

De

ma

nd

Eff

ec

t (M

W)

DULLHigh Demand

BRIGHTLow Demand

Four Weather Variables

• Average Temperature [TO]: average of 4 spot hourly temperatures up to current hour

• Effective Temperature [TE]: TO lagged to 50% with TE from 24 hours previous

• Cooling Power of the Wind [CP]: empirical combination of temperature and wind speed

• Effective Illumination of the Sky [EI]: (EI=MI-ID), where ID is a function of visibility, numbers and types of cloud layers and amounts of precipitation and MI is maximum illumination. In the logarithmic domain.

Winter Week Day Peak Demand ModellingMultiple Regression Model Of Demand

Weekday Darkness Peak Demand =

+ Weather Dependant Demand

+ Day of Week

An econometric regression model of the weekday darkness peak is determined on the four previous winters demand & weather data

+ Seasonal trends

+ error terms

The days affected by Christmas & New Year holidays are excluded from the sample

Mean Darkness Peak Demand

Weather Dependant Demand

Weather Dependant

Demand functionTEt+2TEt

2+EIt+CPt

Example of a Weather forecast data

Example 1: MG3 file for receiving Forecast Data (Samplefile format for 0800 bulletin.)

‘A00’,’MG3’,’19082002’,’073030’,007‘D00’, ‘19082002’, ‘080000’‘D10’, 1,,

‘D01’, ‘LON’, ‘19082002’, ‘090000’, 15.0, 0,,,‘D01’, ‘LON’, ‘19082002’, ‘110000’, 15.0, 0, 04, 0, ‘SSW’‘D01’, ‘LON’, ‘19082002’, ‘130000’, 15.0, 0,,,‘D01’ ‘LON’, ‘19082002’, ‘150000’, 15.0, 0, 04, 0, ‘SSW’‘D01’, ‘LON’, ‘19082002’, ‘170000’, 15.0, 0,,,‘D01’, ‘LON’, ‘19082002’, ‘190000’, 15.0, 0, 04, 0, ‘SSW’‘D01’, ‘LON’, ‘19082002’, ‘210000’, 15.0, 0,,,‘D01’, ‘LON’, ‘19082002’, ‘230000’, 15.0, 0, 04, 0, ‘SSW’‘D01’, ‘LON’, ‘19082002’, ‘010000’, 15.0, 0,,,‘D01’, ‘LON’, ‘19082002’, ‘030000’, 15.0, 0, 04, 0, ‘SSW’‘D01’, ‘LON’, ‘20082002’, ‘050000’, 15.0, 0,,,‘D01’, ‘LON’, ‘20082002’, ‘070000’, 15.0, 0, 04, 0, ‘SSW’‘D01’, ‘LON’, ‘20082002’, ‘090000’, 15.0, 0,,,‘D01’, ‘LON’, ‘20082002’, ‘110000’, 15.0, 0, 04, 0, ‘SSW’‘D01’, ‘LON’, ‘20082002’, ‘130000’, 15.0, 0,,,‘D01’, ‘LON’, ‘20082002’, ‘150000’, 15.0, 0, 04, 0, ‘SSW’‘D01’, ‘LON’, ‘20082002’, ‘170000’, 15.0, 0,,,‘D01’, ‘LON’, ‘20082002’, ‘190000’, 15.0, 0, 04, 0, ‘SSW’‘D01’, ‘LON’, ‘20082002’, ‘210000’, 15.0, 0,,,‘D01’, ‘LON’, ‘20082002’, ‘230000’, 15.0, 0, 04, 0, ‘SSW’‘D01’, ‘LON’, ‘20082002’, ‘010000’, 15.0, 0,,,‘D01’, ‘LON’, ‘20082002’, ‘030000’, 15.0, 0, 04, 0, ‘SSW’‘D01’, ‘LON’, ‘19082002’, ‘050000’, 15.0, 0,,,

Gas Forecasting - suite of models using different techniques

Profile (ARIMA)

STF (Complex regression)

Neural network

ALN (Adaptive logic network)

Inday (Simple regression)

Bayes (Complex regression)

Box 1 (Box Jenkins)

Box 2 (Box Jenkins)

Sumest (Complex regression)

Wintest (Complex regression)

D-1

D-1

D-1

D-1

D-1

D-1

D-1

D-1

D-1

Averageweighted according to

performance over last 7 days (Combination). Further adjustment made based on

recent combination error (CAM)

D

D

D

D

D

What Does a Gas Model Look Like?

PROFILE – WITHIN DAY MODELPROFILE – WITHIN DAY MODEL

PROFILE model uses the Box Jenkins technique to forecast within day gas demand. There are two different models in the program. Model 1 is usually used for the 10am forecast and model 2 for the rest of the day. However, if the 9am temperature is greater than either the 1pm or 3pm temperature then model 1 is used for the 1pm and 4pm forecasts.

Model 1 (at hour k) (used for 10:00 forecast)

7Dt(h) = w07Tt

(3) + w17Tt(6) + w27Tt

(9) + w37Dt(k) + (1-1B) (1-7B

7) at

Model 2 (at hour k) (used for forecasts at other times)

7Dt(h) = w07Tt

(h-1) + w17Dt(6) + w21

k7Dt(j) + (1-1B) (1 - 7B

7)at

where Tt(h) is the temperature at hour h on day t,

Dt(h) is the corresponding hourly demand on day t,

at is the error in the forecast demand for hour h on day t,

B is the backward shift operator i.e. Byt = yt-1

w0, w1, w2, w3, 1, 7 are model parameters.

.

PROFILE model uses the Box Jenkins technique to forecast within day gas demand. There are two different models in the program. Model 1 is usually used for the 10am forecast and model 2 for the rest of the day. However, if the 9am temperature is greater than either the 1pm or 3pm temperature then model 1 is used for the 1pm and 4pm forecasts.

Model 1 (at hour k) (used for 10:00 forecast)

7Dt(h) = w07Tt

(3) + w17Tt(6) + w27Tt

(9) + w37Dt(k) + (1-1B) (1-7B

7) at

Model 2 (at hour k) (used for forecasts at other times)

7Dt(h) = w07Tt

(h-1) + w17Dt(6) + w21

k7Dt(j) + (1-1B) (1 - 7B

7)at

where Tt(h) is the temperature at hour h on day t,

Dt(h) is the corresponding hourly demand on day t,

at is the error in the forecast demand for hour h on day t,

B is the backward shift operator i.e. Byt = yt-1

w0, w1, w2, w3, 1, 7 are model parameters.

.

NTS Supply Forecasting Model types

Every supplypoint

End of

dayWithin

dayRegression of DFNs & AT-Link nomsEvery

hour

When For What Horizon How - Model type

Nat End of

dayWithin

dayRegression of DFNs & AT-Link nomsEvery

hour

Every supplypoint

End of

day

Day ahead Regression of change of supplyEvery

hour

Every supplypoint

Every hour

7 days ahead

Holts-Winters (Time series)Once per day

What is a Holts-Winters Model?

stt

tt

tttt

ttst

tt

SL

YS

bLLb

bLS

YL

1

1

1

11

11

where s is the length of the seasonality. L is the smoothed level of the series, bis the trend of the series and S is the seasonality component.

The Forecast uses the following equation;

stttt SbLF

Understanding Data Questions

What to look for in the data before preparing forecasts

How to treat data when “problems” are recognised

How to prepare forecasts using different models and techniques

When each forecasting model is appropriate

How to use forecasts effectively after they are prepared

Why is a forecast needed?

Who will use the forecast, and what are their specific requirements?

What level of detail or aggregation is required and what is the proper time horizon?

How accurate can we expect the forecast to be?

Will the forecast be made in time to help decision making process?

Does the forecaster clearly understand how the forecast will be used in the organisation?

Key Questions

Projects and Case Studies

The seasonal forecast of electricity demand: asimple Bayesian model with climatological

weather generator

Sergio Pezzulli, Patrizio Frederic, Shanti Majithia,

Coloured areas are clusters, each with a distinctive daily demand profile. Red text is their interpretation.

Data mining --- Clustering of Electricity Profiles

Clustering of Gas Profiles

Kohonen Network (SOM) Analysis

Yellow-ish areas indicate similar profiles, Red-ish areas indicate more varying profiles.

Jan & Dec

Jan Feb Mar & Nov

Apr May & Oct

June July Aug & Sept

Forecast DemandLondon

0.5

1

1.5

2

2.5

1 9 17

25

33

41

49

57

65

73

81

89

97

105

113

121

129

137

145

153

161

169

177

185

193

201

209

MC

M

1 Day Lag Forecast

Real Demand

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

1 10 19 28 37 46 55 64 73 82 91 100 109 118 127 136 145 154 163 172 181 190 199 208

Error

North Thames LDZ, Early Jan 2003

New up coming Challenges

Windpower

Variable

Uncertain

Uncertain uncertainty

Danger: possibility of sudden loss

Weather differences can be at finer geographic resolution

Volatility and UncertaintyHow best to model? Ensemble forecasts?How to make operational decisions?

Site Clustering

Site clustering can be used to produce a more accurate national prediction by taking local conditions into account

The main way of achieving this is to have a ‘reference’ farm which is representative of the cluster

It is possible to then use cluster predictions as inputs to a national model or simply upscaled

One further thought is to forecast both a reference farm and a cluster separately and use them to create a more stable regional prediction

Daily Load Forecasting using ARIMA-GARCH and Extreme Value Theory

University of Loughborough EPSRC Project

Application

Climate Change Impacts on Electricity demand can be categorised into a long term (monthly) and short term (daily and hourly) load forecast.

Long term load forecast using the multiple regression approach completed. The results are satisfactory. 80 years projection requires the UKCIP scenario and BESEECH data (population, GDP, consumer spending).

Short term load forecast using Box Jenkins and Extreme Value Theory is also completed. Waiting for hourly climate data from BADC and CRU before we can extend our daily/hourly projections to 2080s.

ARIMA (p, d, q) Model

The AutoRegressive Integrated Moving Average (ARIMA) model is a broadening of the class of ARMA models to include differencing.

Reason: daily and hourly pattern are volatile and shows a strong seasonal pattern. p: no. of autoregressive terms, d: the number of non-seasonal differences and q = no of lagged forecast errors in the prediction equation.

q

j noisewhite

t

d

kregressor

k

ragemoving ave

jtj

p

iregressiveauto

i ktXityCty1 11

, ˆ

ARIMA(1,1,1) is used

Probability Distributions

nt is a standardized, independence, identically distributed (iid) random draw from some probability distributions.

3 distributions are used for this purpose:-

a) Normal

b) Student-t

c) Extreme Value Distribution

For quantiles > 0.95, extreme value distribution is used.

Example of Scenario Forecasting (with max and min scenarios)

17000

19000

21000

23000

25000

27000

29000

31000

33000

HALF-HOURLY

MW

Forecast issued on Friday 12:00hrs

Maximum risk scenario

Minimum risk scenario

Actual

Combination of Distribution-- ExampleLink between Annual Peak and Weekly Peak

The density traces shows how the median of the simulated winter peak distribution cuts off an area of about 12% on the corresponding distribution of simulated weekly peak demands.

VariablesSimulated Weekly Peak DemandSimulated Winter Peak Demand

Density Traces

dens

ity

47 48 49 50 51 52 53 54 55 56 57 58 59 60GW

0

0.5

1

1.5

2

2.5

3(X 0.0001)

Winter ACS Median

12% Area cut off

Weekly Peak

Distribution

Probability Distribution

51000

52000

53000

54000

55000

56000

57000

58000

59000

60000

61000

620000 5 10 15 20 25 30 35 40 45 50 55 60 64 69 74 79 84 89 94 99

%

MW

Conclusions

Various statistical applications demonstrated

Wide variety of Statistical method used in data rich Energy business

Opportunity for Statistician/Business Analysis