Quite a crisis

Gloucester water quality

incident 2015

Gloucester Water Supply

Gloucester’s water and sewerage services

handed over to MidCoast Water in July 2011

4,000 people in 1,900 homes

WTP capacity 4.5 ML/day

3 service reservoirs and 7 booster pump

stations, town reticulation includes 28km of

AC main

The incident – Day 1 – Tues 24 March

6am: Operators noticed a stronger than usual chlorine smell at the WTP

Morning – normal reticulation sampling taken, with chlorine levels 3.7 & 3.8 mg/L.

Operator reduced the chlorine dose at the WTP based on the samples

12.04pm – first call from a customer – water tasted like sulphur and was bitter

A work order was raised, mains flushing commenced

2pm to 3pm – 2 more calls, one being from Gloucester Hospital

Minor feedback commenced on social media

Plan to further investigate the following day to prevent the use of on-call

The incident – Day 2 – Wed 25 March

9am – first media enquiry – responded to by customer

service. Based on operational advice, the issue was

poorer river quality that had occurred over the

weekend

Customer calls increase rapidly (314, or more than

three time average)

Customer service recommend a recorded message be

added to assist with customers left on hold

Social and traditional media in full flight…

Day 2 was a long day…

Chlorine levels registered above field instrument range in reticulation (>8.8 mg/L)

11.20am Incident notified to Executive – an incident management team put together.

Telstra have planned shutdown of mobile service between approximately 11am and 3pm

Boil water alert was ruled out – needed to be a ‘do not drink’ alert

12 noon NSW Health and NSW EPA notified

Additional crews arrive at Gloucester from Taree to assist with flushing and testing

Door knock and letter drops rules out due to time of response required. Expectation of a

less than 1 day incident

Crates of bottled water purchased and conveyed to Gloucester

Incident management team

Incident controller – Executive Manager Service Delivery

Board members, television, radio – Acting General Manager

Media releases and social media – Public Relations Manager

Site incident control – Group Manager Catchment & Treatment

Customer service coordination and notification – Customer Service Mgr

Corporate support – Executive Manager Corporate Services

Social media

We started using Twitter in Feb 2014

during Level 3 water restrictions

Commenced Facebook in June 2014

after formalising a social media policy

and procedure

Social media

x

x

x

X

X

Social media

Social media

Day 3 – Thurs 26 March

Independent engineering investigator engaged by MidCoast Water to:

- identify the root cause of the incident

- comment on MidCoast Water’s response to the incident, and

- recommend any improvements to the response

Flushing and intensive chlorine monitoring continues

Confirmed to Newcastle media that the incident had ‘nothing to do with

Coal Seam Gas’

Day 4 – Fri 27 March

$50 rebate on water bill announced

9am MCW Councillors and Executive

arrive in Gloucester to speak with

community members

11am Independent investigator Hunter

H2O arrives on site in Gloucester to

commenced site review.

Likely causes

Hunter H2O established five likely causes

1. Dosing system ran on after plant shut down

2. Dosing system left in test mode (25mg/L)

3. Dosing system set to maximum dose rate (25mg/L)

4. Sabotage

5. Syphoning

Recommended the plant be run manually

until rectified

The following weeks Customer sample indicates worse than measured in reticulation

Incident debrief of MidCoast Water staff undertaken

Temporary online chlorine analyser installed, plus daily records being initiated

of hypochlorite storage tank levels

Plant run manually with hypochlorite tank isolated at end of each run

Computer modelling undertaken of a high chlorine dose by MidCoast Water

Further independent investigation

Joint strategic meeting with NSW Health and NSW Office of Water convened

to consider root causes of the incident

The technical cause

Siphoning of hypochlorite solution into the

treated water after plant shut down.

The problem was intermittent and couldn’t be

replicated until the hypochlorite tank was refilled

Hypochlorite dosing pumps and loading valves

improperly designed and commissioned

The root causes

A shortfall in the right staff resources

Inadequate training of current staff

Lack of quality systems and internal controls

No integrated business management system

Lack of system knowledge management

Facilities not up to current design standards and a lack of change management

Incomplete implementation of the drinking water quality plan

The root causes – a graphical example

Areas for improvement

Time taken to realised there was an issue.

Clear evidence of a problem on Tuesday afternoon

No Incident Management Plan

There wasn’t a listing of critical customers (late notification)

There is a misalignment of MCW CCP, DWMP and the ADWG’s

Staff understanding of CCP’s

Areas for improvement

Shortage of resources:

- only one staff member had a sound understanding of the reticulation

- Staff were stood down for fatigue and concern with overtime

The level of chlorine in system was not quantified

The issues with the dosing system could have been picked up earlier

A chlorine analyser at Tyrell Street had not been calibrated or attended for a

number of years

Previous reviews of the WTP had identified the need for online instruments

Areas for improvement

There was a lack of reviews undertaken

on the hypo system

- Design basis and design reviews

- Compliance checks with AS3780

- HAZOP, Chair (1-3)

- Dosing location

- Functional description, etc.

Areas for improvement

There are numerous other risks at the plant

- Lack of control, automation and monitoring

- Compliance with standards – OHS, AS 3780

- Lack of documentation and update manuals

or procedures

- Training of staff and resources

There are other issues identified in the plant

that pose a risk to water quality

What worked well

Once the incident was declared everything worked as if there was a incident

management plan in place

‘Response by the Executive and Staff during the incident was exemplary’

Notification to regulators

Updates to the public (Mainstream and Social Media)

$50 Rebate and payments to commercial customers

Independent Review

‘Meet the Public Day’

Apology Letter to Public

Action planning

Many actions were able to be implemented in the week of the

incident

60 actions identified in regards to engineering/technical causes

and scheduled for implementation over a 6 month period

56 actions identified in regards to the root causes of the

incident and scheduled for implementation over a 2 year period

A step change in resources was required for implementation

Summary

Does this sound like a water treatment plant you might own and operate?

Blame the systems not the people

Hard lessons create a burning platform for change – a much sharper focus

business systems and risk management

Running ‘lean and mean’ can get you in trouble

‘Normal’ may actually be risky. Normal operation needs regular challenge

Transparency is critical for community and regulator confidence.

Have a plan to ensure that it won’t happen again. Share it widely.

Not sure how the situation would have unfolded prior to MidCoast Water

Credit

Independent engineers – Hunter H2O

- Paul Thompson, Alan Thornton, Clara Laydon, Evan Jack

MCW staff and councillors

Thank you