Customer Complaint Surveillance Vendor Integration Forum

Jointly Hosted by the U.S. EPA andAmerican Water Works Association

February 27, 2011

12:00 - 3:00 p.m.

AWWA CS/IMTECH ConferenceDallas, TX

2

Presentation Outline

1. Overview and Approach of Customer Complaint Surveillance

a. Program and approach

b. National adoption

2. Implementing Customer Complaint Surveillancea. Features and characteristics

b. Integration and benefits

2

Overview and Approach of Customer Complaint Surveillance

3

4

Distribution System Vulnerabilities

• Large, complex, and accessible:– Commercial & residential service connections– Fire hydrants– Finished water storage

• Impossible to eliminate all access, but …– key system components can be hardened

• Difficult to contaminate an entire city via the distribution system, but …– fairly easy to impact small sections or individual

buildings

4

5

• Consequences of water contamination:– Adverse impacts on public health

– Loss of water for public safety uses

– Economic damage

– Loss of consumer confidence

• An attack using contaminants …– is likely to achieve multiple terror objectives

– does not have to produce casualties to be successful

– may be perceived as an especially serious threat by the public

Nature of Contamination Threats

5

Water Security (WS) initiative

• EPA’s WS initiative program addresses the risk of intentional contamination of drinking water distribution systems.

• The initiative was established in response to Homeland Security Presidential Directive(HSPD) 9, under which the Agency must, “develop robust, comprehensive, and fully coordinated surveillance and monitoring systems . . . for . . . water quality that provides early detection and awareness of disease, pest, or poisonous agents” (Jan 30, 2004).

6

7

CWS Design Objectives

• Detection of a broad spectrum of contaminant classes.

• Achieve spatial coverage of the entire distribution system.

• Detect contamination in sufficient time for effective response.

• Reliably indicate a contamination incident with a minimum number of false-positives.

• Provide a sustainable architecture to monitor distribution system water quality.

• Growing emphasis on Dual-use aspects.

7

8

IntegratedContamination

Warning System

CWS Monitoring Components

Public health

surveillance

Enhanced security

monitoring

Water quality

monitoring

Sampling and

analysis

Customer complaint

surveillance

9

CWS Architecture

Phase II.Consequence Management

Remediation and Recovery restores a system to normal operations and may include:

· System characterization

· Remedial action

· Post-remediation activities

Phase I.Routine Monitoring & Surveillance

Online Water QualityCredibility Determination Actions confirm or rule out contamination and may include:

· Site Characterization

· Outside data sources

· Laboratory confirmation

Response Actions protect public health during the investigation process and may include:

· Isolation

· Flushing

· Public alerts/notifications

Public Health

Sampling and Analysis

Enhanced Security

Customer Complaints

· Event Detection

· Initial Trigger Validation

PossibleDetermination

9

National Adoption

The WS initiative comprises work in 3 areas:

1) Develop a conceptual design for a system that achieves timely detection and appropriate response to drinking water contamination incidents to minimize public health and economic impacts;

2) Demonstrate, test, and evaluate the conceptual design in contamination warning system pilots at drinking water utilities;

3) Issue practical guidance and conduct outreach to promote voluntary national adoption of effective and sustainable drinking water contamination warning systems.

10

11

National Adoption

A New EPA Approach…

• Implement an “Inside-Out” Approach– Integration of the required coding into established call

management and work management software would allow for a more efficient “inside-out” approach to CCS adoption

– Necessary mechanisms to track, transfer and analyze CCS data are already built into existing software systems

– Make utilities more likely to adopt aspects of a contamination warning system – more like a plug-and-play system

12

Implementing Customer Complaint Surveillance

13

14

Customer Complaint Surveillance

• Features and Characteristics• Integrating Customer Complaint Surveillance

into Contact/Work Management Systems• Vendor Benefits

What is Customer Complaint Surveillance?

“Customer Complaint Surveillance (CCS) encompasses the customer complaint collection process, data management, data analysis and anomaly detection of customer complaints, notification of anomalies, and investigation procedures.”

15

16

Features & Characteristics

Customer Complaint

System

Contact Center

OperationsStaff

Water Quality

Specialist

FO

CU

SF

ILT

ER

FU

NN

EL

Primary Source:Water Utility Contact Center

Secondary Source:Other Agencies

Secondary Source:

Other Utility Departments

Non Water QualityRelated Complaints

(As determined by Customer)

Non Water QualityRelated Complaints

(As determined by CSR)

Water Quality ComplaintsRelated to System

Operations

(Main Breaks, Maintenance, etc.)

Water Quality Specialist analyzes remaining

complaints for indications of possible contamination

Features & Characteristics

• What does water contamination look like?– Aesthetic (organoleptic properties) not associated

with a benign cause.

– Not associated with common utility water quality problems that may have similar complaint descriptions.

Rusty/dirty water Cloudy water Water pressure

– Temporal clustering complaints linked by time.

– Spatial clustering complaints linked by distribution system characteristics.

17

18

Features & Characteristics

• Characteristics of a Customer Complaint System

– Provide a mechanism for a utility to “baseline” their data and establish alarm thresholds

– Near real-time analysis of data using automated surveillance algorithms and code

– Alarm notification of anomalies (alerts)

– Allow for easy integration with other utility systems

19

Features & Characteristics

• Provide a mechanism for a utility to “baseline” their data and establish alarm thresholds– Review / analyze historic data

Analysis of Trigger Values for GCWW paper

– Identify “normal” complaint volume

– Set thresholds for alarms Alarm Estimation Tool (AET)

Baselining Historic Data

20

http://water.epa.gov/infrastructure/watersecurity/techtools/index.cfm

Establish Alarm Thresholds

21

Features & Characteristics

• Detection of Anomalies

– Collect all complaints in a single location

– Identify anomalous volume of water quality calls Multiple automated algorithms running in parallel Executes in near real-time Temporal clustering Spatial clustering

– Provide actionable notifications to appropriate personnel

– Provide procedure for closing out alarms

22

23

Integrating CCS

• Implementing CCS at GCWW

– Reviewed / analyzed historical data

– Established “normal” vs. anomalous complaint volume

– Identified data resources Contact Management Systems

» Interactive Voice Response (IVR)» CSR characterization (keystrokes)» Work/Asset Management System

Built Event Detection System (algorithms)

Notification (email)

GIS – distribution system (Hydra)

24

Features & Characteristics

• Types of Alarms– Analysis Algorithms (at GCWW)

Algorithm code (Java)» Temporal anomalies (GCWW thresholds)

1 day scan 2 day scan 7 day scan CUSUM

» Spatial anomalies Neighborhoods Zip codes Pressure zones Other hydraulically significant areas

Spatial Analysis

25GCWW - Hydra Map

Features & Characteristics

• Provide notifications to appropriate personnel– Immediate alert notification (near real-time)– Possible communication channels

Short Message Service (SMS) Email Auto-dialer User Interface pop-up (SCADA) Twitter

– Notification contains actionable information Alert type details Location details Procedural instructions (investigations)

26

27

Features & Characteristics

Water Quality Department receives an email alerting of a CCS alarm and begins the investigation.

Notification includes:• Algorithm that detected the anomaly

• Dates and times of complaints

• Locations of complaints

• Annotated information about the call

• CCS Alarm Notification

27

Features & Characteristics

28

• Investigation Process:– Interview callers– Review call data

Analyze the location and nature of the complaint to determine if calls are clustered

– Review plant and distribution system operations Assess the potential for operational changes to impact

aesthetics of water– Review distribution system work

Breaks and repairs may impact color, turbidity, and odor of the water

– Review available water quality data Analyze water quality data in the vicinity of the call cluster

to determine if there is a link– Close out alarm

Integrating CCS

• IT Considerations– Design flexible architecture

Standardized approach Parameters configurable by managers

» Notification templates» Notification recipients» Thresholds

– Leverage existing systems and capabilities Existing systems and applications Network infrastructure Programming expertise Security

– Store alarm data for retrospective analysis– Account for time delay in receiving data

29

Vendor Benefits

• End Users

• AWWA/EPA

• Tool Developers

• System Integrators

30

Vendor Benefits

• End Users – Utilities will be looking for this functionality

Saves time Saves money Saves lives Dual-use for improving normal operation Off-the-shelf solutions offer low implementation cost

for utilities

31

Vendor Benefits

• AWWA/EPA– AWWA and EPA are encouraging voluntary adoption

(creating demand)– Growing body of research and guidance– Consistent offering from vendors provides the supply– Vendors benefit from existing pilot implementations

Cincinnati, Dallas, New York City, Philadelphia, San Francisco

– Dual-use, All hazards emphasis Improvements in monitoring and security Federal guidelines / recommendations

32

Vendor Benefits

• Tool Developers– CCS functionality provides:

More appealing product offering » Market differentiation

Easy implementation – most features already in existing tools

Makes it easier for system integrators to implement

• System Integrators– CCS functionality provides:

Consistent implementations Efficiency of off-the-shelf solutions Low implementation cost for the utility

33

Utility Case Studies

• Implementing Customer Complaint Surveillance– Bryan May, Greater Cincinnati Water Works– Charles Zitomer, PE, Philadelphia Water Department– Cassia Sanchez, PE, Dallas Water Utilities

34

Greater Cincinnati Water Works

35

Philadelphia Water Department

36

• Leverage Existing Resources– GIS Investment– Skilled Workforce– Existing Data Systems– Existing Business Processes

• Improve and Streamline Customer Response• Conduct Field Investigations more Efficiently• Integrate with New Systems

– Cityworks (Work Order Management System)– CWS Dashboard to assist with CCS Alarm

Investigations CWS Spatial Dashboard is a Web Application allowing

remote access

37

Philadelphia Water Department Approach

38

Philadelphia Water Department System Architecture

• Using Cityworks as our Work Order Management System– Open Architecture database– Spatially Compliant Data structure– Web Based Application– Configured Cityworks to maximize its benefits for CWS

• All Water Quality Service Requests are funneled through the Call Center and captured in Cityworks

• All Work activities on the Distribution System will be captured in Cityworks– This will help with associating water quality alarms with

potentially related distribution system activities

39

Philadelphia Water Department Implementing CCS

Philadelphia Water Department CCS and EDAs

• EDAs ...– Are used to continuously process

water quality complaint information and determine whether a pre-defined complaint threshold has been exceeded.

• An EDA’s purpose is ...– To bring a potential water quality event to the attention

of a human investigator using real-time customer complaint information.

• EDA Thresholds were developed based on:– Water Quality Complaint Type– Spatial Location (Clustering-leveraging GIS)– Date and Time

40

41

Philadelphia Water Department CCS and EDAs

• Example of an EDA– Using 1-, 2-, and 7-Day Scan Windows– Monitoring multiple water quality

parameters.

42

Philadelphia Water Department CWS Dashboard

• The CWS dashboard and the EDAs will assist the investigator by:– Automatically displaying only the

relevant service request and work order types from the Cityworks data

– Alerting them to activity occurring within a specific radius of the complaint(s).

• The CWS Dashboard and the EDAs will also:– Reduce the investigation and response time by

promptly providing relevant information during alarm conditions.

43

Philadelphia Water Department Example Dashboard Application

44

• PWD’s approach to leveraging existing GIS and Cityworks will:– Allow the department to capture all

water quality complaint calls and work performed in the distribution system on the same map.

– Allow PWD Operations to detect system issues and failures.

– Improve customer service.– Provide PWD with the opportunity to standardize

business processes and protocols for water quality complaint investigation.

Philadelphia Water Department Conclusions

Dallas Water Utilities

45

Dallas Water UtilitiesCCS

46

Looking Forward

• Future Collaboration – AWWA, EPA, Utilities & Private Sector– Input from stakeholders and ideas– Start information sharing– Webinars

• Expansion Pilot Completions– CCS Implementation & Evaluation

Dallas Water Utilities Philadelphia Water Department New York City Department of Environmental

Protection San Francisco Public Utilities Commission

• Real Benefits - Goal Accomplishment

47

Acknowledgements

• American Water Works Association • US EPA WS initiative Pilot Cities:

Cincinnati, Dallas, New York City, Philadelphia, and San Francisco

• US EPA Water Security initiative Team• Computer Sciences Corporation

48