Uncertainty in Automation:Anomaly Detection in Event-Based Systems

Dawn TilburyLinday Allen (PhD) and John Broderick

University of Michigan

2

Outline

• Example problem: inconsistent logical behavior

• Developed solution: Anomaly detection– Model generation using observed data– Performance assessment of models using known

“good” and “bad” behavior– On-line anomaly detection

• Industrial Application– Academic assumptions meet industry realities– Resolution and results

• Lessons learned

3

Example from our testbed

s1

Part1

PartReadyRelease Pallet

LoadPart1Part1

PartReady

Correct, typical behavior Incorrect behavior

• No model of entire system’s correct behavior• Manual inspection required to find this anomaly

• Laborious, offline

ResponseOPC Tag ResponseOPC Tag

4

Approach: Anomaly detection using model generation

• Goal: diagnosis of system level event-based faults in mfg systems without use of pre-existing formal model

• Method: - Generate models based on training data- Detect anomalies on-line by comparing traces

to models- Advise the operator when anomaly occurs

PlantPlant

Logic Control System

FaultdiagnosisAnomalydetection

Operator

ControllerController

All Events

Events

Op. Feedback

Anomalies

Knowns & Unknowns

• Known: – Resources in

system• Robots• CNC machines• Pallets

• Measurable:– OPC tag changes

• Communication events between controllers

• Unknown: – Formal model of the

systemCould be constructed but is time-consuming and error prone

– Logic control codeWritten by different people at different times in different languages

– Correct event orderMany different orders may be acceptable

6

Solution approach

Model Generation

adbe...

efad...

abfc...

ok

ok

ok

Resource Info

eafd...

Performance Assessment

acea...

ok

not

22316

Fault Detection

abfe...

okabfe...

abfe...

not

OR

Given resource information and

strings of “ok” events

Create a set of models that can generate these

strings

Given some “ok” and “not ok” strings, compute the

performance of each model

Given a new string, determine whether the

models accept it (weight by model performance)

If not, where is the anomaly

Anomaly Detection Method

• Inputs:– Streams of events from system in operation– Resource information, including mapping of

events to resources

• Outputs:– Set of models that represent the system

behavior– Model performance on training data

• On-line detection:– Score for each string – anomalous or not

7

Machining Cell: Physical Set-Up

• Problem: G2 will have raw parts and at least one CNC available, but G2 incorrectly waits

• Resources: – Gantries, CNCs, buffer at hand-off

• Events: PLC data recorded via Ford data collection system

G1 G2 M1 M62 3

4

1

Entry Hand-off

Reject

Exit

Data collection set-up

• Data from each machine & gantry– Bits include: Cycle

End, Good/Bad Cycle, Wait Aux, Blocked, and Starved

– PLC message generated each time particular bit changes occur

• Approx. 11,000 parts worth of data

9

IT SystemIT System

PLC

DrivingLogic

DrivingLogic

FunctionBlock

FunctionBlock

PLC

Driving logic

FunctionBlock

FunctionBlock

DrivingLogic

DrivingLogic

(270,000 PLC messages)

Identified Inconsistencies

• What we thought we would get: – Well-defined strings

of events

– Events that acquire/release resources recorded

– Unique mapping of PLC bits to events

– Many strings, starting from the initial state, labeled as “good” or “bad

• What we got: – Not every event

triggers a message multi-bit change (order is uncertain)

– Not all resource events captured in data collection

– Some bits used for multiple purposes

– One huge log file with no defined “beginning”

Resolution of Inconsistencies

Academic Assumptions

Industry Realities Resolution

1 Resource events available

Some events filtered in data collection

I: Logic changed

2 String of ordered events

Multiple bit changes per message possible

A: Heuristic decision algorithm

3 Consistent bit-meaning mapping

Inconsistent bit-meaning mapping

I, A: Logic changed, pre-process data

4 Event streams start in initial state

System runs continuously

A: Nec. condition to create stream

5 Separate, labeled streams

Continuous, unlabeled stream

A: Splitting, labeling algorithm

Ford data Word 18 bits 8-10 give the CNC ID

Gantry waiting: word 19 bit 9 is high

Lessons learned

• Sometimes you can adapt/improve your method to handle given uncertainties– Multiple models when system model unknown– Multiple bit changes uncertain event order– Initial state unknown– Advise operator instead of closing the loop

• Sometimes you have to decrease the uncertainty by improving the system– Consistent bit/event mapping– Unobservable events for acquiring resources

13