Network Reliability Monitoring for ICSGoing beyond NSM and SIEM

Jim Gilsinn• Senior Investigator, Kenexis Consulting– ICS Network & Security Assessments & Designs– Developer, Dulcet Analytics, Reliability Monitoring Tool

• Previous Life – NIST Engineering Lab– 20+ Years Engineering– ICS Cyber Security & Network Performance– Control Systems, Automated Vehicles, Wireless Sensors & Systems

• International Society of Automation (ISA)– ISA99 Committee, Co-Chair (ISA/IEC 62443 Standard Series)– ISA99-WG2, Co-Chair (ICS Security Program)

What is Network Security Monitoring?

• “the collection, analysis, and escalation of indications and warnings to detect and respond to intrusions.”

• “a way to find intruders on your network and do something about them before they damage your enterprise.”

The Practice of Network Security Monitoring, Richard Bejtlich

What is Security Information and Event Management?• SIEM products and services provide “real-time analysis of security

alerts generated by network hardware and applications.”–Data aggregation–Correlation–Alerting–Dashboards–Compliance–Retention–Forensics analysis

Wikipedia

This is NOT A Talk About NSM or SIEM• Richard Bejtlich has some good books on NSM• Chris Sistrunk has some really good presentations on

NSM for ICS• <Pick_a_search_engine> and vendors have information

on both NSM and SIEM–As always, take it with a grain of salt

When NSM Won’t Work?

• “…if you can’t observe the traffic that you care about, NSM will not work well.”

• “Node-to-node activity, though, is largely unobserved at the network level.”

The Practice of Network Security Monitoring, Richard Bejtlich

Example ICS/SCADA Network: Upper-Level Architecture• Most Traffic

Crosses Zone Boundaries

• Less ICS-Specific Protocols

• More Common Platforms

NSM Works Great in Upper-Level Architecture• Most traffic crosses zone boundaries– Network traffic can be observed

• Less ICS-specific protocols– Less ICS-specific protocol knowledge needed to analyze– More available automated network traffic analyzers

• More common platforms– Some platforms allow NSM agents– Security logs available

Example ICS/SCADA Network: Lower-Level Architecture• Most Traffic

Remains Within Zone

• Mostly ICS-Specific Protocols

• ICS-Specific Platforms

NSM Doesn’t Work As Well in Lower-Level Architecture• Most Traffic Remains Within Zone– Traffic doesn’t generally flow through NSM sensors– Most traffic remains in smaller segregated segments within zone

• Mostly ICS-specific protocols– More ICS-specific protocol knowledge to analyze– Less available automated tools to analyze, if any at all

• ICS-specific platforms– Non-standard OS on ICS-specific hardware– Some standard OS exist, with vendor-recommended limitations

NSM Doesn’t Work As Well in Lower-Level Architecture• ICS/SCADA equipment was generally not designed with security in mind– Devices aren’t capable– Information doesn’t exist– Hardware platforms are minimalistic designs

Now What

Network Reliability Monitoring• Negative indicators don’t tell whole story– Absence of security reports doesn’t indicate everything is working properly

• Develop a “known-good” set of traffic signatures– FAT, SAT, commissioning, system changes, periodic testing, etc.– Continuous monitoring possible

• Compare periodic scans to “known-good” signatures• Look for deviations from norm– New traffic streams– Traffic streams performing different than expected– Traffic anomalies

Network Reliability Monitoring Can Be Simple• Complex algorithms not necessary– Hardest part can be isolating traffic streams– Your Wireshark Fu needs to be strong in many cases– Measurement metrics relatively simple (Jitter & Latency)

• Free and low-cost tools exist– Wireshark for traffic stream processing– Spreadsheets for metric analysis

• Basic charts & graphs are easy to generate

Network Reliability Monitoring Can Be Incredibly Hard• Detailed analysis takes knowledge of ICS-protocols– Inner working– Quirks & idiosyncrasies

• Root cause analysis is incredibly difficult– Device real-time architecture– Network performance issues– Lack the tools & techniques to investigate many issues

• Automating the analysis process can be difficult– Building in process knowledge to algorithms

So… What Can You See?

Expected Frequency *Jitter is Variation From Expected Frequency

~10ms Mean Measured Packet Interval±400µs Jitter*

So… What Can You See?

~10.4ms Mean Measured Packet Interval±400µs Jitter*Wider DistributionMore Peaks in Histogram

So… What Can You See? ~1ms Mean Measured Packet Interval±10µs Jitter*Beat Patter @ ~30s

So… What Can You See? ~2ms Mean Measured Packet Interval±1ms Jitter*Beat Patter @ ~50s

So… What Can You See?• OS & application operations– Garbage collection– Antivirus checks & updates– On-screen operator commands

• Network anomalies– Network EMI interference– Signal degradation– Flaky connections

• Security-related incidents

Man-In-The-Middle Testing• Kali Linux VM– Ettercap– ARP Poisoning– All default settings

(script-kiddy style)• Captured traffic off

mirror port• PLC to I/O– EtherNet/IP™– 10ms frequency

• MITM against PLC

MITM Testing – Source IP Address was I/O Block

~10ms Mean Measured Packet Interval±400µs Jitter*

MITM Testing – Source Address was PLC

MITM Testing – Results• PLC and I/O block didn’t care– Both devices continued transmitting in the presence of MITM attack– Likely due to multicast EtherNet/IP protocol

• When PLC traffic isolated from MITM, it showed no change in performance• Ettercap scripts to modify command signals generated– Increase sequence number– Modify output commands to I/O block

• Checked captures without seeing red-flags– VirusTotal– NetworkMiner– Bro

Summary• NSM is good– If you are doing it great– If not, maybe you should

• NSM can’t detect everything, especially for ICS/SCADA networks• There are ways to measure network reliability in the lower layers– ICS/SCADA networks are particularly well suited to this

Questions & Comments?• Jim Gilsinn• Senior Investigator, Kenexis• +1-614-323-2254• Jim.Gilsinn@Kenexis.com• @JimGilsinn