Intelligent Systems DivisionManufacturing Engineering Laboratory

Performance Testing: The Effects of Antivirus Software on the

Operation of PC Based HMI Software

Joe Falco Manufacturing Engineering Laboratory

National Institute of Standards and Technology

February 18, 2004

NIST Industrial Control Security Testbed Architecture

Bottling Plant Simulation

• DeviceNet I/O network

• Three controller options

• PC-based software PLC

• Modicon hardware PLC

• DeltaV Hybrid Controller

• SQL database for data logging

Water Distribution SCADA Simulation

• Ultrasonic Level Transmitters• Analog Flow Meters• Liquid Level Switches• Centrifugal Pumps

• MTU Allen-Bradley ControlLogix/Flex IO• RTUs Allen-Bradley SLC500• DNP 3.0 Serial• Ethernet

Performance Testing

Provide performance measures of PC based control software execution vs. modes of operation of concurrently executing security software

Note: Any results will be reported in aggregate,

or with any vendor-identifying information

removed.

Antivirus vs. HMI Performance

• Map functionality of both antivirus software packages.• Configure HMI software at upper and lower bounds.• Record antivirus installation and default configurations.• Test procedures least intrusive to most intrusive.• Design test procedures to be repeatable.• Monitor PC system resources (CPU, Network Traffic).• Monitor communication packets from HMI to PLC.• Compare loads with and without antivirus software.• Inject test viruses from available access points.• Include testing during virus definition updates.

Antivirus/HMI Test Matrix

HMI-1 vs. AV-1 HMI-2 vs. AV-1

HMI-1 vs. AV-2 HMI-2 vs. AV-2

HMI-1 HMI-2

AV-1

AV-2

Current Status

• Antivirus application functionality mapping completed

• HMI-1 programmed for lower end operation• Performed preliminary testing between

HMI-1, AV-1 and AV-2 applications

Initial Testing

• Manual Scanning of Hard Drive• Manual Scanning of Floppy Drive• Active Scanning• AV1 Manual Scan of Hard Drive over different

CPU priority settings• Data packets collected over 1 minute period• Analyze single data variable packet – calculate

time between consecutive messages.• Baseline• Antivirus mode of operation/ no virus• Antivirus mode of operation/ virus present

Manual Scan of Hard Drive (HMI1/AV1&AV2)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.71 19 37 55 73 91 109

127

145

163

181

199

217

235

253

271

289

307

325

343

361

379

Message Count (1 minute of data collection)

Tim

e B

etw

ee

n C

on

se

cu

tiv

e M

es

sa

ge

s (

se

co

nd

s) Baseline - no scanning

AV1 scanning - no viruses

AV1 scanning - 3 viruses quarantined

AV2 scanning - no viruses

AV2 Scanning - 3 viruses quarantined

Start scan

End scan AV2 End scan

AV1

Directory size : 2.3Gb

Virus Files used: eicar.com eicar_com.zip eicarcom2.zip

More message delays due to AV2 result in fewer messages sent

Manual Scan: Hard Drive

Manual Scan Floppy Drive (HMI1/AV1&AV2)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.71 16 31 46 61 76 91 106

121

136

151

166

181

196

211

226

241

256

271

286

301

316

331

346

361

376

391

Message Count (1 minute of data collection)

Tim

e B

etw

een

Co

nse

cuti

ve M

essa

ges

(se

con

ds)

Baseline - no scanning

AV1 scanning - no viruses

AV1 scanning - 3 viruses quarantined

AV2 scanning - no viruses

AV2 scanning - 3 viruses quarantined

Start scan

Virus Files used: eicar.com eicar_com.zip eicarcom2.zip

Note: In all cases the floppy contained a 1Mb uninfected file

Manual Scan: Floppy Drive

Active Scanning Enabled (HMI1/AV1&AV2)

0

0.05

0.1

0.15

0.2

0.25

0.31 16 31 46 61 76 91 106

121

136

151

166

181

196

211

226

241

256

271

286

301

316

331

346

361

376

391

Message Count (1 minute of data collection)

Tim

e B

etw

een

Co

nse

cuti

ve M

essa

ges

(se

con

ds)

Baseline - no scanning

File copy - no scanning

File copy - AV1 scanning (1 virus quarantined)

File copy - AV2 scanning (1 virus quarantined)

Directory containing 1Mb file and the eicar.com file are copied to the hard drive

while active scanning is enabled.

Initiate File

Active Scanning

Manual Scanning of Hard Drive: CPU Priority Settings (HMI1/AV1)

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.51 18 35 52 69 86 103

120

137

154

171

188

205

222

239

256

273

290

307

324

341

358

375

Message Count (1 minute of data collection)

Tim

e B

etw

een

Co

nse

cuti

ve M

essa

ges

(se

con

ds)

Zero CPU Priority

20% CPU Priority

60% CPU Priority

100% CPU Priority

18.0

Large message delay results in fewer messages sent

Time between consecutive messagesexcedes 18 seconds

at higher priority settings

Directory size : 2.3Gb

Start scan

AV1 : CPU Priority Settings

Next Steps

• Program HMI-1 application at an upper end.• Program HMI-2 application at lower and upper

end.• Document a set of performance test methods

based on results of initial testing.• Perform testing across test methods.• Continue efforts using other security

applications such as personal firewalls and control applications such as software PLCs

Summary

• Introduction to the NIST Process Control Security Testbed.

• Development of performance methods to assess the effects of security software on the performance of PC based control software.

• Presented initial test results for effects of antivirus software on the performance of HMI software.

• Discussed future activities in this area.