Multi-Core Packet Scattering to Disentangle Performance Bottlenecks

Yehuda Afek Tel-Aviv University

Anat Bremler-Barr

David Hay Yotam Harchol Yaron Koral

Joint work with

This work was supported by European Research Council (ERC) Starting Grant no. 259085

Deep Packet Inspection

• IPS/IDS/FW Heaviest processing part: Search for malicious patterns in the payload

1. Pipeline multi-core, not efficient.

– Imbalance of pipeline stations, DPI much heavier

2. Parallel multi-core?

Multi-Core Deep Packet Inspection (DPI)

• Option 1: Each core a subset of patternsCore 1

Core 2

Core 3

Core 4

Pattern Set 1

Pattern Set 2Pattern Set 3

Pattern Set 4

Multi-Core Deep Packet Inspection (DPI)

• Option 1: Each core a subset of patternsCore 1

Core 2

Core 3

Core 4

Pattern Set 1

Pattern Set 2Pattern Set 3

Pattern Set 4

Multi-Core Deep Packet Inspection (DPI)

• Option 1: Each core a subset of patternsCore 1

Core 2

Core 3

Core 4

Pattern Set 1

Pattern Set 2Pattern Set 3

Pattern Set 4

Multi-Core Deep Packet Inspection (DPI)

• Option 1: Each core a subset of patterns

• Option 2: All cores are the same, Load-balance between cores

Core 1

Core 2

Core 3

Core 4

Pattern Set 1

Pattern Set 2Pattern Set 3

Pattern Set 4

Multi-Core Deep Packet Inspection (DPI)

• Option 2: All cores are the same, Load-balance between cores

Core 1

Core 2

Core 3

Core 4

DPI

DPIDPI

DPI

Multi-Core Deep Packet Inspection (DPI)

• Option 2: All cores are the same, Load-balance between cores

Core 1

Core 2

Core 3

Core 4

DPI

DPIDPI

DPI

Complexity DoS Attack Over NIDS• Easy to craft – very hard to process packets

• 2 Steps attack:

Attacker

Internet

2. Steal CC.

1. Kill IPS/FW

Attack on Security Elements

Combined Attack:DDoS on Security Element

exposed the network – theft of customers’

information

Attack on Snort

The most widely deployed IDS/IPS worldwide.

Heavy packets rate

OUR GOAL:A multi-core system

architecture, which is robust against complexity DDoS attacks

Airline Desk Example

Airline Desk Example

A flight ticket

20 min.

Airline Desk Example

An isle seat near window!!

Three carry

handbags!!!

Doesn’t like

food!!!

Can’t find passport!!

Overweight!!!

1 min.

Airline Desk Example

Airline Desk Example

4 min.1 min.

Domain Properties

1. Heavy & Light customers.

2. Easy detection of heavy customers.

3. Moving customers between queues is cheap.

4. Heavy customers have special more efficient processing method.

Special training

packets

packets

packets

packets

Some packets are much “heavier” than others

The Snort-attack experiment

Property 1 in Snort Attack

•DPI mechanism is a main bottleneck in Snort•Allows single step for each input symbol•Holds transition for each alphabet symbol

Snort uses Aho-Corasick DFAHeavy PacketFast & Huge

Best for normal trafficExposed to cache-miss attack

Crafting HEAVY packetsSnort patterns DatabaseMalicious pkts Factory

Chop last 2 bytes

Snort-Attack Experiment

Cache

Main Memory

Normal Traffic Attack Scenario

Cache-miss!!!Does not require many packets!!!

The General Case: Complexity Attacks

• Trivial to Craft --- Hard to process packetsDomain Properties

1. Heavy & Light packets.

2. Easy detection of heavy packets

3. Moving packets between queues is cheap.

4. Heavy packets have special more efficient processing method.

Property 2 in Snort Attack

Detecting heavy packets is feasible

How Do We Detect?

• May be quickly classified• Common states

• Claim: the general case in complexity attacks!!!

threshold

Percent non-common states

How Do We Detect?

Common States

NonCommon States

Heavy packet : # Not Common States # Common States ≤ α After at least

20 bytes

Domain Properties

1. Heavy & Light packets.

2. Easy detection of heavy packets

3. Moving packets between queues is cheap.

4. Heavy packets have special more efficient processing method.

System Architecture

P

rocessor

Ch

ip

Core #8N

IC Core #1Q

Core #2Q

Q

Q

Q

Detects heavy

packets

Core #9

Core #10

Routine Mode:

Load balance between cores

System Architecture

P

rocessor

Ch

ip

Core #8Dedicated Core

#9

NIC Core #1Q

Core #2Q

Q

QB

Dedicated Core #10

BQ

Detects heavy

packets

Alert Mode:Dedicated cores for heavy packets

Others detect and move heavy to Dedicated.

B

B

Inter-Thread Communication• Non-blocking IN-queues

– Only one thread accesses

• Dedicated queues blocking (using test&set locks)

– Non-dedicated threads “steal” packets from the HoL when sending a heavy packet

P

rocessor

Ch

ip

Core #8Dedicated Core

#9

NIC Core #1Q

Core #2Q

Q

QB

Dedicated Core #10

BQ

B

B

Domain Properties

1. Heavy & Light packets.

2. Easy detection of heavy packets

3. Moving packets between queues is cheap.

4. Heavy packets have special more efficient processing method.

Snort uses Aho-Corasick DFA

Full Matrix vs. Compressed

Heavy packets rate

Domain Properties

1. Heavy & Light packets.

2. Easy detection of heavy packets

3. Moving packets between queues is cheap.

4. Heavy packets have special more efficient processing method.

Experimental Results

System Throughput Over Time

Reaction time can be smaller

Different Algorithms Goodput

Additional Application for MCA2

The Hybrid-FA-attack experiment

Hybrid-FA

• Space-efficient data structure for regular expression matching

• Faster than NFA• Structure:

– Head DFA– Border states– Tail DFAs

• More than one state can be activeat the same time!

s0

s7

s12

s1 s2

s3 s5s4

C

C

E

D

B

E D

s14

s13 s6

D

s8

B

s9

Cs10

As11

B

A

A

.*

[^\n]*

Hybrid-FA Attack

Normal Traffic Attack Scenario

Again: Does not require many packets!!!

s0

s7

s12

s1 s2

s3 s5s4

C

C

E

D

B

E D

s14

s13 s6

D

s8

B

s9

C

s10

A

s11

B

A

A

.*

[^\n]*

s0

s7

s8

s9

s10

s11

s12

s2

s5

s13

Input: C D B B C AB

Heavy Packet Detection

threshold

MCA2 With Hybrid-FA

Concluding Remarks• A multi-core system architecture

• Robustness against complexity DDoS attacks

• In this talk we focused on specific NIDS and

complexity attack

– MCA2 can handle more NIDS complexity attacks, like the

Bro Lazy-FA

• We believe this approach can be generalized

(outside the scope of NIDS)

Thank You!!

Deep packet inspection