IPQ806x Hardware acceleration

Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada

Features Designed for Home Gateways (CPE) Flow detection based All-or-nothing

offload Acceleration supports:

IPv4, IPv6, NAT, PPPoE, L2TP, VLAN, Qdisc

Performance gain Linux: 640k pps (bridged) 220kpps (routed) NSS: 7200k pps (bridged) 7200kpps (routed) 11x (bridged) 32x (routed)

Functional behavior 0% cpu load seen in Linux Keep Linux counters up to date Does not require functional changes at an

upper level (user space)

NSS acceleration model

Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada

Inspect all outgoing packets at POSTROUTING chain by registering post routing hooks. Inspect conntrack and device events to destroy and regenerate connection. Inspect NSS status and stats to update the connection state and statistics info in Linux and ECM DB.

ECM Front End Inspect Packets and Events

FRONT ENDNETFILTERPF_BRIDGE

POST ROUTING

L2 PKTs

L2 PKTs

NETFILTERIPvX

POST ROUTING L3 PKTs

L3 PKTs

INTERFACENOTIFIER

MTU / UP/ DOWN /

BOND NOTIFIER

Bond Link/ Release/Enslave

NSS Driver

NSS Status and Stats Sync

NSS Firmware

LINUX CONNTRACK

IPCT_DESTROY/ IPCT_MARK

CONNTRACK NOTIFIER

Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada

Example of IPv4 TCP rule creation

NSS Firmware

Linux

TCP Data

The Accelerated TCP flow

TCP: ACKOr Syn+ACK

Transmit the packet

Add the rule & reply the establish cmd

Match Any Accelerated

Rule?

No

ECM

Send the Packet to Linux

CreateIPv4 Rule

Packet

Linux net stack

NetfiterPrerouting

Routing or Bridging

NetfiterPostrouting

NSS Driver

Connection Established

Check TCP Connection can be

accel?

Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada

Example of IPv4 TCP rule destroy

NSS Firmware

Linux Kernel

TCP Data

The Accelerated TCP flow

TCP: FIN Packet

Send the Packet to Linux

Transmit The packet

DestroyIPv4 Rule

IPv4 RuleSync

Add the rule & reply the establish cmdIs it a

connected TCP Data?

No

Linux net stack

NetfiterPrerouting

Routing / Bridging

NetfiterPostrouting

ECM Remove the connection

NSS Driver

ConntrackConntrack destroy event

ConnectionDestroy ?

Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada

Example of IPv4 rule API Common messaging interfaces

Protocol type: IPv4, IPv6, PPP, LAG

Rule type: CREATE, DESTROY, CONN_STAT_SYNC , NODE_STAT_SYNC

Callback/args pointers: will be passed back in the FW ACK/NACK reply

IPv4 rule create message structure example

struct nss_ipv4_rule_create_msg {/* Request */uint16_t valid_flags; /* Indicate which of the parameters below is filled-in

Indirectly says which operation to be done on the flow */uint16_t rule_flags; /* Bit flags associated with the rule */struct nss_ipv4_5tuple tuple; /* src_ip, dst_ip, src_port, dst_port, proto */struct nss_ipv4_connection_rule conn_rule; /* src_mac, dst_mac, src_iface, dst_iface, src_mtu, dst_mtu,

nat_src_ip, nat_dst_ip, nat_src_port, nat_dst_port */struct nss_ipv4_protocol_tcp_rule tcp_rule; /* TCP related accleration parameters */struct nss_ipv4_pppoe_rule pppoe_rule; /* flow_session_id, flow_remote_mac, ret_session_id, ret_remote_mac */struct nss_ipv4_qos_rule qos_rule; /* flow_qos_tag, ret_qos_tag, */struct nss_ipv4_dscp_rule dscp_rule; /* flow_dscp, ret_dstp */struct nss_ipv4_vlan_rule vlan_primary_rule; /* ingress_vlan_tag, egress_vlan_tag */struct nss_ipv4_vlan_rule vlan_secondary_rule; /* ingress_vlan_tag, egress_vlan_tag for QinQ */

/* Response */uint32_t index; /*Slot ID for cache stats to host OS */

};

Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada

Linux

ECM

Interfaces & Connections statistics update Stats updates sent

periodically from the Firmware Per-interfaces stats

(update net_devices)

Per-connections stats (update conntracks)

Minor modifications to ppp/l2tp/ipsec layers for iface look-up and stats update

NSS Driver

NODE_STATS_SYNCmessage

CONN_STATS_SYNCmessage

ppp0

ethN

tun0

br0

Look-up interface

NSS Firmware

Look-up conntrack

stats++

conntracksstats++

Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada

Qdisc acceleration

# tc qdisc add dev eth0 root handle 1: tbf rate 1000Mbit burst 100k limit 100# tc qdisc add dev eth0 parent 1: handle 10: prio bands 3# tc qdisc add dev eth0 parent 10:1 handle 100: tbf rate 2Mbit burst 10k limit 100# tc qdisc add dev eth0 parent 100: handle 1000: pfifo limit 100# tc qdisc add dev eth0 parent 10:2 handle 200: tbf rate 40Mbit burst 30k limit 100# tc qdisc add dev eth0 parent 200: handle 2000: red limit 100k min 30k max 80k \avpkt 1k burst 55 probability 0.20# tc qdisc add dev eth0 parent 10:3 handle 300: red limit 100k min 30k max 80k \avpkt 1k burst 55 probability 0.30

tbf

tbf tbf

r

e

d

prio

r

e

d

p

f

i

f

o

1:

10:

100: 200:

300:

1000: 2000:

10:3

10:210:1

nsstbf

nsstbl nsstbl

n

s

s

r

e

d

nssprio

n

s

s

r

e

d

n

s

s

p

f

i

f

o

1:

10:

100: 200:300:

1000: 2000:

10:310:210:1

# tc qdisc add dev eth0 root handle 1: nsstbl rate 1000Mbit burst 100k# tc qdisc add dev eth0 parent 1: handle 10: nssprio bands 3# tc qdisc add dev eth0 parent 10:1 handle 100: nsstbl rate 2Mbit burst 10k# tc qdisc add dev eth0 parent 100: handle 1000: nsspfifo limit 100# tc qdisc add dev eth0 parent 10:2 handle 200: nsstbl rate 40Mbit burst 30k# tc qdisc add dev eth0 parent 200: handle 2000: nssred limit 100k min 30k max 80k \avpkt 1k burst 55 probability 0.50# tc qdisc add dev eth0 parent 10:3 handle 300: nssred limit 100k min 30k max 80k \avpkt 1k burst 55 probability 0.50 set_default

Proceedings of netdev 0.1, Feb 14-17, 2015, Ottawa, On, Canada