TM

November 2013

TM 2

Intro: DPAA and TM principles & terminology

QorIQ Ingress/Egress TM features overall description &

interaction

Policing

Queue Management (FQ and CG tail drop, RED/WRED,

CSCN)

Flow Control

T-serie Scheduling and Shaping (CEETM)

Note: features highlighted for T-series also apply to DPAA-based QorIQ Qonverge B4

3 TM

Eth

FMan classifies and distributes flows

BMan provides Buffer Pools services

QMan provides multi-producer/consumer queues services

1G/10G

Network Interfaces

Qman

PCD (Parse/Classif/Distrib)

BMan

FMan

DMA

1

4

5

2 3

Ingress

(Frame Queues) FQs

System

Memory

(Virtual Storage Profiles

= set of pools)) 4

Core Core Core

Portal Portal Portal Portal Portal Portal

4 TM

Core Core Core

Eth

1G/10G

Network Interfaces

QMan

PCD

BMan

FMan

5

1

4

3 (Frame Queues) FQs

(Storage Profiles

= set of pools))

System

Memory

2

Portal Portal Portal

Internal RAM

DMA

Load from Buffer

Enqueue Descriptor

Dequeue Descriptor

Transmit packet

Release

Buffer

Egress

5 TM

Control traffic against network usage policies/rules

Manage congestion situations

Provide QoS

Parse/classify Buffering Policing Queue Mgmt Rx

Input queues

Flow control

(Back-pressure)

Ingress

Dequeue Scheduling

&

Shaping

Tx

Output queues

Egress

TM purposes

Flow control

(Back-pressure)

Tx

Rx

Input Port

Output Port

Controlled by

6 TM

AF: Assured Forwarding

BE: Best Effort

CBS: Committed Burst Size

CEETM: Customer Edge Egress Traffic Management

CG: Congestion Group

CIR: Committed Information Rate

CQ: Class Queue

CQCG: CQ Congestion Group

CR: Committed Rate

CS : Congestion State

CSCN: Congestion State Change Notification

ER: Excess Rate

ERN: Enqueue Rejection Notification

FQ: Frame Queue

LFQID: Logical Frame Queue ID

PBS: Peak Burst Size

PIR: Peak Information Rate

PFC: Priority-based Flow Control

QoS: Quality of Service

RED: Random Early Discard

WRED: Weighted RED

SLA: Service Level Agreement

SP: Strict Priority

TD: Tail Drop

TM: Traffic Management

WQ/WFQ: Weighted Queueing/Weighted Fair Queueing

TM 7

Intro: DPAA and TM principles & terminology

QorIQ Ingress/Egress TM features overall description &

interaction

Policing

Queue Management (FQ and CG tail drop, RED/WRED,

CSCN)

Flow Control

T-serie Scheduling and Shaping (CEETM)

Note: features highlighted for T-series also apply to DPAA-based QorIQ Qonverge B4

8 TM

Core/SW

QMan

SW Portal

Queue Congestion

1) management

2) avoidance

3) notification

(based on FQs & Congestion groups)

DC Portal

Policer

PCD

Internal FIFO

BMI

Tx Pause

Control frames

MAC

BMan

Pool depleted

FIFO threshold

Core/SW

SW Portal

FQ ID + R/G/Y

Congestion state change

notification - CSCN

Error Notify

If core is the producer

If FMan is the producer (option 2)

FMan Discard & dealloc.

Discard & dealloc.

Discard &

dealloc.

Discard

(no ext. Buffer)

Discard

(no int.. Buffer)

QMan status/Interrupts

(to Core/SW)

BMan status/Interrupts

(to Core/SW)

Tail Drop/

WRED Reject

ERN Queue

Flow Control

1

2

3

FQs

Enqueue Reject Notification

4

10 TM

Core/SW

QMan

SW Portal

Rx Pause frames

MAC

Core/SW

SW Portal

Egress Scheduling/Shaping optionnaly

Implemented at Core/SW

FMan

Tx Rate Limiter

Flow Control

Dedic-Ch

DC-Portal

T-series only CEETM Egress Enhanced

hierarchical scheduling & shaping

DMA and Tx Frames

One Sub-portal per port

MAC Tx Rate Limiter

Flow Control

QMan regular channel

basic scheduling

(4x SP levels, RR, no

shaping)

T-series only Support Priority Flow Control

(PFC, IEEE 802.1Qbb)

TM 12

Intro: DPAA and TM principles & terminology

QorIQ Ingress/Egress TM features overall description &

interaction

Policing

Queue Management (FQ and CG tail drop, RED/WRED,

CSCN)

Flow Control

T-serie Scheduling and Shaping (CEETM)

Note: features highlighted for T-series also apply to DPAA-based QorIQ Qonverge B4

13 TM

Burst

Size

Reference Rate increment (Bytes/sec) (up to BS)

?

Packets in

Tokens

Case of Policing Check ingress traffic w.r.t traffic profile

Case of Shaping Transmit egress traffic w.r.t traffic profile

Packets out

Traffic profile

14 TM

Burst

Size

Reference Rate increment (Bytes/sec) (up to BS)

?

Packets in

Tokens

Packets out

Input queues

Ingress

Input Port Discard / Drop

Traffic profile

Eligeability to enqueue

& coloring controlled by

profile

15 TM

Implements RFC2698, RFC4115

Three Color output "GREEN", "YELLOW", & "RED"

Four parameter policing

CIR - Committed Information Rate

PIR - Peak Information Rate

CBS - Committed Burst Size

PBS - Peak Burst Size

Color aware or color blind policing

Color passed to QMan on enqueue for further WRED congestion mgmt, passed to Core/SW through Frame Descriptor

Policing can be based on packet count or byte count

Quick drop mode on RED colored packets

Memory internal to policer holds 256 profiles

Statistics per profile

Packet colouring can be a combination of FMan classification and/or traffic characteristics

Po

lice

r #n

(0-2

55)

Packet pass thru

Policer ID#

resulting from

FMan classification

To Qman

Traffic profile = CIR, PIR, CBS, PBS

Principle:

- Dual token bucket meters a packet

stream according to two rates, a Peak

Rate and a Committed Rate and their

associated Burst size.

- mark as Red if exceeds PIR, otherwise

mark as Yellow or Green depending on

whether it exceeds or not CIR.

16 TM

P_Bucket

PBS

Tp

Incremented by

PIR bytes/sec

(up to PBS)

? Green

C_Bucket

CBS Tc

Incremented by

CIR bytes/sec

(up to CBS)

?

Yellow

Red

For each packet coming in (B bytes is size of packet) :

if B Tp(t) > 0 then excess PIR, Packet marked as Red (then normally dropped)

Tp & Tc unchanged

else if B - Tc(t) > 0 then excess CIR, Packet marked as Yellow (and enqueued)

Tp Tp - B

else then Conformed traffic, Packet is marked as Green (and enqueued)

Tp Tp B

Tc Tc - B

Packets in

PIR: Peak Information Rate

CIR; Committed Information Rate

PBS: Peak Burst Size

CBS: Committed Burst Size

To next

step

queueing

TM 17

Intro: DPAA and TM principles & terminology

QorIQ Ingress/Egress TM features overall description &

interaction

Policing

Queue Management (FQ and CG tail drop, RED/WRED,

CSCN)

Flow Control

T-serie Scheduling and Shaping (CEETM)

Note: features highlighted for T-series also apply to DPAA-based QorIQ Qonverge B4

19 TM

On FQ Tail Drop:

The threshold in bytes at which tail drop from this FQ occurs is TD_MANT * 2TD_EXP

configured in FQD[TD_MANT, TD_EXP]

Tail drop enabled if FQD.FQ_CTRL[Tail_Drop_enable] bit set

There is no hysteresis on tail drop congestion management,

if the byte count of the FQ has exceeded the tail drop threshold and Tail Drop is enabled

then all subsequent enqueues are rejected

On Congestion Group Tail Drop (or CS TD):

Same principle as FQ TD but applies to CG aggregated queues content

Supports both drop and/or notification

CS tail drop enabled if CG[CSTD_EN] bit set, notification enabled if CG[CSCN_EN] bit set

CS can be based on Byte or Frame count

While it is possible to have both WRED and CSTD_EN enabled at the same time, it is

unlikely that this would be useful since the WRED algorithm effectively implements a tail

drop above MaxTH. CSTD_EN is more likely to be used in situations where WRED would

be considered as overkill, for management flows or non-traffic flows for example.

20 TM

RED (Random Early Discard) primarily for managing congestion in a way

appropriate to TCP end-to-end flow control

Current

average

length Trash

minTH maxTH

100% (minTH) 0% Dis

ca

rd p

rob

ab

ilit

y

0

100%

maxTH

slope maxP

100% 0% Dis

card

pro

bab

ilit

y

0

100%

(simple) RED W(eighted)RED

Apply different threshold+slope, depending on

how each packet is pre-marked /coloured thru

classification or policing

21 TM

RED (Random Early Discard) primarily for managing congestion in a way

appropriate to TCP end-to-end flow control

TCP Global Synchronization Issue

t

TC

P tra

ffic

Without Congestion Avoidance

t

TC

P tra

ffic

With Congestion Avoidance (RED)

multiple flows are throttling back followed by a

sustained period of lowered link utilization, thus

causing queue oscillation

packets are partially dropped before the queues

fill up, this gives flows such as TCP connections

the opportunity to slow down the sending rate

before the queues get full.

TM 22

Intro: DPAA and TM principles & terminology

QorIQ Ingress/Egress TM features overall description &

interaction

Policing

Queue Management (FQ and CG tail drop, RED/WRED,

CSCN)

Flow Control

T-serie Scheduling and Shaping (CEETM)

Note: features highlighted for T-series also apply to DPAA-based QorIQ Qonverge B4

23 TM

Flow control

Core/SW

QMan

SW Portal

Queue Congestion Mgmt

(based on FQ & Congestion group)

DC Portal

Policer

PCD

Internal FIFO

BMI

Tx Pause

Control frames

MAC

BMan

Pool depleted

FIFO threshold

Core/SW

SW Portal

FQ ID + R/G/Y

Congestion

notification

Error Notify

FMan Discard & dealloc.

Discard & dealloc.

Discard &

dealloc.

Discard

(no ext. Buffer)

Discard

(no int.. Buffer)

QMan status/Interrupts

(to Core/SW)

BMan status/Interrupts

(to Core/SW)

Tail Drop/

WRED Reject

ERN Queue

Flow Control

1

2

3

FQs

Enqueue Reject Notification

4

25 TM

802.1Qbb Priority based Flow Control (PFC)

Ingress

In T-series FMan & QMan implementation, the pause frame complies with priority based flow

control 802.1Qbb, and contains an 8 bit class enable vector. Each congestion group has a class

enable vector associated with it. This vector is programmed in the congestion groups priority

mapping table

Egress

In T-series FMan & QMan-CEETM implementation, each of the 16 class queues within a class

queue channel can be mapped to one of the 8 traffic classes

An incoming PFC frame can specify which traffic class to pause for each priority signaled in. The

QMan traffic class corresponding to the bit in the class enable vector of the pause frame is not

scheduled to transmit packets as long as the flow control condition is valid. One or more priorities

can be assigned to a Traffic Class (TC).

802.1Qaz Enhanced Transmission Selection (ETS)

Weighted bandwidth fairness is supported

Meets standard performance requirement for ETS: bandwidth granularity of 1% and +/-10%

accuracy

TM 26

Intro: DPAA and TM principles & terminology

QorIQ Ingress/Egress TM features overall description &

interaction

Policing

Queue Management (FQ and CG tail drop, RED/WRED,

CSCN)

Flow Control

T-serie Scheduling and Shaping (CEETM)

27 TM

Core/SW

QMan

SW Portal

Rx Pause frames

MAC

Core/SW

SW Portal

Egress Scheduling/Shaping optionnaly

Implemented at Core/SW

FMan

Tx Rate Limiter

Flow Control

Dedic-Ch

DC-Portal

T-series only CEETM Egress Enhanced

hierarchical scheduling & shaping

BMI DMA and Tx Frames

One Sub-portal per port

MAC Tx Rate Limiter

Flow Control

QMan regular

channel

basic

scheduling

(4x SP levels,

RR, no shaping)

28 TM

Class scheduler

SP

WQ

Dual-rate

Shaping

Class scheduler

SP

WQ

W

Q

Dual-rate

Shaping

SP

SP

WQ

Dual-rate

Shaping

Channel scheduler

WQ

Class scheduler

Egress Interface

Single-rate

Shaping

Channel 0

Channel 1

Channel n

SP: Strict Priority

WQ: Weighted Queuing

QMan egress queuing Hierarchical (2-level)

scheduling and dual rate shaping for Ethernet

interfaces

29 TM

Egress

Port 1

Dual-rate

Shaping

Ethernet

Interface

Dual-rate

Shaping

Ethernet

Interface

Egress

Port M

Scheduling/

shaping

Scheduling/

shaping FMan

FMan

VoIP

RT

AF

+

BE

30 TM

31 TM

Same queue interface and semantics as the standard QMan scheme. Producer enqueues to a Logical FQ ID (LFQID) mapped to a CEETM CQ

32 Channels available for allocation across the 8 Ethernet interfaces within a Frame Manager (10G and 1G)

Configurable combinations of strict priority and fair scheduling (weighted queuing) between the queues

Dual-rate shaping (paired CR and ER shapers) at all shaping points to meet service-level agreements (SLAs) parameters (1 Kbps...10 Gbps range, 1 Kbps granularity across the entire range) .

Algorithms for shaping and fair scheduling are based on bytes.

No intermediate buffering between Class Queues and the FMan Tx hardware

Performed in an atomic manner; all context at all levels is examined and updated synchronously.

Dynamic re-configuration for lossless interface failover

service-level agreements (SLAs) parameters (1 Kbps...10 Gbps range, 1 Kbps granularity across the entire range)

32 TM

Class scheduler

SP

WQ

Dual-rate

Shaper

Class scheduler

SP

WQ

W

Q

Dual-rate

Shaper

SP

SP

ShF

Q

Dual-rate

Shaper

Channel scheduler

for LNI #n

uF

Q

Class scheduler

Egress Interface

Single-rate

Shaping

Channel 0

Channel 1

Channel #n

32 channels (Class-Queue

Scheduler) per CEETM available for

allocation across the 8x Ethernet

interfaces within a Frame Manager

One CEETM per FMan

16 class queues (CQs) per channel

8 Strict priority

8 (4+4) fair weighted

4K LFQID mapped to 512 CQs ( T4/T2)

Note: T104x has 8 channels,

128 CQs and 1K LFQIDs

x32

8 Channel Scheduler/Shapers per

CEETM available for allocation

across the 8x Ethernet interfaces

within a Frame Manager

One CEETM per FMan

Eth Port

x8

( T4/T2)

33 TM

Class scheduler

SP

WQ

Dual-rate

Shaper

Class scheduler

SP

WQ

W

Q

Dual-rate

Shaper

SP

SP

ShF

Q

Dual-rate

Shaper

Channel scheduler

for LNI #n

uF

Q

Class scheduler

Egress Interface

Single-rate

Shaping

Channel 0

Channel 1

Channel #n

10-level Strict Priority among single Class Queues and Groups

WBFS (Weighted Bandwidth Fair

Scheduling) at lower sub-level among

group of 4 or 8 Class Queues

x32 Eth Port

Scheduling decisions triggered by transmit opportunities

Unshaped Aware Fair

Scheduling for equalized

traffic among unshaped

channels

Shaped Aware Fair

Scheduling propr. Algorithm to

for ensuring channels achieve

their shape to the highest

degree possible.

34 TM

Byte aware dual rate shaping

Shapers are token bucket based with configurable rate and

burst limit

Priority schedules CR packets over ER packets

Paired CR/ER shapers may be configured as independent

or coupled on a per pair basis; coupled means that credits to

the CR shaper in excess of its token bucket limit is credited

to the ER bucket

Class scheduler

SP

WQ

Class scheduler

SP

WQ

W

Q

SP

SP

SF

Q

ER

Channel scheduler

uF

Q

Class scheduler

Egress Interface

Single-rate

Shaping

Dual-rate

Shaping

CR

ER

CR

ER

CR

Dual-rate

Shaping

Dual-rate

Shaping

The color green denotes logic units and

signal paths that relate to the request and

fulfillment of committed rate (CR) packet Tx

opportunities.

The color yellow denotes the same for

excess rate (ER).

Channel 0

Channel 1

Channel n

Per Eth port global shaper / Rate-limiter

37 TM

16 class queues per channel

8 Strict priority

8 (4+4) fair weighted

T4240

Number of channels per Frame Manager instance (e.g. total of 2 instances in T4240)

32 channels available for allocation across 8 Ethernet interfaces within a Frame Manager (10G and 1G)

T1040

Number of channels per Frame Manager instance (e.g. total of 1 instances in T1040)

8 channels available for allocation across 5 Ethernet interfaces within a Frame Manager

TM 38

Questions ?

TM