+ All Categories
Home > Documents > Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

Date post: 02-Jun-2018
Category:
Upload: digonto-oni
View: 223 times
Download: 0 times
Share this document with a friend

of 59

Transcript
  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    1/59

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    2/59

    E-VPN & PBB-EVPN: the Next Generaof MPLS-based L2VPN

    [email protected]

    [email protected]

    [email protected]

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    3/59

    2014 Cisco and/or its affiliates. All rights reserved.

    Agenda

    Technical Overview

    Flows and Use Cases Ciscos PBB-EVPN Implementation

    Summary

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    4/59

    Technical OverviewHighlights and Solution Requirements

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    5/59

    2014 Cisco and/or its affiliates. All rights reserved.

    Data Center Interconnect requirements not fully addressed by curr

    L2VPN technologies

    DCI Brings New Requirements

    Ethernet Virtual Private Network (E-VPN) and Provider Backbone

    EVPN (PBB-EVPN)designed to address these requirements

    All-active Redundancy and Load

    Simplified Provisioning and Oper

    Optimal Forwarding

    Fast Convergence

    MAC Address Scalability

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    6/59 2014 Cisco and/or its affiliates. All rights reserved.

    Towards EVPN

    Existing VPLS solutions do not offer an

    All-Active per-flow redundancy Looping of Traffic Flooded from PE

    Duplicate Frames from Floods from theCore

    MAC Flip-Flopping over Pseudowire

    E.g. Port-Channel Load-Balancing does notproduce a consistent hash-value for a framewith the same source MAC (e.g. non MACbasedHash-Schemes)

    Solve Challenges of VPLS for All-Active Redundancy

    PE1

    PE2

    PCE1

    Echo !

    PE1

    PE2

    PCE1

    M1

    M1

    PE1

    PE2

    PE

    P

    CE1MAC

    Flip-Flop

    M1

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    7/59 2014 Cisco and/or its affiliates. All rights reserved.

    Solution Requirements

    All-Active Redundancy to maximize bisectional bandwidth

    Load-balance traffic among PEs and exploit core ECMP based on flow entbe L2/L3/L4 or combinations)

    Support geo-redundant PE nodes with optimal forwarding

    Flexible Redundancy Grouping of PEs

    All-Active Redundancy and Load Balancing

    WAN

    Site 1Site 2

    Site N

    Flow-based Load

    balancing

    Flow-based Multi-pathing

    Backdoor

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    8/59 2014 Cisco and/or its affiliates. All rights reserved.

    Solution Requirements

    Active / Active Multi-Homing withflow-based load balancing in CEto PE direction

    Maximize bisectional bandwidth

    Flows can be L2/L3/L4 or combinations

    Flow-based load balancing in PEto PE direction

    Multiple RIB entries associated for agiven MAC

    Exercises multiple links towards CE

    All-Active Redundancy and Load Balancing (cont.)

    P

    E

    P

    E

    Vlan X -

    F1

    Vlan X

    F2

    Flow Based Load-balancingCE to PE dire

    P

    E

    P

    E

    Flow Based Load-balancing

    PE to PE dire

    Vlan X -

    F1Vlan X

    F2

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    9/59 2014 Cisco and/or its affiliates. All rights reserved.

    Flow-based Multi-Pathing

    Load balancing acrossequal cost multiple pathsin the MPLS core

    Load balancing at PE andP routers based on

    Entropy MPLS labels

    Solution RequirementsAll-Active Redundancy and Load Balancing (cont.)

    PE

    PE

    P

    P

    P

    P

    Flow Based Multi-Pathing in the CoreVlan X - F1Vlan X

    F2Vlan X

    F3Vlan X

    F4

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    10/59 2014 Cisco and/or its affiliates. All rights reserved.

    Solution Requirements

    Server Virtualization fueling growth in MAC Address scalability: 1 VM = 1 MAC address.

    1 server = 10s or 100s of VMs

    MAC address scalability most pronounced on Data Center WAN Edge for Lextensions over WAN.

    Example from a live network: 1M MAC addresses in a single SP data center

    MAC Address Scalability

    WAN

    DC Site 1

    DC Site 2DC S

    1Ks

    10Ks

    1Ms

    N * 1M

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    11/59 2014 Cisco and/or its affiliates. All rights reserved.

    Next generation solution for Ethernet

    multipoint connectivity services Leverage similarities with L3VPN

    PEs run Multi-Protocol BGP toadvertise & learn MAC addresses overCore

    Learning on PE Access Circuits viadata-plane transparent learning

    No pseudowire full-mesh required Unicast: use MP2P tunnels

    Multicast: use ingress replication over MP2Ptunnels or use LSM

    Under standardization at IETFdraft-ietf-l2vpn-evpn

    Ethernet VPN

    Highlights

    MPLS

    PE1

    CE1

    PE2

    VID 100

    SMAC: M1

    DMAC: F.F.F

    BGP MAC adv

    E-VPN NLRI

    MAC M1 via PE

    Data-plane addresslearning from Access

    Control-plane addr

    advertisement / leaover Core

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    12/59 2014 Cisco and/or its affiliates. All rights reserved.

    Combines Ethernet Provider Backbone Bridging

    (PBB - IEEE 802.1ah) with Ethernet VPN PEs perform as PBB Backbone Edge Bridge (BEB)

    Reduces number of BGP MAC advertisementsroutes by aggregating Customer MACs (C-MAC) via Provider Backbone MAC (B-MAC)

    Addresses virtualized data centers with C-MAC countinto the millions

    PEs advertise local Backbone MAC (B-MAC)

    addresses in BGP C-MAC and C-MAC to B-MAC mapping learned in

    data-plane

    Under standardization at IETFdraft-ietf-l2vpn-pbb-evpn

    PBB Ethernet VPN

    Highlights

    MPLS

    PE1

    CE1

    PE2

    B-MAC:B-M1

    BGP MAC adv.

    E-VPN NLRI

    MAC B-M1 via P

    B-MAC:

    B-M1

    Control-plane

    advertisemen

    over Core (B-M

    Data-plane address

    learning from Access

    Local C-MAC to local B-

    MAC binding

    Data-plane address

    learning from Core

    Remote C-MAC to remoteB-MAC binding

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    13/59

    Technical OverviewConcepts

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    14/59 2014 Cisco and/or its affiliates. All rights reserved.

    E-VPN / PBB-EVPN Concepts

    Ethernet Segment

    Represents a site

    connected to one or more

    PEs Uniquely identified by a

    10-byte global Ethernet

    Segment Identifier (ESI)

    Could be a single device

    or an entire network

    Single-Homed Device (SHD)

    Multi-Homed Device (MHD)

    Single-Homed Network (SHN)

    Multi-Homed Network (MHN)

    BGP Routes

    E-VPN and PBB-EVPN

    define a single new BGP

    NLRI used to carry all E-VPN routes

    NLRI has a new SAFI (70)

    Routes serve control

    plane purposes,

    including:

    MAC address reachability

    MAC mass withdrawal

    Split-Horizon label adv.

    Aliasing

    Multicast endpoint discovery

    Redundancy group discovery

    Desi nated forwarder election

    E-VPN Instance (EVI)

    EVI identifies a VPN in the

    network

    Encompass one or morebridge-domains,

    depending on service

    interface type

    Port-based

    VLAN-based (shown above)

    VLAN-bundling

    VLAN aware bundling (NEW)

    New

    com

    Expcarr

    incl

    MAC

    C-M

    Red

    MAC

    Spli

    P

    E

    BD

    BD

    EVI

    EVI

    PE1

    PE2

    CE1

    C

    E2

    SHD

    MHD

    ESI1

    ESI2

    Route Types

    [1] Ethernet Auto-Discovery (AD) Route

    [2] MAC Advertisement Route

    [3] Inclusive Multicast Route

    [4] Ethernet Segment Route

    E

    ESI MP

    ES-Imp

    MAC M

    Default

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    15/59

    2014 Cisco and/or its affiliates. All rights reserved.

    E-VPN Instance (EVI) & Service Interfaces

    E-VPN Instance (EVI) identifies a VPN in the MPLS/IP network

    EVI may encompass one or more bridge-domains, depending o

    interface type:

    C

    E

    CE

    U

    NI

    U

    NI

    U

    NI

    Port Based Service

    InterfaceAll CE-

    VLANs

    VPN A

    PE

    BDEVI

    C

    E

    C

    E

    U

    NI

    U

    NI

    U

    NI

    VLAN Based Service

    Interface

    VLA

    N X

    VLAN

    Y

    VPN

    A VPN

    B

    P

    E

    BD

    BD

    EVI

    EVI

    C

    E

    C

    E

    U

    NI

    U

    NI

    U

    NI

    VLAN Bundling Service

    InterfaceCE-

    VLAN

    subsetVPN

    A

    PE

    BDEVI

    U

    NI

    VLAN A

    InterfacCE-

    VLAN

    subset

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    16/59

    2014 Cisco and/or its affiliates. All rights reserved.

    Ethernet SegmentDefinition

    Ethernet Segment is a site connected to one or more PEs

    Ethernet Segment could be a single device(i.e. CE) or an entire network Single-Homed Device (SHD)

    Multi-Homed Device*(MHD) using Ethernet Multi-chassis Link Aggregation Group

    Single-Homed Network (SHN)

    Multi-Homed Network*(MHN)

    Uniquely identified by a 10-byte global Ethernet Segment Identifier (ESI)

    PE1

    PE2

    PE4

    PE3

    CE1

    CE2

    CE6CE3

    CE4

    CE5

    SHD

    MHDMHN

    SHNMHD

    ESI1

    ESI2

    ESI3

    E

    PE5

    (*) Includes Dual-Home

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    17/59

    2014 Cisco and/or its affiliates. All rights reserved.

    Ethernet Segment

    ESI Auto-Sensing

    MHD with Multi-chassis LAG

    ESI is auto-discovered via LACP

    ESI is encoded using the CEs LACPparameters:

    PE1

    PE2

    CE

    LACPDU

    LACPDU

    MPLS

    MHN with

    ESI is auto-discove

    snooping

    ESI is encoded usin

    parameters:

    PMST

    CE1

    CE2

    BPDU

    BPDU

    System

    Priority

    2 bytes 6 bytes 2 bytes

    System MAC

    AddressPort Key Bridge Priority

    2 bytes 6 bytes

    Root Bridge M

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    18/59

    2014 Cisco and/or its affiliates. All rights reserved.

    Split Horizon

    For Ethernet SegmentsE-VPN

    PE advertises in BGP a split-horizon label (ESI MPLS Label) associated weach multi-homed Ethernet Segment

    Split-horizon label is only used for multi-destination frames (UnknownUnicast, Multicast & Broadcast)

    When an ingress PE floods multi-destination traffic, it encodes the Split-Horizon label identifying the source Ethernet Segment in the packet

    Egress PEs use this label to perform selective split-horizon filtering over thattachment circuit

    PE1

    PE2

    CE1

    ESI-1

    CE4

    Challenge:

    How to prevent flooded traffic from echoing

    back to a multi-homed Ethernet Segment?Echo !

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    19/59

    2014 Cisco and/or its affiliates. All rights reserved.

    Split Horizon

    For Ethernet SegmentsPBB-EVPN

    PEs connected to the same MHD use the same B-MAC address for theEthernet Segment

    1:1 mapping between B-MAC and ESI (for All-Active Redundancy with flow-based

    Disposition PEs check the B-MAC source address for Split-Horizon filterin

    Frame not allowed to egress on an Ethernet Segment whose B-MAC matches theMAC source address in the PBB header

    PE1

    PE2

    CE1

    ESI-1

    CE4

    Challenge:

    How to prevent flooded traffic from echoing

    back to a multi-homed Ethernet Segment?Echo !

    B-MAC1

    B-MAC1

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    20/59

    2014 Cisco and/or its affiliates. All rights reserved.

    Split Horizon

    For Core Tunnels

    Traffic received from an MPLS tunnel over the core is never forwa

    back to the MPLS core This is similar to the VPLS split-horizon filtering rule

    PE1

    PE2

    CE1

    ESI-1

    CE4

    Challenge:

    How to prevent flooded traffic from looping back

    over the core?Loop !

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    21/59

    2014 Cisco and/or its affiliates. All rights reserved.

    Designated Forwarder (DF)

    DF Election

    PEs connected to a multi-homed Ethernet Segment discover each othBGP

    These PEs then elect among them a Designated Forwarder responsibforwarding flooded multi-destination frames to the multi-homed Segm

    DF Election granularity can be:

    Multiple DFs for load-sharing

    Per Ethernet Tag on Ethernet Segment (E-VPN)

    Per I-SID on Ethernet Segment (PBB-EVPN)

    PE1

    PE2

    CE1

    ESI-1Challenge:

    How to prevent duplicate copies of flooded

    traffic from being delivered to a multi-homed

    Ethernet Segment?

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    22/59

    2014 Cisco and/or its affiliates. All rights reserved.

    Designated Forwarder (DF)

    DF FilteringMHD All-Active with Per-Flow Load Balancing

    PE1

    PE2

    CE

    MPLS

    MHD / MHN All-Active with

    Load Balancin

    PE1

    PE2 MMHNCE1

    CE2

    PE1

    PE2

    CE

    MPLS

    Filtering

    Direction:

    Core to Segment

    Filtered Traffic: Flooded multi-destination

    Filtering

    Direction:

    Core t

    Segm

    Filtered Traffic: Flood

    destin

    Unica

    !DF Filtering

    DF Filtering

    !

    DF Filtering

    !

    Multi-destination

    Traffic

    Unicast Traffic

    Legend

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    23/59

    2014 Cisco and/or its affiliates. All rights reserved.

    Aliasing

    E-VPN

    PEs advertise in BGP the ESIs of local multi-homed Ethernet Seg

    All-Active Redundancy Mode indicated

    When PE learns MAC address on its AC, it advertises the MAC inalong with the ESI of the Ethernet Segment from which the MAC wlearnt

    Remote PEs can load-balance traffic to a given MAC address acrPEs advertising the same ESI

    MAC1

    PE1

    PE2

    CE1

    ESI-1

    Challenge:

    How to load-balance traffic towards a multi-

    homed device across multiple PEs when MAC

    addresses are learnt by only a single PE?

    I can

    reach

    ESI1(All-Active)

    I can

    reach

    ESI1(All-Active)

    MAC1

    I can

    MAC1 v

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    24/59

    2014 Cisco and/or its affiliates. All rights reserved.

    Aliasing

    PBB-EVPN

    PEs connected to the same MHD use the same B-MAC address for the ESegment

    1:1 mapping between B-MAC and ESI (for All-Active Redundancy with flow-based

    PEs advertise their B-MAC addresses independent of the C-MAC learning

    Remote PEs can load-balance traffic to a given C-MAC across all PEs advthe same associated B-MAC

    MAC1

    PE1

    PE2

    CE1

    ESI-1

    Challenge:

    How to load-balance traffic towards a multi-

    homed device across multiple PEs when MAC

    addresses are learnt by only a single PE?

    I can

    reach B-

    MAC1

    I can

    reach B-

    MAC1

    MAC1

    I can

    MAC1

    MA

    B-M1

    B-M1

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    25/59

    2014 Cisco and/or its affiliates. All rights reserved.

    MAC Mass-Withdraw

    PEs advertise two sets of information:

    MAC addresses along with the ESI from the address was learnt

    Connectivity to ESI(s)

    If a PE detects a failure impacting an Ethernet Segment, it withdrawfor the associated ESI

    Remote PEs remove failed PE from the path-list for all MAC addresses with an ESI

    This effectively is a MAC mass-withdraw function

    E-VPN

    MAC1

    PE1

    PE2

    CE1

    ESI-1

    Challenge:

    How to inform remote PEs of a failure affecting

    many MAC addresses quickly while the control-

    plane re-converges?

    MAC1,

    MAC2,

    MACn

    I can

    reach

    ESI1(All-Active)

    I can reach

    MAC1 via ESI1

    I can reach

    MAC2 via ESI1

    I can reach

    MACn via ESI1

    I can

    reach

    ESI1(All-Active)

    I lost ESI1

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    26/59

    Flows and Use CasesPBB-EVPN Startup Sequences

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    27/59

    2014 Cisco and/or its affiliates. All rights reserved.

    PBB-EVPN Startup Sequence

    Segment Auto-Discovery

    ESI and B-MAC Auto-Sensing

    Redundancy Group Membership

    Auto-Discovery

    VPN Auto-Disc

    Multicast Tun

    Dis

    Backbone MAC (B-MAC)Reachability Advertisement

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    28/59

    2014 Cisco and/or its affiliates. All rights reserved.

    PBB-EVPN Startup Sequence (cont.)

    ESI and B-MAC Auto-Sensing

    Segment Auto-Discovery

    ESI and B-MAC Auto-Sensing

    MPLS

    PE1

    CE1

    PE2

    LACP PDU

    exchange

    Source B-MAC used at PBB-EVPN PE on a

    given ESI can be auto-generated* from CEs

    LACP information -> CEs LACP System ID

    MAC with U/L** (Universal / Locally

    Administered) bit flipped

    Example: 0211.0022.0033

    CE LACP info:

    LACP System ID (MAC) (6B)

    e.g. 0011.0022.0033

    LACP System Priority (2B)e.g. 0000

    LACP Port Key (2B)

    e.g. 0018

    ESI (10B) can be auto-generated*

    from CEs LACP information ->

    concatenation of CEs LACP

    System Priority + Sys ID + Port Key

    Example:0000. 0011.0022.0033.0018

    (*) ESI and B-MAC can als

    (**) U/L is second-least-sig

    System

    Priority

    2 bytes 6 by

    Sys

    A

    B-MAC

    B-MAC

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    29/59

    2014 Cisco and/or its affiliates. All rights reserved.

    PBB-EVPN Startup Sequence (cont.)

    BGP Ethernet Segment Route

    MPLS

    PE1

    CE1

    PE2

    PE 1 Eth Seg

    RD = R

    ESI = E

    ES-Import e

    e.g. 0011.0

    MAC address portionof ESI (6B)

    PE 2 Eth S

    RD

    ES

    ES-Impor

    e.g. 001

    Segment Auto-Discovery

    ESI and B-MAC Auto-Sensing

    Redundancy Group MembershipAuto-Discovery

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    30/59

    2014 Cisco and/or its affiliates. All rights reserved.

    PBB-EVPN Startup Sequence

    Designated Forwarder (DF) Election*

    MPLS

    PE1

    CE1

    PE2

    Segment Auto-Discovery

    ESI and B-MAC Auto-Sensing

    Redundancy Group MembershipAuto-Discovery

    Modulo Operation

    I-SID

    I-SID mod N

    (N = # of PEs)

    (e.g. I-SID mod 2)

    100 0

    101 1

    102 0

    103 1

    PE Ordered List

    Position PE

    0 PE1

    1 PE2

    Modulo Operation

    I-SID (I-SID mod 2)

    100 0

    101 1

    102 0

    103 1

    Excha

    Segm

    Result of modulo

    operation is used todetermine DF and

    BDF status

    DFD

    BDF

    I-SID

    Example:

    PE2 DF for I-SIDs 101, 103

    PE2 BDF for I-SIDs 100, 102

    (*) DF election with Service Carving shown (i.e. one DF per I-SID in the segment)

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    31/59

    2014 Cisco and/or its affiliates. All rights reserved.

    PBB-EVPN Startup Sequence (cont.)

    BGP MAC Advertisement Route (B-MAC)

    Segment Auto-Discovery

    ESI and B-MAC Auto-Sensing

    Redundancy Group MembershipAuto-Discovery

    Backbone MAC (B-MAC)Reachability Advertisement

    MPLS

    PE1

    CE1

    PE2

    PE1 MAC Rou

    RD = RD-1a

    ESI = all 1s

    MAC = B-M1

    Label = L1

    RT ext. commu

    RT-a

    MP2P VPN Label

    downstream allocated labelused by other PEs to send

    traffic to advertised (MAC,EVI)

    PE3 / P

    VPN M

    RT-a B

    PE2 MAC Route

    RD = RD-2a

    ESI = all 1s

    MAC = B-M1

    Label = L2

    RT ext. community

    RT-a

    B-M1

    B-M1

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    32/59

    2014 Cisco and/or its affiliates. All rights reserved.

    PBB-EVPN Startup Sequence

    BGP Inclusive Multicast Route

    VPN Auto-Discovery

    Multicast Tunnel ID / Endpoint

    Discovery1

    PE 1 Inclusive

    RD =

    PMSI Tun

    Tunnel Type

    Label

    RT ext. c

    R

    MPLS

    PE1

    CE1

    PE2

    Tunnel TypeIngress

    Replication or P2MP LSP

    Mcast MPLS Labelused to

    transmit BUM traffic -

    downstream assigned (ing.repl.) or upstream assigned

    (Aggregate Inclusive P2MP

    LSP2)

    PMSI - P-Multic

    BUMBroadca

    RTRT associated with a

    given EVI

    PE 2 Inclusive Multicast Route

    RD = RD-2a

    PMSI Tunnel Attribute

    Tunnel Type (e.g. Ing. Repl.)

    Label (e.g. L2)

    RT ext. community

    RT-a

    (1) Inclusive Multicast Route advertized per I-SID

    (2) Multicast MPLS label is not set for Inclusive Trees (P2MP LSP)

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    33/59

    Flows and Use CasesPBB-EVPN Life of a Packet

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    34/59

    2014 Cisco and/or its affiliates. All rights reserved.

    MPLS

    PE1

    CE1

    PE2

    PE3

    CE3

    PE4

    PBB-EVPN Life of a PacketIngress ReplicationMulti-destination Traffic Forwarding

    MPLS

    PE1

    CE1

    PE2

    VID 100

    SMAC: M1

    DMAC: F.F.F

    M

    L

    P

    B

    g

    PSN MPLS labelto reach PE3

    PE2drops BUM

    traffic originated

    on same source

    B-MAC (B-M1)

    PE1 receives broadcast

    traffic from CE1. PE1

    adds PBB encapsulation

    and forwards it using

    ingress replication3copies created

    PE 2 Inclusive Multicast

    Route

    RD = RD-2a

    PMSI Tunnel Attribute

    Tunnel Type = Ing. Repl.

    Label = L2

    RT ext. community

    RT-a

    Mcast MPLS Labelused to

    transmit BUM traffic -

    downstream assigned (for

    ingress replication)

    During start-up sequence,

    PE1, PE2, PE3, PE4 sentInclusive Multicast route

    which include Mcast label

    B-M1

    B-M1

    B-M2

    B-M2

    B-M1

    B-M1

    L2 PBB

    L3 PBB

    L4 PBB

    PE3 MAC

    I-SID xy

    C-MAC B

    M1 B

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    35/59

    2014 Cisco and/or its affiliates. All rights reserved.

    MPLS

    PE1

    CE1

    PE2

    PE3

    CE3

    PE4

    PBB-EVPN Life of a Packet (cont.)Unicast Traffic Forwarding and Aliasing

    PE1 MAC Route

    RD = RD-1a

    ESI = all 1s

    MAC = B-M1

    Label = L1

    RT ext. community

    RT-a

    PE3 RIB

    VPN MAC ESI

    RT-a B-M1 n/a

    Path List

    NH

    PE1

    PE2

    VID 100

    SMAC: M1

    DMAC: F.F.F

    MP2P VPN Label

    downstream allocated label

    used by other PEs to send

    traffic to advertised MAC

    MAC advertised

    by route

    B-M1

    B-M1

    B-M2

    B-M2

    PE3 MAC Table

    I-SID xyz

    C-MAC B-MAC

    M1 B-M1

    During start-up sequence,

    PE1 & PE2 advertised

    MAC route for B-MAC (B-

    M1)

    PE2 MAC Route

    RD = RD-2a

    ESI = all 1s

    MAC = B-M1

    Label = L2

    RT ext. community

    RT-a

    MPLS

    PE1

    CE1

    PE2

    MP

    ass

    for for t

    PSN MPLS label

    to reach PE2

    MP2P VPN

    Label

    assigned byPE1 for

    incoming traffic

    for target EVI

    PSN MPLS label

    to reach PE1

    PE

    on

    M1

    M1

    B-M1

    B-M1

    L2 PBB

    L1 PBB

    Data-plane based

    MAC learning for C-

    MAC / B-MAC

    association

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    36/59

    Flows and Use CasesPBB-EVPN Operational / Failure scenarios

    PBB EVPN O i l S i

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    37/59

    2014 Cisco and/or its affiliates. All rights reserved.

    PBB-EVPN Operational ScenariosMAC Mobility

    MPLS

    PE1

    CE1

    PE2

    MPLS

    PE1

    CE1

    PE2

    PE3

    CE3

    PE4

    VID 100

    SMAC: M1

    DMAC: M2

    PE1 learns C-MAC M1 on local

    port and forwards across core

    according to C-MAC DA to

    Remote B-MAC mapping

    1Host M1 movesfrom CE1 to CE3s

    location

    3

    M1M1

    Via data-plane

    learning, PE3 learnsC-MAC M1 via B-

    MAC B-M1

    2

    Via data-plane

    learning, PE1updates C-MAC M1

    location (via B-MAC

    B-M2)

    5

    B-M1

    B-M1

    B-M2

    B-M2

    L1 L2 PBB

    PE1 MAC Table

    I-SID xyz

    C-MAC B-MAC

    M1 -

    PE3 MAC Table

    I-SID xyz

    C-MAC B-MAC

    M1 B-M1

    PE1 MAC Table

    I-SID xyz

    C-MAC B-MAC

    M1 B-M21

    1 4

    2

    5

    B-M1

    B-M1

    L3 L4 PBB

    MAC Mobility event handled entirely by data-

    plane learning

    PBB EVPN F il S i / C

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    38/59

    2014 Cisco and/or its affiliates. All rights reserved.

    PBB-EVPN Failure Scenarios / ConvergenceLink / Segment FailureActive/Active per Flow

    MPLS

    PE1

    CE1

    PE2

    PE3

    CE3

    PE4

    PE3, PE4 RIB

    VPN MAC ESI

    RT-a B-M1 n/a

    Path List

    NH

    PE1

    PE2

    PE1 withdraws B-MACadvertised for failed

    segment (B-M1)

    2

    PE2 reruns DF election.

    Becomes DF for all I-

    SIDs on segment

    4

    PE3 / PE4

    remove PE1 frompath list for B-

    MAC (B-M1)

    3

    PE1 detects failureof one of its

    attached segments

    1

    PE1

    B-M1

    B-M1

    B-M2

    B-M2PE1 withdraws EthernetSegment Route

    2

    PBB EVPN F il S i / C

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    39/59

    2014 Cisco and/or its affiliates. All rights reserved.

    PBB-EVPN Failure Scenarios / Convergence

    PE Failure Core Isolation

    MPLS

    PE1

    CE1

    PE2

    PE3

    CE3

    PE4

    PE3, PE4 RIB

    VPN MAC ESI

    RT-a B-M1 n/a

    Path List

    NH

    PE1

    PE2

    BGP RR / PE2 detects

    BGP session time-out

    with PE1

    2PE3 / PE4

    invalidate routes

    from PE1

    3

    PE1 experiences a

    node failure (e.g.

    power failure)

    1BGP RR / PE3 detects

    BGP session time-outwith PE1

    2

    PE2 reruns DF election.

    Becomes DF for all I-

    SIDs on segment

    4

    BGP RR / PE4

    detects BGP

    session time-

    out with PE1

    2

    PE1

    B-M1

    B-M1

    B-M2

    B-M2

    MPLS

    PE1

    CE1

    PE2

    P

    VPN

    RT-a

    BGP RR / PE2

    detects BGP

    session time-

    out with PE1

    2

    PE2 reruns DF election.

    Becomes DF for all I-

    SIDs on segment

    5

    BGP RR / PE4

    detects BGP

    session time-

    out with PE1

    2

    PE1 looses

    connectivity to thecore

    1

    LACP PDU

    PE1 sends LACP

    OUT_OF_SYNC forCE1 to take port out

    of the bundle

    3

    B-M1

    B-M1

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    40/59

    Use Cases

    PBB-EVPN Model

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    41/59

    2014 Cisco and/or its affiliates. All rights reserved.

    PBB-EVPN Model

    BD

    BD

    I-ComponentB-Component

    E-VPN

    Forwarder

    BD

    BD

    Ethernet

    SegmentIdentifier

    ESI 1

    ESI 2

    BD

    BD

    B-MAC1

    B-MAC2

    EFP

    Customer Bridge

    Domain Core Bridge

    Domain

    PBB EVPN

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    42/59

    2014 Cisco and/or its affiliates. All rights reserved.

    PBB-EVPNSample Use Access

    Null Ethernet SegmentIdentifier (ESI)

    PE1

    CE1

    MPLSCore

    PE2

    BMAC1ESI W

    BMAC1ESI W

    Dual Home Device (DHD)Active / Active Per-Flow LB

    VIDX

    VIDX

    PE1

    CE1

    PE2

    BMAC2ESI W

    BMAC1ESI W

    Dual Home DevActive / Active Pe

    VIDX

    VIDY

    PE1

    CE1

    MPLSCore

    ESI Null

    Single Home Device (SHD)Single Home Network (SHN)

    VID X

    VID X

    Identical B-MAC on PBB-

    EVPN PEs (PE1 / PE2)

    Identical ESI on PBB-EVPN

    PEs

    Different B-M

    EVPN PEs (P

    Identical ESI

    PEs

    Per service (

    (manual or a

    CE2

    ESI Null

    PBB EVPN

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    43/59

    2014 Cisco and/or its affiliates. All rights reserved.

    PBB-EVPNSample Use Access (cont.)

    PE1

    CE1

    MPLSCore

    PE3

    BMAC 1ESI W

    BMAC 1ESI W

    Multi Home Device (MHD)Active / Active Per-Flow LB

    VID X

    VID X

    Multi Home Device (MHD)

    Active / Active Per-Service LB

    More than two (2) PEs in

    redundancy group

    Same as DHD Act/Act

    per-flow LB

    More than two (2) PEs in

    redundancy group

    Same as DHD Act/Act

    per-service LB

    PE2

    PE1

    CE1

    MPLSCore

    PE3

    BMAC 3ESI W

    BMAC 1ESI W

    VID X

    VID Z

    PE2

    VID YVID X

    BMAC 1ESI W

    BMAC 2ESI W

    PBB EVPN

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    44/59

    2014 Cisco and/or its affiliates. All rights reserved.

    PBB-EVPNSample Use Access (cont.)

    PE1

    MPLSCore

    PE2CE2

    CE1

    G.8032

    Open Sub-ring

    Dual Home Network (DHN)ITU-T G.8032

    Treated as SHN by PBB-

    EVPN PEs (PE1 / PE2)

    Null ESI; No DF election / No

    service carving

    Ring operation controlled by

    R-APS protocol

    PE1

    MPLSCore

    PE2CE2

    CE1

    REP

    Dual Home Network (DHN)REP

    Treated as SHN by PBB-EVPN PEs (PE1 / PE2)

    Null ESI; No DF election / No

    service carving

    Segment operation

    controlled by REP protocol

    R-APS

    RPL

    Link

    ALT

    port

    REP

    Edge NoNeighbou

    r

    REP-AGREP-AG

    CE2

    CE1

    Dual Home Active / Activ

    Different

    EVPN P

    Identical

    PEs

    Per serv

    (manual

    ESI Null

    ESI Null

    ESI Null

    ESI Null BMAC 2

    ESI W

    BMAC 1

    ESI W

    VID X

    VID Y

    VID X

    VID Y

    VID X

    VID Y

    VID X

    VID Y

    VID X

    VID Y

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    45/59

    PBB-EVPN IOS-XR ImplementationConfiguration and Examples

    PBB-EVPN Single Home Device (SHD)

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    46/59

    2014 Cisco and/or its affiliates. All rights reserved.

    PBB EVPN Single Home Device (SHD)

    PE1

    interface Bundle-Ether1.777 l2transport

    encapsulation dot1q 777

    l2vpn

    bridge group gr1

    bridge-domain bd1

    interface Bundle-Ether1.777

    pbb edge i-sid 100 core-bridge-domain core_bd1

    bridge group gr2

    bridge-domain core_bd1

    pbb-core

    evpn evi 1000

    router bgp 64

    address-family l2vpn evpn

    !

    neighbor

    remote-as 64

    address-family l2vpn evpn

    CE1

    PBB B-component

    No need to define B-VLAN

    Mandatory- Globallyunique identifier for all PEs

    in a given EVI

    PBB I-component

    Includes I-SID assignment

    BGP configuration with

    new E-VPN AF

    Global B-MAC SA

    Auto RT for EVIAuto RD for EVI

    Auto RD for Segment Route

    Note

    requishow

    PBB-EVPN Dual Home Device (DHD)

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    47/59

    2014 Cisco and/or its affiliates. All rights reserved.

    PBB-EVPN Dual Home Device (DHD)Active / Active per-FLOW Load Balancing

    CE1

    PE1

    redundancy

    iccp

    group 66

    mlacp node 1mlacp system mac 0aaa.0bbb.0ccc

    mlacp system priority 1

    mode singleton

    interface Bundle-Ether25

    mlacp iccp-group 66

    interface Bundle-Ether25.1 l2transport

    encapsulation dot1q 777

    l2vpn

    bridge group gr1

    bridge-domain bd1interface Bundle-Ether25.1

    pbb edge i-sid 100 core-bridge-domain core_bd1

    bridge group gr2

    bridge-domain core_bd1

    pbb-core

    evpn evi 1000

    router bgp 64

    address-family l2vpn evpn

    neighbor remote-as 64

    address-family l2vpn evpn

    ICCP in singleton mode (i.e.No

    peer neighbor configuration)

    PE2 should use same RG

    number

    PE 2 should use different mlacpnode id

    PE2 should use same mlacp

    system mac and system priority

    PBB I-component and B-component configuration. ISIDs

    must match on both PEs

    No need to define B-VLAN

    MandatoryEVI ID configuration

    BGP configuration with

    new EVPN AF

    Auto ESI

    Auto B-MAC SA

    A/A Per-flow LB (default)

    Auto RT for EVI

    Auto RD for EVI

    Auto RD for Segment Route

    Note

    requishow

    PBB-EVPN Dual Home Device (DHD)

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    48/59

    2014 Cisco and/or its affiliates. All rights reserved.

    PBB-EVPN Dual Home Device (DHD)Active / Active per-Service Load Balancing and Dynamic Service Ca

    CE1

    PE1

    interface Bundle-Ether25.1 l2transport

    encapsulation dot1q 777

    evpn

    interface Bundle-Ether25

    ethernet-segment

    identifiersystem-priority1 system-id0000.0b25.00ce

    load-balancing-mode per-service

    l2vpn

    bridge group gr1

    bridge-domain bd1

    interface Bundle-Ether25.1

    pbb edge i-sid 100 core-bridge-domain core_bd1

    bridge group gr2

    bridge-domain core_bd1

    pbb-core

    evpn evi 1000

    router bgp 64

    address-family l2vpn evpn

    neighbor remote-as 64

    address-family l2vpn evpn

    Global B-MAC SA

    Default Service CarvingAuto RT for EVI

    Auto RD for EVI

    Auto RD for Segment Route

    A/A per-service (per-ISID)

    load balancing with

    dynamic Service Carving

    ESI must match on both

    PEs

    BGP configuration with

    new EVPN AF

    PBB I-component and B-

    component configuration.ISIDs must match on both

    PEsNo need to define B-VLAN

    MandatoryEVI ID

    configuration Note

    requishow

    show

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    49/59

    Summary

    Summary

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    50/59

    2014 Cisco and/or its affiliates. All rights reserved.

    y

    E-VPN / PBB-EVPN are next-generation L2VPN solutions based ocontrol-plane for MAC distribution/learning over the core

    E-VPN / PBB-EVPN were designed to address following requireme All-active Redundancy and Load Balancing

    Simplified Provisioning and Operation

    Optimal Forwarding

    Fast Convergence

    In addition, PBB-EVPN and its inherent MAC-in-MAC hierarchy pro

    Scale to Millions of C-MAC (Virtual Machine) Addresses

    MAC summarization co-existence with C-MAC (VM) mobility

    E-VPN / PBB-EVPN applicability goes beyond DCI into Carrier Ethcases

    References

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    51/59

    2014 Cisco and/or its affiliates. All rights reserved.

    draft-ietf-l2vpn-evpn

    draft-ietf-l2vpn-pbb-evpn

    draft-ietf-l2vpn-trill-evpn

    AcronymsIP and MPLS

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    52/59

    2014 Cisco and/or its affiliates. All rights reserved.

    y

    Acronym Description

    AC Attachment Circuit

    AS Autonomous System

    BFD Bidirectional Failure Detection

    CoS Class of Service

    ECMP Equal Cost Multipath

    EoMPLS Ethernet over MPLS

    E-VPN Ethernet Virtual Private Network

    EVI E-VPN Instance

    FRR Fast Re-Route

    IGP Interior Gateway Protocol

    LDP Label Distribution Protocol

    LER Label Edge Router

    LFIB Labeled Forwarding Information Base

    LSM Label Switched Multicast

    LSP Label Switched Path

    LSR Label Switching Router

    MPLS Multi-Protocol Label Switching

    NLRI Network Layer Reachability Information

    PSN Packet Switch Network

    Acronym Description

    PW Pseudo-Wire

    PWE3 Pseudo-Wire End-to-End

    QoS Quality of Service

    RD Route Distinguisher

    RIB Routing Information Base

    RR Route Reflector

    RSVP Resource Reservation P

    RSVP-TE RSVP based Traffic Eng

    RT Route Target

    TE Traffic Engineering

    tLDP Targeted LDP

    VC Virtual Circuit

    VCID VC Identifier

    VFI Virtual Forwarding Instan

    VPLS Virtual Private LAN Serv

    VPN Virtual Private Network

    VPWS Virtual Private Wire Serv

    VRF Virtual Route Forwarding

    VSI Virtual Switching Instanc

    AcronymsEthernet/Bridging

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    53/59

    2014 Cisco and/or its affiliates. All rights reserved.

    Acronym Description

    ACL Access Control List

    BD Bridge Domain

    BPDU Bridge Protocol Data Unit

    CE Customer Equipment (Edge)

    C-VLAN / CE-VLAN

    Customer / CE VLAN

    CoS Class of Service

    DHD Dual Homed Device

    LACP Link Aggregation Control Protocol

    LAN Local Area Network

    MEF Metro Ethernet Forum

    MEN Metro Ethernet Network

    MIRP Multiple I-Tag Registration Protocol

    mLACP Multi-Chassis LACP

    MST / MSTP Multiple Instance STP

    MSTG-AG MST Access Gateway

    Acronym Description

    MVRP Multiple VLAN Registration Protoc

    PE Provider Edge device

    PoA Point of Attachment

    REP Resilient Ethernet Protocol

    REP-AG REP Access Gateway

    RG Redundancy Group

    STP Spanning Tree Protocol

    AcronymsProvider Backbone Bridging

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    54/59

    2014 Cisco and/or its affiliates. All rights reserved.

    Acronym Description

    B-BEB B-Component BEB

    BCB Backbone Core Bridge

    B-DA Backbone Destination AddressBEB Backbone Edge Bridge

    B-MAC Backbone MAC Address

    B-SA Backbone Source Address

    B-Tag B-VLAN Tag

    B-VLAN Backbone VLAN

    C-DA Customer Destination Address

    CE Customer Equipment (Edge)

    C-MAC Customer MAC Address

    C-SA Customer Source Address

    80 C-VLAN Tag

    C-VLAN / CE-VLAN

    Customer / CE VLAN

    DA Destination MAC Address

    FCS Frame Check Sequence

    IB-BEB Combined I-Component & B-Component BEB

    Acronym Description

    I-BEB I-Component BEB

    IEEE Institute of Electrical and

    I-SID Instance Service Identifi

    I-Tag I-SID Tag

    MAC Media Access Control

    N-PE Network-facing Provider

    PB Provider Bridge

    PBB Provider Backbone Bridg

    PBBN Provider Backbone Bridg

    PBN Provider Bridging Netwo

    PE Provider Edge deviceQ-in-Q VLAN tunneling using tw

    SA Source MAC Address

    S-Tag S-VLAN Tag

    S-VLAN Service VLAN (Provider

    UNI User to Network Interfac

    U-PE User-facing Provider Ed

    VLAN Virtual LAN

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    55/59

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    56/59

    BGP Routes and Attributes

    BGP Routes

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    57/59

    2014 Cisco and/or its affiliates. All rights reserved.

    (PBB) E-VPN defines a single new BGP NLRI used to carry all E-V

    The NLRI has a new SAFI (70).

    (PBB) E-VPN speakers must first exchange BGP capability for E-VSAFI per RFC4760.

    Overview

    Route Type

    Length

    Route Type Specific

    1 byte

    1 byte

    Variable

    1. Ethernet Auto

    Discovery (AD) R

    2. MAC Advertis

    Route

    3. Inclusive MulRoute

    4. Ethernet Seg

    Route

    BGP Routes

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    58/59

    2014 Cisco and/or its affiliates. All rights reserved.

    Route Types and Usage

    Route Usage Ap

    Ethernet A-D Route MAC Mass-Withdraw

    Aliasing

    Advertising Split-Horizon Labels

    E-V

    MAC Advertisement

    Route

    Advertise MAC Address Reachability

    Advertise IP/MAC Bindings

    E-V

    PB

    Inclusive Multicast

    Route

    Multicast Tunnel Endpoint Discovery E-V

    PB

    Ethernet Segment

    Route

    Redundancy Group Discovery

    DF Election

    E-V

    PB

    BGP Routes

  • 8/10/2019 Apricot2014 - E-VPN & Pbb-evpn the Next Generation of Mpls-based l2vpn

    59/59

    2014 Cisco and/or its affiliates. All rights reserved.

    Route Attributes and Usage

    Attribute Usage Ro

    App

    ESI MPLS Label

    Extended Community

    Encode Split-Horizon Label for Ethernet

    Segment.

    Indicate Redundancy Mode (Active/Standby vs.

    All-Active)

    Eth

    Rou

    ES-Import Extended

    Community

    Limit the import scope of the Ethernet Segment

    routes.

    Eth

    Seg

    MAC Mobility Extended

    Community

    E-VPN: Indicate that a MAC address has moved

    from one segment to another across PEs.

    PBB-EVPN: Signal C-MAC address flush

    notification

    MA

    Adv

    Rou

    Default Gateway

    Extended Community

    Indicate the MAC/IP bindings of a gateway MA

    Adv

    Rou


Recommended