Building Data Center Networks with VXLAN EVPN … · • Introduction to Data Center Fabric ......

Building Data Center Networks with VXLAN EVPN Overlays – Part I

Lukas Krattiger, Principal Engineer

BRKDCT-2949

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public

Cisco Spark

Questions? Use Cisco Spark to communicate with the speaker after the session

1. Find this session in the Cisco Live Mobile App

2. Click “Join the Discussion”

3. Install Spark or go directly to the space

4. Enter messages/questions in the space

How

cs.co/ciscolivebot#BRKDCT-2949


Session Objective

• A short Overview on Data Center Fabric

• A close look at Single Fabric Overlay and Underlay

• Details on Single Fabric Control- & Data-Plane

• Multi-Tenancy in VXLAN BGP EVPN environments

• First-Hop Gateway with Distributed Anycast Gateway

• Multi-Homing with Virtual Port-Channel (VPC) for VXLAN

5BRKDCT-2949

• Introduction to Data Center Fabric• Leaf, Spine, Super-Spine (CLOS)

• Overlay

• Underlay

• VXLAN with BGP EVPN• Control & Data Plane

• Multi-Tenancy

• Distributed Anycast Gateway

• VPC

• A Deployment Story

Agenda

Introduction to Data Center Fabrics

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 8BRKDCN-2949

Data Center “Fabric” Journey (Standalone)

Layer-2Layer-2 Layer-2

Layer-2 Layer-2Layer-2 Layer-2

Hypervisor HypervisorHypervisor HypervisorBaremet al Baremet al Baremet alBaremet al Hypervisor Hypervisor

Spanning-Tree

Layer-3

Layer-2

HSRP HSRP


• Overlay

• Underlay


• Multi-Tenancy


• VPC


Agenda

© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public 10

The Leaf / Spine Topology (Clos* Network)

• Wide ECMP: Unicast or Multicast

• Uniform Reachability

• Deterministic Latency

• High Redundancy• On Node or Link Failure

SpineSpine Spine Spine

Leaf LeafLeaf LeafLeaf Leaf Leaf

*Clos, Charles (1953) "A study of non-blocking switching networks"BRKDCN-2949




11

A Scale Out Architecture

• Leaf• Smallest Operational Entity

• Spines• Wide vs. Big

• Uplinks• Symmetric to all Spines or Pods

• SAYG: Scale as You Grow

More Spine – More Bandwidth – More Resiliency

More Leaf – More Ports – More Capacity

BRKDCN-2949


The Super-Spine



POD 2



POD 1

SuperSpine SuperSpine

SuperSpine

BRKDCN-2949


The Super-Spine



POD 2



POD 1


SuperSpine

• Scale Out• Not Limited to Port Density

• Simpler Capacity Planning

• Beyond a single Server Room• Allows Interconnecting Pods

• Retains Intra-Pod Topology with

Flexible Inter-Pod Connectivity

BRKDCN-2949


Data Center Fabric Properties



• Any Subnet, Anywhere, Rapidly• Any Network on Any Leaf

• Reduced Failure Domain• Any Default Gateway on Any Leaf

- Distributed

• Extensible Scale and Resiliency

BRKDCN-2949


Overlay Based Data Center: Fabrics

• Mobility

• Segmentation

• Scale

• Automated and Programmable

• Abstracted Consumption Model

• Layer-2 and Layer-3 Service

• Physical and Virtual Workloads


Overlay

VTEP VTEPVTEP VTEPVTEP VTEP VTEP

BRKDCN-2949


• Router/Switch End-Points

• Protocols for Resiliency/Loops

• Traditional VPNs

• VXLAN, OTV, VPLS, LISP, FP

Overlay Based Data Center: Edge Devices

Overlay

VTEP VTEP

Baremetal Baremetal

VTEP VTEP

Baremetal Baremetal

Network Overlays

• Virtual End-Points only

• Single Admin Domain

• VXLAN, NVGRE, STT

Overlay

- -

Host Overlays

-

Hypervisor

VTEP

-

Hypervisor

VTEP

Hypervisor

VTEP

Hypervisor

VTEP

• Physical and Virtual

• Resiliency and Scale

• Cross-Organizations/Federation

• Open Standards

Overlay

VTEP VTEP

Baremetal Baremetal

Hybrid Overlays

-

Hypervisor

VTEP

-

Hypervisor

VTEP


• Overlay

• Underlay


• Multi-Tenancy


• VPC


Agenda


Overlay Taxonomy - Underlay

Underlay



HypervisorHypervisor HypervisorHypervisor BaremetalBaremetal BaremetalBaremetal

Virtual Server Physical

Server

Edge Device

Layer-3 Interface Peering

LAN Segment

BRKDCN-2949


Overlay Taxonomy - Overlay


Overlay



Tunnel Encapsulation

(VNI Namespace)

Virtual Server Physical

Server

LAN Segment

VTEP

VTEP: VXLAN Tunnel End-Point

VNI/VNID: VXLAN Network Identifier

BRKDCN-2949


Understanding Overlay Technologies

Overlay Services• Layer-2

• Layer-3

• Layer-2 and Layer-3

Tunnel EncapsulationUnderlay Transport

Network

Control-Plane• Peer-Discovery

• Route Learning and Distribution• Local Learning

• Remote Learning

Data-Plane• Overlay Layer-2/Layer-3 Unicast Traffic

• Overlay Broadcast, Unknown Unicast,

Multicast traffic (BUM traffic)

forwarding• Ingress Replication (Unicast)

• Multicast

BRKDCN-2949


Back Then

Yet Another Encapsulation

Flood & Learn (Multicast-based)

Data-Plane only 4 Years ago

VXLAN for the Data Center – Intra-DC

Control-Plane

Active VTEP Discovery

Multicast and Unicast

21BRKDCN-2949

VXLAN Evolves as the Control Plane Evolves!


A single Fabric with Overlay



POD 1

BRKDCN-2949

Overlay

• A Single Overlay Domain

• End-to-End Encapsulation• Closest to the Source

• Closest to the Destination

• External Connectivity• @ Leaf = Border Leaf

• @ Spine = Border Spine


What is the Elephant in the Room?

BRKDCN-2949


The Super-Spine and the Overlay (Multi-POD)



POD 2



POD 1


SuperSpine

BRKDCN-2949

Overlay Overlay





POD 2



POD 1


SuperSpine

BRKDCN-2949

Overlay Overlay

• Still, a Single Overlay Domain

• End-to-End Encapsulation• Closest to the Source


• External Connectivity• @ Leaf = Border Leaf


• @ Super-Spine or Connected to

Super-Spine


Attributes of Multi-POD in VXLAN BGP EVPN



POD 2



POD 1


SuperSpine

BRKDCN-2949

Overlay Overlay

• Underlay• Nicely Structured and Tiered Topologies

• Allows Efficient Scale-Out

• More End-Points = More Leaf

• More Bandwidth, Resilience or Capacity = More Spine or Tiers

• Different Control-Plane Instances (BGP AS)

• Overlay• End-to-End Encapsulation, Flat, No Hierarchy

• Single Control-Plane ”reach” – all in one ”kitchen sink”





POD 2



POD 1


SuperSpine

BRKDCN-2949

Overlay Overlay

• Scale-Out Model to Build a Large

Intra-DC Network?

• Data Center Interconnect (DCI)?

• Domain Normalization

(Coexistence and/or Migration)?


Back Then

Yet Another Encapsulation

Flood & Learn (Multicast-based)

Data-Plane only 4 Years ago

VXLAN for the Data Center – Intra-DC

Control-Plane

Active VTEP Discovery

Multicast and Unicast

28BRKDCN-2949

VXLAN Evolves as the Control Plane Evolves!

Today

VXLAN for DCI – Inter-DC

Multi-Site

Control- & Data-Plane Separation

Failure Domain Isolation


Use-Cases for ”DCI” Connectivity

BRKDCN-2949

Scale-Out Model to Build a Large Intra-

DC Network

Data Center Interconnect (DCI)

Domain Normalization

(Coexistence and/or Migration)


VXLAN for Interconnecting Networks

BRKDCN-2949

Single Fabric with End-to-End

Encapsulation

Build Hierarchy in the Underlay

– Flatten it in the Overlay

VXLAN Multi-Pod

OverlayVTE

PVTE

PVTE

PVTE

P

Baremeta

l

Baremeta

l

Fabric #2

OverlayVTE

PVTE

P

Baremeta

l

Baremeta

l

VTEP

VTEP

Fabric #1EVPN Control-

Plane Domain 1

EVPN Control-

Plane Domain 2

Single Data-Plane – End-to-End

BGP EVPN

OverlayVTE

PVTE

PVTE

PVTE

P

Baremetal

Baremetal

Fabric #2

OverlayVTE

PVTE

P

Baremetal

Baremetal

VTEP

VTEP

Fabric #1EVPN Control-Plane

Domain 1

EVPN Control-Plane

Domain 2

Data-Plane Domain 1 Data-Plane Domain 2DCI

Data-Plane

Multiple Fabrics – Normalized

through Ethernet

Multiple Fabrics Interconnect

using DCI (Layer 2 and Layer 3)

VXLAN Multi-Fabric





BRKDCN-2949

Single Fabric with End-to-End

Encapsulation

Build Hierarchy in the Underlay

– Flatten it in the Overlay

VXLAN Multi-Pod

OverlayVTE

PVTE

PVTE

PVTE

P

Baremeta

l

Baremeta

l

Fabric #2

OverlayVTE

PVTE

P

Baremeta

l

Baremeta

l

VTEP

VTEP

Fabric #1EVPN Control-

Plane Domain 1

EVPN Control-

Plane Domain 2

Single Data-Plane – End-to-End

BGP EVPN

OverlayVTE

PVTE

PVTE

PVTE

P

Baremetal

Baremetal

Fabric #2

OverlayVTE

PVTE

P

Baremetal

Baremetal

VTEP

VTEP


Domain 1

EVPN Control-Plane

Domain 2


Data-Plane

Multiple Fabrics – Normalized

through Ethernet

Multiple Fabrics Interconnect

using DCI (Layer 2 and Layer 3)

VXLAN Multi-Fabric


Data-Plane

OverlayVTE

PVTE

PVTE

PVTE

P

Baremetal

Baremetal

Fabric #2

OverlayVTE

PVTE

P

Baremetal

Baremetal

VTEP

VTEP


Domain 1

EVPN Control-Plane

Domain 2BGP EVPN

Multiple Fabrics with

Integrated DCI

Integrated DCI – Scaling

within and between Fabrics

VXLAN Multi-Site


VXLAN Multi-Site – Introducing Overlay Hierarchies



Site 2



Site 1


SuperSpine

BRKDCN-2949

Overlay Overlay

Multi-Site Overlay

VTEP

Border Gateways (BGW)(Key Functional Components of

VXLAN Multi-Site Architecture)

VTEP


VXLAN Multi-Site – Introducing Overlay Hierarchies



Site 2



Site 1


SuperSpine

BRKDCN-2949

Overlay Overlay

Multi-Site Overlay

VTEP VTEP

• Multiple Overlay Domains

• Per-Site Encapsulation• Closest to the Source


• Exit/Transit via Border Gateway

(BGW)

• Multi-Site and/or External

Connectivity• @ Leaf = Border Leaf


• Super-Spine becomes Transit


VXLAN Multi-Site for Interconnecting Networks

Your Happy Place!


• Overlay

• Underlay


• Multi-Tenancy


• VPC


Agenda


MTU and Overlays

• Data Center often require Jumbo

MTU• Most Server NIC support up to

9000 Bytes

• Network Switches support MTU

up to 9216* Bytes• Accommodates Jumbo MTU plus

Overlay overhead

• Avoid Fragmentation• Adjust the Transport Network with

appropriate MTU

*Cisco Nexus 5600 only supports a MTU of 9192 Byte for Layer-3 TrafficBRKDCN-2949


Underlay



38

Interface Principles

• Routed Ports and Interfaces• Layer-3 Interfaces between Leaf

and Spine(no switchport)

• For each Point-2-Point (P2P)

connection, minimum /31

required

• Alternative, use IP Unnumbered

(/32)

• Use Loopback as Source-

Interface for VTEP (NVE*)

*NVE: Network Virtualization EdgeBRKDCN-2949


Underlay



39

IP Addressing Principles

• Prepare a IP Addressing Plan

• Separate Interface functions

through IP Addressing

(Aggregates)• Unicast Routing – Routing

Protocol Peering (p2p*)

• Unicast Routing – Routing

Identifier (RID)

• VTEP and VPC

• Multicast Rendezvous-Point (RP)

• IPv4 only (today)

p2p* Links / IP Unnumbered

VTEP Loopback

Routing Identifier

Rendezvous Point

p2p Agg: 10.1.1.0/24RID Agg: 10.10.10.0/24VTEP Agg: 10.200.200.0/24RP Agg: 10.254.254.0/24

*p2p: Point-to-Point

Routing Identifier

BRKDCN-2949


IP Addressing Principles

Underlay




interface ethernet4/4

description p2p-to-Leaf

ip address 10.1.1.2/30

interface ethernet1/49

description p2p-to-Spine


interface loopback0

description RID

ip address 10.10.10.101/32interface loopback1

description VTEP

ip address 10.200.200.101/32

interface loopback254

description RP

ip address 10.254.254.1/32

interface loopback0

description RID

ip address 10.10.10.201/32


BRKDCN-2949


Underlay



41

Some Math – IP Addressing Principles (P2P)

• Example from depicted Topology• 4 Spine * 7 Leaf (28 Links)

• 11 Router ID (RID Loopback)

• 7 VTEP (Loopback)

• 28 Link * 2 (/31) = 56 IP Addresses

• 11 Router ID (RID) = 11 IP Addresses

• 7 VTEP = 7 IP Addresses

• Total: 74 IP Addresses Required

BRKDCN-2949


Underlay



42

Simplifying the Math – IP Unnumbered

• Example from depicted Topology• 4 Spine + 7 Leaf (11 Loopback)

• 11 Router ID (RID Loopback)

• 7 VTEP (Loopback)

• 11 Unnumbered IF = 11 IP Addresses

• 11 Router ID (RID) = 11 IP Addresses

• 7 VTEP = 7 IP Addresses

• Total: 29 IP Addresses Required

BRKDCN-2949


Unicast Routing – OSPF and IS-IS

• OSPF – watch your Network Type

• Network Type Point-2-Point• Preferred (only LSA type-1)

• No DR/BDR election

• Suits well for routed interfaces/ports

(optimal from a LSA DB perspective)

• Full SPF calculation on Link Change

• IS-IS – what was this CLNS?• Independent of IP (CLNS)

• Well suited for routed

interfaces/ports

• No SPF calculation on Link change;

only if Topology changes

• Fast Re-convergence

• Not everyone is familiar with it

BRKDCN-2949


Unicast Routing – BGP

• eBGP Underlay Routing –

Service Provider style• Two Different Models

• Two-AS

• Multi-AS

• BGP is a Distance Vector

Protocol

• actually Path Vector Protocol• AS* are used to calculate the

Path (AS_Path)

BRKDCN-2949


Unicast Routing – eBGP Two-AS Model

• eBGP Two-AS, yes it works!

• eBGP peering for Underlay• Spine is not a Route-Reflector

(eBGP) – Retain Route-Targets

• Disable BGP AS-Path check

• Next-Hop needs to be

Unchanged

• Underlay is Reachability!• Advertise your Loopbacks

Underlay



All-Spine AS#65500

All-Leaf AS#65501

BRKDCN-2949


Unicast Routing – eBGP Multi-AS Model

Underlay



All-Spine AS#65500• eBGP Two-AS, yes it works!

• eBGP peering for Underlay• Spine is not a Route-Reflector

(eBGP) – Retain Route-Targets

• Disable BGP AS-Path check

• Next-Hop needs to be

Unchanged

• Underlay is Reachability!• Advertise your Loopbacks

• Changes Overlay Routing Policy• Manually define Route-Targets

BRKDCN-2949


Unicast Routing – eBGP Model

• Two different BGP Peering

• eBGP peering for Underlay• Global IPv4/v6 Address-Family

• Use Physical Interface IP

• eBGP peering for Overlay• Global EVPN Address-Family

• Use Loopback Interface IP

• BFD not so ok

Underlay



BRKDCN-2949


Unicast Routing – Why two different BGP Peering?

Spine

Spine

Leaf Leaf

AS#65500

BGP Peering (IPv4/IPv6)




1) Interface Down – BGP Down

• Point-2-Point Link Fails

• BGP Peering is teared down• Lights-Out Event or BFD

• Fast reaction to Routing Table

• Underlay Network Converges• ECMP kicks in if

available/configured

• IGPs do this Automatically

BRKDCN-2949



Spine

Spine

Leaf Leaf

BGP Peering (EVPN)


AS#65500

2) Interface Down – BGP Not Down

• Point-2-Point Link Fails

• Loopback to Loopback Peering

remains Up• If Alternate Path available

• Timers should allow Time for

Network Re-Convergence

• No BFD

• Unchanged Overlay Reachability• No Mass Delete/Re-Learn

• Underlay Path change only

BRKDCN-2949



Spine

Spine

Leaf Leaf


2) Interface Down – BGP Not Down

AS#65500

3) Leaf Down – Prefix are Withdrawn (RNH*)

*RNH: Recursive Next-Hop

• Leaf and p2p Interfaces Fail• Either IGP or BGP converges

• Loopback to Loopback Peering

remains Up• BGP Dead-Timer (180s)

• Recursive Next-Hop will trigger

Convergence Event• Next-Hop (VTEP) disappeared in

Underlay

• Overlay withdraws Prefixes

5 192.168.10.0/24

Next-Hop:

10.200.200.1022 0000.3001.1101

2 0000.3001.1101, 192.168.10.101

5 192.168.10.0/24

Next-Hop:

10.200.200.1022 0000.3001.1101

2 0000.3001.1101, 192.168.10.101

BRKDCN-2949


Underlay - Unicast Routing and Overlay

• Generic Concept for Underlay /

Overlay Separation• Use Different Routing Protocol

• Use Same Routing Protocol

• RNH* for Overlay works with ALL

Underlay Routing Protocols• Ensure /32-Reachability for

VTEPs

• Other Routes can impact

(Aggregates, Default-Route)

IGP + BGP for

true Protocol

Separation

BGP for single

Routing Protocol

approachSpecific to BGP

as a Overlay

Control-Protocol

BGP Knobs can

Help here

BRKDCN-2949


Underlay - Multicast Routing and Rendezvous-Point

• PIM Any-Source-Multicast (ASM)

• Platform Support• Nexus 9000 / Nexus 7000 (F3/M3)

• ASR 1000 / ASR 9000

• RP Redundancy• PIM Anycast-RP or MSDP

• Source-Trees (Unidirectional)• 1 Source Tree per VTEP per

Multicast Group

• Bidirectional PIM (Bidir)

• Platform Support• Nexus 5600 / Nexus 7000 (F3/M3)

• ASR 1000 / ASR 9000

• RP Redundancy• Phantom-RP

• Shared-Trees (Bidirectional)• 1 Shared Tree per Multicast Group

• Follows Unicast Routing Path

BRKDCN-2949


Underlay – PIM ASM with PIM Anycast-RP

Underlay




RP RP

S,G S,G S,G S,G S,G

BRKDCN-2949


PIM ASM – S,G for 5 VTEP

S,G

S,G

S,G

S,G

S,G

S=VTEP1

S=VTEP2 S=VTEP3

S=VTEP4

S=VTEP7


Underlay – PIM ASM with PIM Anycast-RP

Underlay




RP RPinterface loopback254

description RP

ip address 10.254.254.1/32

ip pim sparse-mode

interface loopback0

description RID

ip address 10.10.10.202/32

ip pim sparse-mode


description RP

ip address 10.254.254.1/32

ip pim sparse-mode

interface loopback0

description RID

ip address 10.10.10.201/32

ip pim sparse-mode

ip pim anycast-rp 10.254.254.1 10.10.10.201

ip pim anycast-rp 10.254.254.1 10.10.10.202

ip pim rp-address 10.254.254.1 (Leaf&Spine)

BRKDCN-2949


Underlay - Multicast Routing and Rendezvous-Point

• The Spine makes a good

Rendezvous-Point (RP)• Use multiple RP for Redundancy

• Watch your Multicast-Group and

OIF* scale

• VXLAN uses Multicast for BUM• Broadcast, Unknown Unicast,

Multicast

• 1:1 Multicast-to-VNI mapping

• 1:N Multicast-to-VNI mapping

• Ingress-Replication can be valid

as well

*OIF: Outgoing InterfaceBRKDCN-2949


Underlay – Ingress Replication

Underlay



• A Packet Multiplication• EVPN assists, VNI Topology

• Various Platform Support• Ie Nexus 9000

• Ingress Replication• Host sends 1 Packet to Edge-Device

• Edge-Device Encapsulates 1 Packet

and multiplies it

• Ingress VTEP sends 1 Packet per

Neighbor

BRKDCN-2949

VXLAN with BGP EVPN


• Overlay

• Underlay


• Multi-Tenancy


• VPC


Agenda


What is … ?

• VXLAN

• Standards based Encapsulation• RFC 7348

• Uses UDP-Encapsulation

• Transport Independent• Layer-3 Transport (Underlay)

• Flexible Namespace• 24-bit field (VNID) provides ~16M

unique identifier

• Allows Segmentations

• EVPN

• Standards based Control-Plane• RFC 7432

• Uses Multiprotocol BGP

• Uses Various Data-Planes• VXLAN (EVPN-Overlay), MPLS,

Provider Backbone (PBB)

• Many Use-Cases Covered• Bridging, MAC Mobility, First-Hop &

Prefix Routing, Multi-Tenancy (VPN)

BRKDCN-2949


Introducing Ethernet VPN (EVPN)

Overlay (NVO3)

(draft-ietf-bess-evpn-overlay)

Provider Backbone Bridges

(draft-ietf-l2vpn-pbb-evpn)

MPLS

(draft-ietf-l2vpn-evpn)

EVPN MP-BGP – RFC 7432

BRKDCN-2949


VXLAN and EVPN related RFCs & Drafts (IETF)

ID Title Category

RFC 7348 Virtual Extensible Local Area Network Data Plane

RFC 7432 BGP MPLS based Ethernet VPNs Control Plane

draft-ietf-bess-evpn-overlay A Network Virtualization Overlay Solution using EVPN Control Plane

draft-ietf-bess-evpn-inter-subnet-forwarding Integrated Routing and Bridging in EVPN Control Plane

draft-ietf-bess-l2vpn-evpn-prefix-advertisement IP Prefix Advertisement in E-VPN Control Plane

draft-tissa-nvo3-oam-fm NVO3 Fault Management / OAM Management Plane

65BRKDCN-2949


Multiprotocol BGP (MP-BGP) Primer

Spine

Spine

Leaf Leaf

AS#65500

• Multiprotocol BGP (MP-BGP)

• Extension to Border Gateway

Protocol (BGP)• RFC 4760

• VPN Address-Family• Allows different types of Address-

Families (i.e. VPNv4/v6, MVPN,

L2VPN, EVPN)

• Various Information transported

over single BGP Peering

BRKDCN-2949



Spine

Spine

Leaf Leaf

AS#65500

vrf context A

rd 10.10.10.101:3

address-family ipv4 unicast

route-target import 65500:5000

route-target export 65500:5000

vrf context A

rd 10.10.10.102:8



route-target export 65500:5000• VPN Segmentation for Tenant

Routing

• Route Distinguisher (RD)• 8-byte field

• A Value to make a VPN Prefix

unique• RD + VPN Prefix• [10.10.10.101:5000 + 192.168.10.0/24]

BRKDCN-2949



Spine

Spine

Leaf Leaf

AS#65500

vrf context A

rd auto




vrf context A

rd auto



route-target export 65500:5000• Cisco provides automated Route

Distinguisher derivation

• Macros uses Type 1 format• 4-byte Router ID (RID)

• 4-byte VRF ID (internal number)

• Example of auto derived RD:

• 10.10.10.101:3

BRKDCN-2949



Spine

Spine

Leaf Leaf

AS#65500

vrf context A

rd auto




vrf context A

rd auto




• VPN Segmentation for Tenant

Routing

• Route Target (RT)• 8-byte field

• A Value to import/export a VPN

Prefix• Each RD + VPN Prefix have an

RT• [10.10.10.101:5000 + 192.168.10.0/24]

• [65500:5000, 65500:5000]

BRKDCN-2949



Spine

Spine

Leaf Leaf

AS#65500

vrf context A

rd auto


route-target import auto

route-target export auto

vrf context A

rd auto


route-target import auto

route-target export auto• Cisco provides automated Route

Target derivation

• Macros uses following values• 4-byte Autonomous System

• 4-byte VNI

• Example of auto derived RD:

• Import, Export or Both

• 65500:5000

BRKDCN-2949



Spine

Spine

Leaf Leaf

AS#65500

vrf context A

rd 10.10.10.101:3




vrf context A

rd 10.10.10.102:8




RD Prefix Next-Hop Route Target

10.10.10.101:3 192.168.10.0/24 10.200.200.101 65500:5000, 65500:5000

10.10.10.101:7 192.168.20.0/24 10.200.200.101 65500:5001, 65500:5001

BRKDCN-2949



Spine

Spine

Leaf Leaf

AS#65500

vrf context A

rd 10.10.10.101:3




vrf context A

rd 10.10.10.102:8





10.10.10.101:3 192.168.10.0/24 10.200.200.101 65500:5000, 65500:5000

10.10.10.101:7 192.168.20.0/24 10.200.200.101 65500:5001, 65500:5001

BRKDCN-2949



Spine

Spine

Leaf Leaf

AS#65500

vrf context A

rd 10.10.10.101:3




vrf context A

rd 10.10.10.102:8





10.10.10.101:3 192.168.10.0/24 10.200.200.101 65500:5000, 65500:5000

10.10.10.101:7 192.168.20.0/24 10.200.200.101 65500:5001, 65500:5001

BRKDCN-2949



Spine

Spine

Leaf Leaf

AS#65500

vrf context A

rd 10.10.10.101:3




vrf context A

rd 10.10.10.102:8




5 192.168.10.0/24Next-Hop:

10.200.200.101

BRKDCN-2949


EVPN - Host and Subnet Route Distribution

• Host Route Distribution

decoupled from the Underlay

protocol

• Use MultiProtocol-BGP (MP-

BGP) on the Leaf nodes to

distribute internal Host/Subnet

Routes and external reachability

information

• Route-Reflectors (RR) deployed

for scaling purposes


Overlay


RR RR

BRKDCN-2949


EVPN Control Plane - Host and Subnet Routes

• BGP EVPN NLRI*

• Host MAC (Route Type 2)• MAC only, Single VNI, Single

Route Target

• Host MAC+IP (Route Type 2)• MAC and IP, Two VNI, Two Route

Target, Router MAC

• Internal and External Subnet

Prefixes (Route Type 5)• IP Subnet Prefix, Single VNI,

Single Route Target

*NLRI: Network Layer Reachability Information (BGP Update Format)


Overlay


BRKDCN-2949


Host Advertisements


Overlay


Baremetal BaremetalBaremetal

Type MAC / Length L2VNI / RT IP / Length L3VNI / RT Next-Hop Seq.

2 0000.3001.1101 / 48 3001, 65500:3001 10.200.200.101

Host AMAC: 0000.3001.1101

Host BMAC: 0000.3001.1102

Host CMAC: 0000.3002.2101

*L2VNI: VNI for all Bridging operation (”VLAN-VNI”)


2 0000.3001.1101 / 48 3001, 65500:3001 10.200.200.101

2 0000.3001.1102 / 48 3001, 65500:3001 10.200.200.104

BRKDCN-2949


Host Advertisements


Overlay




2 0000.3001.1101 / 48 3001, 65500:3001 10.200.200.101

Host AMAC: 0000.3001.1101

Host BMAC: 0000.3001.1102

Host CMAC: 0000.3002.2101

*L2VNI: VNI for all Bridging operation (”VLAN-VNI”)


2 0000.3001.1101 / 48 3001, 65500:3001 10.200.200.101

2 0000.3001.1102 / 48 3001, 65500:3001 10.200.200.104


2 0000.3001.1101 / 48 3001, 65500:3001 10.200.200.101

2 0000.3001.1102 / 48 3001, 65500:3001 10.200.200.104

2 0000.3002.2101 / 48 3002, 65500:3002 10.200.200.107

• Host MAC (Route Type 2)• MAC

• MPLS Label1 (L2VNI*)

• Route Target for MAC-VRF

• MAC attributes are Mandatory

BRKDCN-2949


V2# show bgp l2vpn evpn 0000.3001.1101

BGP routing table information for VRF default, address family L2VPN EVPN

Route Distinguisher: 10.10.10.101:32777

BGP routing table entry for [2]:[0]:[0]:[48]:[0000.3001.1101]:[0]:[0.0.0.0]/216,

version 4

Paths: (1 available, best #1)

Flags: (0x000202) on xmit-list, is not in l2rib/evpn, is locked

Advertised path-id 1

Path type: internal, path is valid, is best path, no labeled nexthop

AS-Path: NONE, path sourced internal to AS

10.200.200.101 (metric 3) from 10.10.10.201 (10.10.10.201)

Origin IGP, MED not set, localpref 100, weight 0

Received label 3001

Extcommunity: RT:65500:3001 ENCAP:8

Originator: 10.10.10.101 Cluster list: 10.10.10.201

Route Type:MAC/IP

Ethernet Segment

Identifier (ESI)

Ethernet Tag Identifier (Ethtag)

MAC Address Length

MAC Address

Next-Hop IP Address

L2VNI(MPLS Label1)

L2VNIRoute Target

Encap:8 VXLAN

BRKDCN-2949



Overlay


80

Host Advertisements

Baremetal

Host AMAC: 0000.3001.1101

IP: 192.168.10.101

Baremetal

Host BMAC: 0000.3001.1102

IP: 192.168.10.102

Baremetal

Host CMAC: 0000.3002.2101

IP: 192.168.20.101


2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 5000, 65500:5000 10.200.200.101


2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 5000, 65500:5000 10.200.200.101

2 0000.3001.1102 / 48 3001, 65500:3001 192.168.10.102 /32 5000, 65500:5000 10.200.200.104

*L3VNI: VNI for all Routing operation (”VRF-VNI”)BRKDCN-2949



Overlay


81

Host Advertisements

Baremetal

Host AMAC: 0000.3001.1101

IP: 192.168.10.101

Baremetal

Host BMAC: 0000.3001.1102

IP: 192.168.10.102

Baremetal

Host CMAC: 0000.3002.2101

IP: 192.168.20.101


2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 5000, 65500:5000 10.200.200.101


2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 5000, 65500:5000 10.200.200.101

2 0000.3001.1102 / 48 3001, 65500:3001 192.168.10.102 /32 5000, 65500:5000 10.200.200.104


2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 5000, 65500:5000 10.200.200.101

2 0000.3001.1102 / 48 3001, 65500:3001 192.168.10.102 /32 5000, 65500:5000 10.200.200.104

2 0000.3002.2101 / 48 3002, 65500:3002 192.168.20.101 /32 5000, 65500:5000 10.200.200.107

• Host MAC+IP (Route Type 2)• MAC and IP

• MPLS Label1 (L2VNI)

• Route Target for MAC-VRF

• MPLS Label2 (L3VNI*)

• Route Target for IP-VRF

• Router MAC

• IP Attributes are Optional

• Populated through ARP/ND



V2# show bgp l2vpn evpn 0000.3001.1101



BGP routing table entry for [2]:[0]:[0]:[48]:[0000.3001.1101]:[32]:[192.168.10.101]/272,

version 4






10.200.200.101 (metric 3) from 10.10.10.201 (10.10.10.201)


Received label 3001 5000

Extcommunity: RT:65500:3001 RT:65500:5000 ENCAP:8 Router MAC:0200.0ade.de01


Route Type:MAC/IP

Ethernet Segment

Identifier (ESI)


MAC Address Length

MAC Address

Next-Hop IP Address

L2VNI(MPLS Label1)

L2VNIRoute Target

Encap:8 VXLAN

Router MAC

IP Address Length IP Address

L3VNIRoute Target

L3VNI(MPLS Label2)

BRKDCN-2949



Overlay


83

Subnet Route Advertisements

10101011010101010101010

Subnet A192.168.10.0/24

Type IP / Length L3VNI / RT Next-Hop Seq.

5 192.168.10.0 /24 5000, 65500:5000 10.200.200.101

• Internal and External Subnet

Prefixes (Route Type 5)• IP Prefix

• MPLS Label (L3VNI)

• Route Target for IP-VRF

• Router MAC

• Populated through External

Routing Protocol

BRKDCN-2949



Overlay


84

Subnet Route Advertisements

10101011010101010101010

10101011010101010101010

Subnet A192.168.10.0/24

Subnet A192.168.10.0/24

Subnet B192.168.20.0/24


5 192.168.10.0 /24 5000, 65500:5000 10.200.200.101


5 192.168.10.0 /24 5000, 65500:5000 10.200.200.101

5 192.168.10.0 /24 5000, 65500:5000 10.200.200.104


5 192.168.10.0 /24 5000, 65500:5000 10.200.200.101

5 192.168.10.0 /24 5000, 65500:5000 10.200.200.104

5 192.168.20.0 /24 5000, 65500:5000 10.200.200.107

• IP Prefix Learning• via BGP with VRF-Lite

• via LISP on Nexus 7000/7700

• via other routing protocol (static

or dynamic)

• Default: Export of IP Host and IP

Prefix Routes advertisements• Filter and Summarize where

appropriate

BRKDCN-2949


V2# show bgp l2vpn evpn 192.168.10.0



BGP routing table entry for [5]:[0]:[0]:[24]:[192.168.10.101]/224,

version 4






10.200.200.101 (metric 3) from 10.10.10.201 (10.10.10.201)


Received label 5000

Extcommunity: RT:65500:5000 ENCAP:8 Router MAC:0200.0ade.de01


Route Type:IP Prefix

Ethernet Segment

Identifier (ESI)


IP Address Length IP Address

Next-Hop IP Address

L3VNI(MPLS Label)

L3VNIRoute Target

Encap:8 VXLAN Router MAC

BRKDCN-2949


Data-Plane (VXLAN)

86

Introducing VXLAN

*plus 4-byte if IEEE 802.1q exists as part of Inner MAC Header

Outer MAC Outer IP UDP VXLAN Inner MAC Inner IP Payload CRC

MAC 802.1q IP Payload CRC

Original Layer-2 Frame

20-byte + 8-byte +8-byte + 14-byte* = 50 Bytes

of total overhead

Src, Dst VTEP and Hop-by-

Hop MAC

Src and DstVTEP IP Address

UDP DstPort 4789

VXLAN VNI

UDP Src PortHash of L2/L3/L4

headers of original Frame

BRKDCN-2949


VXLAN Frame Format – MAC in IP Encapsulation

Outer MAC Outer IP UDP VXLAN Inner MAC Payload CRC

Field Value Bites Total

VXLAN Flags RRRRIRRR 8

8 B

yte

s

Reserved 24

VNI 16M Possible Segments 24

Reserved 8


Dest. MAC Address Next-Hop MAC Address 48

14 B

yte

s

(4 B

yte

s O

ptio

nal)

Src. MAC Address Next-Hop MAC Address 48

VLAN Type 0x8100 16

VLAN ID Tag 16

Ether Type 0x0800 16


IP Header Misc. Data 72

20 B

yte

sProtocol 0x11 (UDP) 8

Header Checksum Various 16

Source IP Src, VTEP IP 32

Destination IP Dest. VTEP IP 32


Source Port L2/L3/L4 Hash 16

8 B

yte

s

Destination Port 4789 (UDP) 16

UDP Length 16

Checksum 0x0000 16

BRKDCN-2949


No Path Diversity

Spine

Spine

Leaf Leaf

AS#65500BaremetalBaremetal

101010110101010

10101010

101010110101010

10101010

101010110101010

10101010

• Equal Cost Multi-Pathing (ECMP)

uses Header information to form

Path Diversity

• Some Tunnel Protocol provide no

diversity in IP or Protocol Header

• As a Result, all Packets travel

the same Path

• No Path Diversity or Entropy

BRKDCN-2949


VTEPVTEP

89

Introducing VXLAN – Entropy

Spine

Spine


101010110101010

10101010

101010110101010

10101010

101010110101010

10101010

101010110101010

10101010

• VXLAN provides variable UDP

Source Port in Outer Header

• Hash of the inner Layer-2/Layer-

3/Layer-4 Headers of the original

Ethernet Frame.

• Enables entropy for ECMP Load

balancing in the Network

BRKDCN-2949


VTEPVTEP

Spine

Spine

AS#65500

90

Introducing VXLAN – Entropy


Entropy

happens here

Data-Plane (VXLAN)BRKDCN-2949


Difference between VXLAN (F&L) and VXLAN (EVPN)?

F&L – Flood & Learn

• Data-Plane Encapsulation• Layer-2 MAC-in-IP Encapsulation

• Follows Ethernet Semantics –Learning through Flooding

• No Layer-3• First-Hop Gateway, Multi-Tenancy

• Uses Multicast for BUM• BUM – Broadcast, Unknown Unicast,

Multicast• Some static Ingress Replication (IR)

possible

• EVPN – BGP EVPN

• Control-Plane + Data-Plane Solution

• Layer-2 MAC-in-IP Encapsulation with Reachability Protocol

• Follows local/remote learning through Control-Plane• Learn Local, advertise to Remote

• Integrated Layer-2 and Layer-3• First-Hop Gateway, Multi-Tenancy

• Uses Multicast or Ingress Replication for BUM• Ingress Replication (IR) / Head-End

Replication (HER)


VXLAN and BGP EVPN – Putting it Together



2 0000.3001.1101/483001

65500:3001192.168.10.101/32

5000

65500:500010.200.200.101

Data-Plane (VXLAN)

Control-Plane (BGP EVPN)

Bridging

Dst VTEP IP

10.200.200.101

L2VNI

3001

Dst MAC

0000.3001.1101

Dst IP

192.168.10.101

BRKDCN-2949


VXLAN and BGP EVPN – Putting it Together



2 0000.3001.1101/483001

65500:3001192.168.10.101/32

5000

65500:500010.200.200.101

Data-Plane (VXLAN)

Control-Plane (BGP EVPN)

Routing

Dst VTEP IP

10.200.200.101

L3VNI

5000

Router MAC

0200.0ade.de01

Dst IP

192.168.10.101

Extended Community Router MAC

0200.0ade.de01

BRKDCN-2949


Routing and the Router MAC – Ethernet

Switch Switch

Baremetal

Host CMAC: 0000.3002.2101

IP: 192.168.20.101

Baremetal

Host AMAC: 0000.3001.1101

IP: 192.168.10.101

interface: Eth2/1

MAC: 0200.0ade.de01

IP: 10.200.200.1

interface: Eth2/1

MAC: 0200.0ade.de07

IP: 10.200.200.7

SVI10192.168.10.1

SVI20192.168.20.1

SMAC DMAC SIP DIPPayload

0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101


0000.3001.1101 2020:0000:AAAA 192.168.10.101 192.168.20.101


2020.0000AAAA 0000.3002.2101 192.168.10.101 192.168.20.101

Router MAC

BRKDCN-2949


Routing and the Router MAC – VXLAN

Baremetal

Host CMAC: 0000.3002.2101

IP: 192.168.20.101

Baremetal

Host AMAC: 0000.3001.1101

IP: 192.168.10.101

VXLANVTEP VTEP

SVI10192.168.10.1

SVI20192.168.20.1

interface: NVE1

MAC: 0200.0ade.de01

IP: 10.200.200.1

interface: NVE1

MAC: 0200.0ade.de07

IP: 10.200.200.7

SIP DIP VXLAN SMAC DMAC SIP DIPPayload

10.200.200.101 10.200.200.107 5000 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101


2020.0000AAAA 0000.3002.2101 192.168.10.101 192.168.20.101


0000.3001.1101 2020:0000:AAAA 192.168.10.101 192.168.20.101

Router MAC

BRKDCN-2949


Packet Walk – ARP Request


Overlay


Baremetal

Host AMAC: 0000.3001.1101

IP: 192.168.10.101

Baremetal

Host BMAC: 0000.3001.1102

IP: 192.168.10.102

Baremetal

Host CMAC: 0000.3002.2101

IP: 192.168.20.101

ARP Request for 192.168.10.102

SMAC:

0000.3001.1101

DMAC:

FFFF.FFFF.FFFF

SIP DIP VXLAN SMAC DMACARP Request for

192.168.10.10210.200.200.101 239.0.0.1 3001 0000.3001.1101 FFFF.FFFF.FFFF


SMAC:

0000.3001.1101

DMAC:

FFFF.FFFF.FFFF


2 0000.3001.1101 / 48 3001, 65500:3001 10.200.200.101

BRKDCN-2949


Packet Walk – ARP Response


Overlay


Baremetal

Host AMAC: 0000.3001.1101

IP: 192.168.10.101

Baremetal

Host BMAC: 0000.3001.1102

IP: 192.168.10.102

Baremetal

Host CMAC: 0000.3002.2101

IP: 192.168.20.101

SIP DIP VXLAN SMAC DMACARP Response for

192.168.10.10210.200.200.104 10.200.200.101 3001 0000.3001.1102 0000.3001.1101

ARP Response for 192.168.10.102

SMAC:

0000.3001.1102

DMAC:

0000.3001.1101


SMAC:

0000.3001.1102

DMAC:

0000.3001.1101


2 0000.3001.1101 / 48 3001, 65500:3001 10.200.200.101


2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101/32 5000, 65500:5000 10.200.200.101

BRKDCN-2949


Packet Walk – Bridging


Overlay


Baremetal

Host AMAC: 0000.3001.1101

IP: 192.168.10.101

Baremetal

Host BMAC: 0000.3001.1102

IP: 192.168.10.102

Baremetal

Host CMAC: 0000.3002.2101

IP: 192.168.20.101


2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101/32 5000, 65500:5000 10.200.200.101

2 0000.3001.1102 / 48 3001, 65500:3001 192.168.10.102/32 5000, 65500:5000 10.200.200.104


10.200.200.101 10.200.200.104 3001 0000.3001.1101 0000.3001.1102 192.168.10.101 192.168.10.102

SMAC DMAC SIP DIP

0000.3001.1101 0000.3001.1102 192.168.10.101 192.168.10.102

SMAC DMAC SIP DIP

0000.3001.1101 0000.3001.1102 192.168.10.101 192.168.10.102

BRKDCN-2949


Packet Walk – Routing


Overlay


Baremetal

Host AMAC: 0000.3001.1101

IP: 192.168.10.101

Baremetal

Host BMAC: 0000.3001.1102

IP: 192.168.10.102

Baremetal

Host CMAC: 0000.3002.2101

IP: 192.168.20.101


2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101/32 5000, 65500:5000 10.200.200.101

2 0000.3002.2102 / 48 3002, 65500:3002 192.168.20.101/32 5000, 65500:5000 10.200.200.107


10.200.200.101 10.200.200.107 5000 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101

SMAC DMAC SIP DIP

0000.3001.1101 2020.0000.AAAA 192.168.10.101 192.168.20.101

SMAC DMAC SIP DIP

2020.0000.AAAA 0000.3002.2101 192.168.10.101 192.168.20.101

Router MAC

BRKDCN-2949


Packet Walk – Routing (Silent Host)


Overlay


Baremetal

Host AMAC: 0000.3001.1101

IP: 192.168.10.101

Baremetal

Host BMAC: 0000.3001.1102

IP: 192.168.10.102

Baremetal

Host CMAC: 0000.3002.2101

IP: 192.168.20.101


2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 5000, 65500:5000 10.200.200.101

5 192.168.20.0/24 5000, 65500:5000 10.200.200.107


10.200.200.101 10.200.200.107 5000 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101

SMAC DMAC SIP DIP

0000.3001.1101 2020.0000.AAAA 192.168.10.101 192.168.20.101

BRKDCN-2949


Packet Walk – Routing (Silent Host)


Overlay


Baremetal

Host AMAC: 0000.3001.1101

IP: 192.168.10.101

Baremetal

Host BMAC: 0000.3001.1102

IP: 192.168.10.102

Baremetal

Host CMAC: 0000.3002.2101

IP: 192.168.20.101


2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 5000, 65500:5000 10.200.200.101

5 192.168.20.0/24 5000, 65500:5000 10.200.200.107


10.200.200.101 10.200.200.107 5000 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101

SMAC DMAC SIP DIP

0000.3001.1101 2020.0000.AAAA 192.168.10.101 192.168.20.101


SMAC:

2020.0000.AAAA

DMAC:

FFFF.FFFF.FFFF


SMAC:

0000.3002.2101

DMAC:

2020.0000.AAAA


2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 5000, 65500:5000 10.200.200.101

5 192.168.20.0/24 5000, 65500:5000 10.200.200.107

2 0000.3002.21o1 / 48 3002, 65500:3002 192.168.20.101 5000, 65500:5000 10.200.200.107

BRKDCN-2949


• Overlay

• Underlay


• Multi-Tenancy


• VPC


Agenda


What is Multi-Tenancy ?

• Segregation at Layer-2• VLAN

• Layer-2 VNI (L2VNI)

• VLAN Significance• Per-Fabric

• Per-Switch

• Per-Port

• Segregation at Layer-3• VRF

• Layer-3 VNI (L3VNI)

• VRF Significance• Per-Fabric

• Per-Switch

BRKDCN-2949


Layer-2 Multi-Tenancy – Bridge Domains

• Bridge Domain• Layer-2 Segment from End-Point

to End-Point

• Bridge Domains in VXLAN

consists of• The Ethernet Segment (VLAN)

between Host and Edge Device

• The Hardware Resources within

the Edge Device

• The VXLAN Segment (VNI)

between Edge Device and Edge

Device


Overlay



Host AVLAN 10

Host BVLAN 100

Host CVLAN 20

VNI 3001 (L2VNI)

VLAN 10VLAN 100

BRKDCN-2949


Layer-3 Multi-Tenancy – Routing Domains

• Routing Domain• Multiple Subnets sharing the

same Layer-3 forwarding policy

• Routing Domain in VXLAN

consists of• The Routing Domain local to the

Edge Device (VRF)

• The Routing Domain (VPN)

across the Edge Devices

• Multi-Protocol BGP with EVPN

Address-Family


Overlay



Host A192.168.10.101

Host B192.168.10.102

Host C192.168.20.101

VNI 5000 (L3VNI)

BRKDCN-2949


• Overlay

• Underlay


• Multi-Tenancy


• VPC


Agenda


Distributed IP Anycast Gateway

• Distributed First-Hop Routing on

Edge Device• All Edge Device share same

Gateway IP and MAC address

• Pervasive Gateway approach

• Gateway is always active• No redundancy protocol for hello

or state exchange

• Distributed and smaller state• Only local End-Points ARP

entries


Overlay


192.168.10.12020.0000.AAAA

192.168.20.12020.0000.AAAA

BRKDCN-2949


Distributed IP Anycast Gateway

• Distributed First-Hop Routing on

Edge Device• All Edge Device share same

Gateway IP and MAC address

• Pervasive Gateway approach

• Gateway is always active• No redundancy protocol for hello

or state exchange

• Distributed and smaller state• Only local End-Points ARP

entries


Overlay


192.168.10.12020.0000.AAAA

192.168.20.12020.0000.AAAA

BRKDCN-2949



Overlay


109

Anycast – One-to-Nearest Association

Baremetal

Host A

Baremetal

Host B

Baremetal

Host C

• Network Addressing and Routing

Methodology

• Datagrams sent from a single

Sender to the Topologically

Nearest Node

• Group of potential Receivers, all

identified by the same

Destination Address



Packet Walk – Symmetric IRB (A to C)


Overlay


Baremetal

Host AMAC: 0000.3001.1101

IP: 192.168.10.101

Baremetal

Host BMAC: 0000.3001.1102

IP: 192.168.10.102

Baremetal

Host CMAC: 0000.3002.2101

IP: 192.168.20.101


2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101/32 5000, 65500:5000 10.200.200.101

2 0000.3002.2102 / 48 3002, 65500:3002 192.168.20.101/32 5000, 65500:5000 10.200.200.107


10.200.200.101 10.200.200.107 5000 0200.0ade.de01 0200.0ade.de07 192.168.10.101 192.168.20.101

SMAC DMAC SIP DIP

0000.3001.1101 2020.0000.AAAA 192.168.10.101 192.168.20.101

SMAC DMAC SIP DIP

2020.0000.AAAA 0000.3002.2101 192.168.10.101 192.168.20.101

BRKDCN-2949


Packet Walk – Symmetric IRB (C to A)


Overlay


Baremetal

Host AMAC: 0000.3001.1101

IP: 192.168.10.101

Baremetal

Host BMAC: 0000.3001.1102

IP: 192.168.10.102

Baremetal

Host CMAC: 0000.3002.2101

IP: 192.168.20.101


2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101/32 5000, 65500:5000 10.200.200.101

2 0000.3002.2102 / 48 3002, 65500:3002 192.168.20.101/32 5000, 65500:5000 10.200.200.107


10.200.200.107 10.200.200.101 5000 0200.0ade.de07 0200.0ade.de01 192.168.20.101 192.168.10.101

SMAC DMAC SIP DIP

2020.0000.AAAA 0000.3001.1101 192.168.20.101 192.168.10.101

SMAC DMAC SIP DIP

0000.3002.2101 2020.0000.AAAA 192.168.20.101 192.168.10.101

BRKDCN-2949


• Overlay

• Underlay


• Multi-Tenancy


• VPC


Agenda


VPC Gateway Redundancy

• VPC – Virtual Port-Channel• Multi-Chassis Link Aggregation

• Layer-2 Multihoming

• Extended for VXLAN

• Host-side• Dual-Connect Hosts

• Using Port-Channels

• Fabric-side• Individual VTEPs

• Using a common Anycast VTEP

• Seen as one VTEP from remote

Nodes


Overlay

Leaf Leaf Leaf

VPC

VTEP VTEP

VPC

VTEP VTEP


BRKDCN-2949


VPC Gateway Redundancy – A VXLAN perspective

Overlay

VPC

VTEP VTEP

Baremetal

Individual Node with unique

Identity

Individual Node with unique

Identity

Both sharing an

Anycast VTEP

BRKDCN-2949



Overlay

VPC

VTEP VTEP

Baremetal

interface loopback0

description RID

ip address 10.10.10.103/32

interface loopback1

description VTEP

ip address 10.200.200.103/32

ip address 10.200.200.123/32 secondary

interface loopback0

description RID

ip address 10.10.10.102/32

interface loopback1

description VTEP

ip address 10.200.200.102/32

ip address 10.200.200.123/32 secondary

Anycast VTEP

IP Address

Anycast VTEP

IP Address

BRKDCN-2949


Host Advertisements with VPC


Overlay

Leaf Leaf Leaf

VPC

VTEP VTEP

VPC

VTEP VTEP


Host AMAC: 0000.3001.1101

IP: 192.168.10.101


2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 5000, 65500:5000 10.200.200.123


2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 5000, 65500:5000 10.200.200.123

2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 5000, 65500:5000 10.200.200.123

Host BMAC: 0000.3001.1102

IP: 192.168.10.102

Host CMAC: 0000.3002.2101

IP: 192.168.20.101

BRKDCN-2949


Host Advertisements with VPC


Overlay

Leaf Leaf Leaf

VPC

VTEP VTEP

VPC

VTEP VTEP


Host AMAC: 0000.3001.1101

IP: 192.168.10.101


2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 5000, 65500:5000 10.200.200.123


2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 5000, 65500:5000 10.200.200.123

2 0000.3001.1101 / 48 3001, 65500:3001 192.168.10.101 /32 5000, 65500:5000 10.200.200.123

• Independent Devices in the EVPN

Control-Plane• Individual Router and Peering

• Unique Route Distinguisher (RD)

• Independent Underlay Routing

Devices

• Common VXLAN Device• Next-Hop is Anycast VTEP

• Underlay ECMP Load Share to

Anycast VTEPHost B

MAC: 0000.3001.1102IP: 192.168.10.102

Host CMAC: 0000.3002.2101

IP: 192.168.20.101

BRKDCN-2949


VTEP

VTEP

VP

C

VTEP

118

ECMP to the Anycast VTEP – Underlay

Spine

Spine


101010110101010

10101010

101010110101010

10101010

101010110101010

10101010

101010110101010

10101010

Host AMAC: 0000.3001.1101

IP: 192.168.10.101

Host BMAC: 0000.3001.1102

IP: 192.168.10.102

BRKDCN-2949


VTEP

VTEP

VP

C

VTEP

119

Bridging to a VPC Domain – VXLAN

Spine

Spine



10.200.200.104 10.200.200.123 3001 0000.3001.1102 0000.3001.1101 192.168.10.102 192.168.10.101

Host AMAC: 0000.3001.1101

IP: 192.168.10.101

Host BMAC: 0000.3001.1102

IP: 192.168.10.102

BRKDCN-2949


Baremetal

VTEP

VTEP

VP

C

VTEP

120

Routing to a VPC Domain – VXLAN

Spine

Spine

AS#65500Baremetal


10.200.200.107 10.200.200.123 5000 0200.0ade.de01 2020.2323.2323 192.168.20.101 192.168.10.101

Host AMAC: 0000.3001.1101

IP: 192.168.10.101

Host CMAC: 0000.3002.2101

IP: 192.168.20.101

Local Station or

Virtual MAC

BRKDCN-2949



• VPC provides Layer-2 Gateway

Redundancy

• From the VXLAN perspective,

the next-hop is always the

Anycast VTEP (VIP)• Optimal for direct attached Hosts

• 1:1 Multicast-to-VNI mapping

• VPC operates at Layer-2• MAC is Synchronized

• Local IP (ARP) is Synchronized

• Routing Tables are not

Synchronized

BRKDCN-2949


Subnet Route Advertisement with VPC


Overlay

Leaf Leaf Leaf

VPC

VTEP VTEP

VPC

VTEP VTEP

Subnet Y192.168.22.0/24

Subnet X192.168.11.0/24


5 192.168.11.0 /24 5000, 65500:5000 10.200.200.123

5 192.168.22.0 /24 5000, 65500:5000 10.200.200.123

• Subnet Route Advertisement • Route Type 5

• Next-Hop is Anycast VTEP

• Ensure Sync of Subnet• Dual-Connect Networks (Point-2-

Point not Layer-3 over VPC)

• Synchronize Routing Table

• Advertise Route Type 5 with

individual VTEP IP (PIP)Baremetal

Host BMAC: 0000.3001.1102

IP: 192.168.10.102

BRKDCN-2949


Baremetal

VTEP

VTEP

VP

C

VTEP

123

Subnet Route Advertisement with VPC

Spine

Spine

AS#65500


10.200.200.107 10.200.200.123 5000 0200.0ade.de01 2020.2323.2323 192.168.20.101 192.168.11.101

Host CMAC: 0000.3002.2101

IP: 192.168.20.101

Subnet X192.168.11.0/24 101010110101010

10101010

101010110101010

10101010

BRKDCN-2949


Baremetal

VTEP

VTEP

VP

C

VTEP

124

VPC – Dual-Attach Networks

Spine

Spine

AS#65500


10.200.200.107 10.200.200.123 5000 0200.0ade.de01 2020.2323.2323 192.168.20.101 192.168.11.101

Host CMAC: 0000.3002.2101

IP: 192.168.20.101

Subnet X192.168.11.0/24 101010110101010

10101010

101010110101010

10101010

Layer-3 Point-2-Point(not Layer-3 over

VPC!)

BRKDCN-2949


Baremetal

VTEP

VTEP

VP

C

VTEP

125

VPC – Synchronizing the Routing

Spine

Spine

AS#65500


10.200.200.107 10.200.200.123 5000 0200.0ade.de01 2020.2323.2323 192.168.20.101 192.168.11.101

Host CMAC: 0000.3002.2101

IP: 192.168.20.101

Subnet X192.168.11.0/24 101010110101010

10101010

101010110101010

10101010

Dedicated Routing Session (per-VRF)

BRKDCN-2949


Baremetal

VTEP

VTEP

VP

C

VTEP

126

VPC – Advertise Subnet Individually (Advertise-PIP)

Spine

Spine

AS#65500


10.200.200.107 10.200.200.102 5000 0200.0ade.de07 0200.0ade.de02 192.168.20.101 192.168.11.101

Host CMAC: 0000.3002.2101

IP: 192.168.20.101

Subnet X192.168.11.0/24 101010110101010

10101010

101010110101010

10101010


5 192.168.11.0 /24 5000, 65500:5000 10.200.200.102

BRKDCN-2949


Baremetal

VTEP

VTEP

VP

C

VTEP

127

VPC – Advertise Subnet Individually (Advertise-PIP)

Spine

Spine

AS#65500

SIP DIP VXLAN SMAC DMAC SIP DIP

Payload10.200.200.107 10.200.200.102 5000 0200.0ade.de07 0200.0ade.de02 192.168.20.101 192.168.11.101

10.200.200.107 10.200.200.103 5000 0200.0ade.de07 0200.0ade.de03 192.168.20.101 192.168.11.101

Host CMAC: 0000.3002.2101

IP: 192.168.20.101

Subnet X192.168.11.0/24 101010110101010

10101010

101010110101010

10101010


5 192.168.11.0 /24 5000, 65500:5000 10.200.200.102


5 192.168.11.0 /24 5000, 65500:5000 10.200.200.102

5 192.168.11.0/24 5000, 65500:5000 10.200.200.103

BRKDCN-2949


• Overlay

• Underlay


• Multi-Tenancy


• VPC


Agenda

A Deployment Story


Scalable Data Center Fabric

• VXLAN based Data Center Fabric

• BGP EVPN Control-Protocol (Overlay)

• OSPF for Underlay Routing (Unicast)

• PIM ASM with Anycast-RP for BUM Replication (Underlay)

• Distributed IP Anycast Gateway

130BRKDCN-2949


A Deployment Story

Underlay






A Deployment Story – Underlay Routing

Underlay




interface loopback0

ip address 10.10.10.201/32

ip router ospf UNDERLAY area 0.0.0.0

router ospf UNDERLAY

router-id 10.10.10.201

interface Ethernet1/1

mtu 9192


ip ospf network point-to-point


ip pim sparse-mode


mtu 9192




ip pim sparse-mode


mtu 9192

ip address 10.1.1.10/30



ip pim sparse-mode

…

interface loopback0

ip address 10.10.10.101/32



router-id 10.10.10.101


mtu 9192




ip pim sparse-mode

…




Underlay




interface loopback0

ip address 10.10.10.201/32



router-id 10.10.10.201


mtu 9192




ip pim sparse-mode


mtu 9192




ip pim sparse-mode


mtu 9192

ip address 10.1.1.10/30



ip pim sparse-mode

…

interface loopback0

ip address 10.10.10.101/32



router-id 10.10.10.101


mtu 9192




ip pim sparse-mode

…

interface loopback0

ip address 10.10.10.102/32



router-id 10.10.10.102


mtu 9192




ip pim sparse-mode

…




Underlay




interface loopback0

ip address 10.10.10.201/32



router-id 10.10.10.201


mtu 9192




ip pim sparse-mode


mtu 9192




ip pim sparse-mode


mtu 9192

ip address 10.1.1.10/30



ip pim sparse-mode

…

interface loopback0

ip address 10.10.10.101/32



router-id 10.10.10.101


mtu 9192




ip pim sparse-mode

…

interface loopback0

ip address 10.10.10.102/32



router-id 10.10.10.102


mtu 9192




ip pim sparse-mode

…

interface loopback0

ip address 10.10.10.103/32



router-id 10.10.10.103


mtu 9192




ip pim sparse-mode

…




Underlay




interface loopback0

ip address 10.10.10.202/32



ip address 10.254.254.1/32


ip pim sparse-mode

ip pim anycast-rp 10.254.254.1 10.254.254.202

ip pim anycast-rp 10.254.254.1 10.254.254.203

ip pim rp-address 10.254.254.1

interface loopback0

ip address 10.10.10.203/32



ip address 10.254.254.1/32


ip pim sparse-mode

ip pim anycast-rp 10.254.254.1 10.254.254.202

ip pim anycast-rp 10.254.254.1 10.254.254.203


interface loopback0

ip address 10.10.10.101/32






Underlay




interface loopback0

ip address 10.10.10.202/32



ip address 10.254.254.1/32


ip pim sparse-mode

ip pim anycast-rp 10.254.254.1 10.254.254.202

ip pim anycast-rp 10.254.254.1 10.254.254.203


interface loopback0

ip address 10.10.10.203/32



ip address 10.254.254.1/32


ip pim sparse-mode

ip pim anycast-rp 10.254.254.1 10.254.254.202

ip pim anycast-rp 10.254.254.1 10.254.254.203


interface loopback0

ip address 10.10.10.101/32



interface loopback0

ip address 10.10.10.102/32






Underlay




interface loopback0

ip address 10.10.10.202/32



ip address 10.254.254.1/32


ip pim sparse-mode

ip pim anycast-rp 10.254.254.1 10.254.254.202

ip pim anycast-rp 10.254.254.1 10.254.254.203


interface loopback0

ip address 10.10.10.203/32



ip address 10.254.254.1/32


ip pim sparse-mode

ip pim anycast-rp 10.254.254.1 10.254.254.202

ip pim anycast-rp 10.254.254.1 10.254.254.203


interface loopback0

ip address 10.10.10.101/32



interface loopback0

ip address 10.10.10.102/32



interface loopback0

ip address 10.10.10.103/32





A Deployment Story – the VTEP

Underlay




interface loopback1

ip address 10.200.200.101/32


interface nve1

source-interface loopback1

host-reachability protocol bgp




Underlay




interface loopback1

ip address 10.200.200.101/32


interface nve1



interface loopback1

ip address 10.200.200.102/32


interface nve1






Underlay




interface loopback1

ip address 10.200.200.101/32


interface nve1



interface loopback1

ip address 10.200.200.102/32


interface nve1



interface loopback1

ip address 10.200.200.103/32


interface nve1





A Deployment Story – Overlay Control-Plane

Underlay




router bgp 65500

router-id 10.10.10.202

neighbor 10.10.10.0/24 remote-as 65500

update-source loopback0

address-family l2vpn evpn

send-community both

route-reflector-client

router bgp 65500

router-id 10.10.10.203




send-community both


router bgp 65500

router-id 10.10.10.101

neighbor 10.10.10.202 remote-as 65500



send-community both




send-community both




Underlay




router bgp 65500

router-id 10.10.10.202




send-community both


router bgp 65500

router-id 10.10.10.203




send-community both


router bgp 65500

router-id 10.10.10.101




send-community both




send-community both

router bgp 65500

router-id 10.10.10.102




send-community both




send-community both




Underlay




router bgp 65500

router-id 10.10.10.202




send-community both


router bgp 65500

router-id 10.10.10.203




send-community both


router bgp 65500

router-id 10.10.10.101




send-community both




send-community both

router bgp 65500

router-id 10.10.10.102




send-community both




send-community both

router bgp 65500

router-id 10.10.10.103




send-community both




send-community both



A Deployment Story – Layer-2 Service



Overlay


vlan 100

vn-segment 30001

name Blue

vlan 200

vn-segment 30002

name Green

evpn

vni 30001

rd auto

route-target both auto

vni 30002

rd auto


interface nve1



member vni 30001

mcast-group 239.239.239.1

member vni 30002

mcast-group 239.239.239.2



A Deployment Story – Layer-3 Service



Overlay


vrf context VRF-A

vni 50001

rd auto



route-target both auto evpn



route-target both auto evpn

interface nve1



member vni 50001 associate-vrf

router bgp 65500

vrf VRF-A


advertise l2vpn evpn

redistribute direct route-map TAG

vlan 2001

vn-segment 50001

interface Vlan2001

mtu 9192

vrf member VRF-A

ip forward

no ip redirects



A Deployment Story – First-Hop Gateway



Overlay


router bgp 65500

vrf VRF-A



redistribute direct route-map TAG

interface Vlan100

mtu 9192

vrf member VRF-A

ip address 192.168.1.1/24 tag 21921

fabric forwarding mode anycast-gateway

interface Vlan200

mtu 9192

vrf member VRF-A

ip address 10.10.10.1/24 tag 21921

fabric forwarding mode anycast-gateway

route-map TAG permit 10

match tag 21921



A Deployment Story


HypervisorHypervisor HypervisorHypervisor BaremetalBaremetalBaremetal

Overlay


Subnet B192.168.20.0/24

interface Ethernet 2/1.10

vrf member VRF-A

ip address 172.16.0.1/30

encapsulation dot1q 5


vrf member VRF-B

ip address 172.16.0.1/30


router bgp 65500

vrf VRF-A



aggregate-address 10.10.10.0/24 summary-only

aggregate-address 192.168.1.0/24 summary-only


update-source Ethernet2/1.10


…


vrf member VRF-A

ip address 172.16.0.2/30



vrf member VRF-B

ip address 172.16.0.2/30


router bgp 65599

vrf VRF-A



update-source Ethernet1/15.21


…


Summary


Summary

• Multi-Tier Topologies based on Leaf and Spines (aka Clos)

• New paradigm with Hierarchical Overlays

• Overlays (VXLAN) for Network Virtualization

• Different flavors of Overlay Solution (Flood&Learn and BGP EVPN)

• Layer-3 in the Underlay – Defines the Topology

• Layer-2 and Layer-3 in the Overlay – Defines the Services

• End-Points State exists in the Overlay

• BGP EVPN for integrated Layer-2 and Layer-3 Services

• Control-Plane driven

• Optimal Routing and Bridging

• Avoid hair pinging and reduced failure domains

BRKDCN-2949


If you haven’t had enough VXLAN BGP EVPN

150BRKDCN-2949


Links & Resources

• VXLAN Multi-Site Intro

• https://blogs.cisco.com/datacenter/vxlan-innovations-vxlan-evpn-multi-site-part-2-of-2

• VXLAN Multi-Site @ Cisco Live online

• https://www.ciscolive.com/global/on-demand-library/?search=BRKDCN-2035#/

• ”eBGP” for EVPN

• https://learningnetwork.cisco.com/blogs/community_cafe/2017/11/02/vxlan-ebgp-evpn-the-incarnation-of-a-hybrid-guest-post

• Configuration Example

• https://communities.cisco.com/community/technology/datacenter/data-center-networking/blog/2015/05/19/vxlanevpn-configuration-example

151BRKDCN-2949

https://blogs.cisco.com/datacenter/vxlan-innovations-vxlan-evpn-multi-site-part-2-of-2

https://www.ciscolive.com/global/on-demand-library/?search=BRKDCN-2035#/

https://learningnetwork.cisco.com/blogs/community_cafe/2017/11/02/vxlan-ebgp-evpn-the-incarnation-of-a-hybrid-guest-post

https://communities.cisco.com/community/technology/datacenter/data-center-networking/blog/2015/05/19/vxlanevpn-configuration-example


Cisco Spark

Questions? Use Cisco Spark to communicate with the speaker after the session

1. Find this session in the Cisco Live Mobile App

2. Click “Join the Discussion”

3. Install Spark or go directly to the space

4. Enter messages/questions in the space

How

cs.co/ciscolivebot#BRKDCN-2949


• Please complete your Online Session Evaluations after each session

• Complete 4 Session Evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt

• All surveys can be completed via the Cisco Live Mobile App or the Communication Stations

Don’t forget: Cisco Live sessions will be available for viewing on-demand after the event at www.ciscolive.com/global/on-demand-library/.

Complete Your Online Session Evaluation

http://ciscolive.com/Online

http://www.ciscolive.com/global/on-demand-library/


Continue Your Education

• Demos in the Cisco campus

• Walk-in Self-Paced Labs

• Tech Circle

• Meet the Engineer 1:1 meetings

• Related sessions

154BRKDCT-2949

Thank you

Date post:	01-Aug-2018
Category:	Documents
Upload:	dinhtu
View:	240 times
Download:	5 times

Building Data Center Networks with VXLAN EVPN … · • Introduction to Data Center Fabric ......

Documents