W H I T E P A P E R

SD-WAN for Telcos / Service Providers:

Topology and Routing

ServiceProviderTopology

ServiceProviderDeploymentModel(Provider-hostedorOn-premisesDeployment)Inaserviceproviderenvironment,theVMwareSD-WANGatewayby VeloCloud isabletobe deployedasapartnergateway,suchthatthepartnergatewayisconnectingto boththeinternetandserviceprovider'sMPLSbackbone.Figure8.1showsatypical serviceproviderarchitecture.

Figure8.1-VMwareSD-WANoverallarchitecturefortheserviceprovider-hostedoffering

One of the main advantages of the service provider-hosted model is in thefunction of the gateway itself. This partner gateway functionality provides anadvantagetotheserviceproviderinhostingthegatewaythemselves.

PartnerGatewayMulti-tenantVMwareSD-WANGatewaysaredeployedatserviceproviderpoints-of-presence (PoP) to provide secure handoffs segmented by customer, andsegmentedwithinthecustomertotheapplicationsornetworks.

There isnodifference in softwarebetweenaVMwareSD-WANPartnerGateway andaVMwarehostedSD-WANCloudGateway.Instead,thedifferencecomeswith the allocation of the SD-WAN Partner Gateway role to the gateway itself,which provides additional functionality on top of the standard cloud-hosted gateway configuration.

This functionality allows for handing off the traffic in a per-enterprise and per-segmentfashiontothesecondinterfaceallocatedonthegatewaytowardsthe PE/DCrouter.Thishandoffiscoveredinmoredetaillater in this paper, in the Routing section.

Figure8.2:SD-WANpartnergateway

Thisfunctionalityallowsfordifferentiatedandsecureservicedeliveryviathesamepartnergateway,allowingforrapiddeploymentandscalingoftheinfrastructure.

TheotherdefiningcharacteristicofthepartnergatewayisthattheallocationtoanSD-WANedgeisdeterministic.Thisallocationcanbeperformedattheprofilelevelor at an edge level override. This deterministic assignment is a critical factor inhelpingtoensurethatthepropergatewaysareassignedtoSD-WANedges,basedontheassociatedservicesbeingprovidedfromthatgatewaytotheendcustomer.

MPLSIntegrationDeploymentUseCaseforaServiceProvider

VMware SD-WAN adapts to the service provider network architecture toaccommodatetheshiftofapplicationstothecloudandto increasetheavailablebandwidth at a lower cost while maintaining the expected levels of reliability,performance,andsecurity.ByofferingVMware SD-WANasaservice, theserviceprovider can protect its MPLS network investment by augmenting it with other

connection types. In addition, the service provider can add value for theircustomers by providing a more agile and better-performing network. This alsoallowsaserviceprovidertoextendbeyonditstypicalnetworkfootprintandreachabroadercustomerbase.

ProviderServicesviatheOverlay

Thepartnergatewayprovidesawaytogiveaccesstootherserviceofferingsthatthe service providerhas as part of their current product set, such as voice,PCIenvironments,cloudfirewall,andpublicIPbackhaul.Eachoftheseofferingswouldbefront-endedbyapartnergatewaysittingat theassociatedservicecenterandadvertising reachability of those services to any edges attached to that partnergateway. The prescribed partner gateway deployment would follow the sameredundancystrategy,withtheidealscenarioallocatinggeo-redundantgatewaystomitigateanyoutagesthatmayimpactanentireservicecenter.Inscenarioswheregeo-redundancy is not possible or practical, redundancy in the same servicecenterisstillrecommended,withpartnergatewaysdeployedondiscretecomputestacks.Figure8.3highlightsatripleplayserviceusecasewherethesamegatewayisofferingaccesstoan individualcustomerwithVRFseparatedhandoffstoeachserviceoffering.

Figure8.3:AccessingproviderservicesviatheSD-WANpartnergateway

InboundPublicIPbackhaul

Thereisalsotheusecasewhereanenterprisewishestohostaninboundserviceavailable to outside users such as a website or VPN connectivity in a highlyavailable fashion. Insuchusecases, theserviceprovidercanallocateapublic IPfrom their pool to be placed on the LAN side of the edge and provided at thecustomer site. Inbound requests for this IPwill hit the service provider internetedge, be routed through their core to the PE for the appropriate gateway andforwardedonto theedgeand theclientasdepicted inFigure8.4.This topologyallowstheservicetoremainavailableregardlessofwhattransportisconnectedtotheedge,evenwhenitisnotownedbytheserviceproviderofferingthepublicIP.

AfewkeypointsregardingthisinboundpublicIPbackhaul:

• ThecustomerhastheirownpublicIPblockassignedbySP,doesnotwantNAT

• Multiple links provide aggregation, resiliency while retaining customer IPblock

• SPoptionallyprovidessecurityforserviceforinternettraffic

• AlltrafficisforcedtogothroughSPfirst

• IfpublicIPisnotarequirement,canoptionallyassignprivateblock.

Figure8.4illustratesthetopology.

Figure8.4:InboundpublicIPaddressbackhaul

SharedServicesviaPolicy-basedNAT

There are timeswherea service providermay offer a common service tomanycustomers that is not natively VRF-aware. In those cases, the possibility ofoverlapping subnets between customers exists, and it becomes necessary toperformapolicy-basedNATtofacilitateserviceofferingswithouthavingtoworryabout address collision. The policy NAT is performed on the partner gatewayhandoffinterfacedirectlypriortobeingsentontoitsdestinationviathePE.Thisallowsthesharedresourcetoreturnthetrafficbacktothegatewaythatoriginatesthetrafficwhichwillthenbeforwardedtotheappropriatecustomerandedge.Intheexample following, a source policy NAT could be defined to allow for trafficcoming from Customer A - 10.0.0.0/24 to translate to 1.1.1.1/32 and fromCustomerB-10.0.0.0/24totranslateto2.2.2.2/32.

Figure8.5:Sharedserviceviapolicy-basedNAT

Mid-mileTransport

Manyserviceprovidersoperateextensive inter-regionnetworks thathave tightlycontrolledSLAs.Incustomerdeploymentsthatspanmultipleregions,theserviceprovider can selectively route this traffic via region-specific partner gateways tothen be carried across the mid-mile transport towards partner gateways inanother region. This helps to provide a path to mitigate potential long-haulconnectivity issues thatmightotherwisebepresent in a traditional internet-onlyconnectivityscenario.Figure8.6showsahigh-levelviewofthisusecase:

Figure8.6:Mid-miletransportwithVMwareSD-WANpartnergateway

Thisscenarioisalsooftencalledlast-milereplacement.TheVMwareSD-WANEdgeisabletousetheinternetasthelastmiletoreachthepartnergatewayasanentrypointtotheserviceprovider'sMPLSbackbone,whichmakestheprivatelocalloopnolongermandatoryattheSD-WANEdge.

KeyTakeaways:

• Fundamentally, it is the same software for SD-WAN PartnerGatewayand VMware SD-WAN Gateway. The difference comes from the roleeachplaysinthenetwork.

• PartnerGateway is hosted on thepremises of the serviceprovider orenterprise customer. It offers secure hand-off of traffic on a per-enterpriseorper-segmentbasis toPE/DC router,andallowsa serviceprovidertoofferthefollowingdifferentiatedservices:

- ProviderservicesviaSD-WANOverlay- Inboundinternetbackhaul

- Sharedservicesviapolicy-basedNAT- Mid-miletransport

ServiceProviderRouting

PartnerGatewayAssignment

In the Over The Top model, VMware SD-WAN Gateway assignment to edges isperformedusinggeo-location.Intheserviceproviderdeploymentmodel,partnergatewaysarestaticallyassignedbytheserviceprovider.Generally,aprimaryandsecondarypartnergatewayinthesameregionwillbeassignedtotheVMwareSD-WANEdge.Thestaticassignmentofpartnergatewaysallowstheserviceprovidernetworkengineer to control trafficflowbetweenVMwareSD-WANEdgesand toproviderservices.

TrafficPathwithMid-mileTransportandPartnerGateway

ItisimportanttounderstandthetrafficpathtakenbythetrafficfromoneVMwareSD-WAN Edge to the other VMware SD-WAN Edge when the traffic is passingthroughthepartnergatewayandMPLSbackbone.

InFigure8.7,theVMwareSD-WANEdgeandassociatedpartnergatewaysontheleft-handsidearelocatedinregion1.TheVMwareSD-WANEdgeandassociatedpartnergatewaysontheright-handsidearelocatedinregion2.

When a client on the LAN side of VMware SD-WAN Edge11 sends packets toVMwareSD-WANEdge21,itdoessoin3steps.

• First, the packets willbe sent from VMware SD-WAN Edge11 to partnergatewayGW11overtheSD-WANOverlay.

• Second,thepacketswillberoutedfromGW11 intheMPLSbackboneviaPE11toreachpartnergatewayGW21viaPE21.

• Third,thepacketswillbesentfrompartnergatewayGW21totheVMwareSD-WANEdge21overtheSD-WANOverlay.

Oneimportantpointtonotehereistheregion1andregion2partnergatewaysdonotformanySD-WANOverlaybetweeneachother.ThisreachabilityisachievedbyBGProutingonthebackbone.

Figure8.7:TrafficpathbetweentwoVMwareSD-WANEdgeslocatedintwodifferentregions

Aspreviously mentioned, the gateway ismulti-tenanted and able to connect tomultiple VRFs with the PE. Each VRF is identified by either a single VLAN tag(802.1q)ordoubleVLANtag(QinQ).Optionally,notagcanbeappliedforusecaseswhereanexplicitpercustomerVRFhandoffisnotneeded.Thisisahighlyscalablearchitecture, such that capacity can be horizontally scaled out by addingmoregatewayswhennecessary.

TrafficPathbetweenSD-WANandTraditionalBranches

ThearchitecturealsoallowscompatibilitywithtraditionalsitesthatonlyhaveMPLSconnections. Figure 8.8 illustrates the conditionwhen there is traditional routerconnectedtothecustomerVRF.

Inthesamplescenario,abranchwithaCErouterCE21isconnectedtotheserviceproviderMPLSbackbone.RouterCE21 isadvertisingaprefixL1whichistheLANsidesubnet.PrefixL1isadvertisedviaBGPtothePEsinthecustomerVRF.DuetothepartnergatewaysusingBGPtopeerwiththePErouters,theyalsolearnL1andcanadvertiseittotheconnectedVMwareSD-WANEdges.AsaresulteachVMwareSD-WANEdgeisabletoreachthetraditionalMPLSsiteviatheirpartnergateways.Thisisusefulduringthemigrationorinsomesituationswhereaparticularsite isnotabletomigratetoSD-WAN.

Figure8.8:UsingSD-WANpartnergatewaytoconnectwithatraditionalMPLSsite

TrafficPathbetweenVMwareSD-WANEdgeswithCommonPartnerGateway

Thedecision onwhether the traffic is directbetween the two VMwareSD-WANEdges or if traffic has to go through the common gateway, will depend on theCloudVPN configuration which was referenced earlier in Chapter 4: OverlayTopologies. If there is at least one common partner gateway assigned for twoVMwareSD-WANEdges,thetrafficbetweenthesetwoVMwareSD-WANEdgeswillnotenter theMPLSbackbone. Instead the trafficflowsdirectlybetween the twoVMwareSD-WANEdgesorviathepartnergatewaysusingtheSD-WANoverlay.

• Dynamic Branch to Branch VPN is enabled:", that is become "DynamicBranch to Branch VPN is enabled: It will flow between VMware SD-WANEdge11toVMwareSD-WANEdge12viadirectSD-WANoverlay. ItwillflowbetweenVMwareSD-WANEdge11 toVMwareSD-WANEdge12 viadirectSD-WANoverlay.

• Dynamic Branch to Branch VPN is disabled:" that is become "DynamicBranch to Branch VPN is disabled: It will flow from a VMware SD-WANEdge11topartnergatewayGW11thentoVMwareSD-WANEdge12.ItwillflowfromaVMwareSD-WANEdge11partnergatewayGW11toaVMwareSD-WANEdge12.

Figure 8.9demonstrates how the traffic flow is affected by the partnergatewayassignment:

Figure8.9:Differentpartnergatewayassignmentscenario

SD-WANOverlayovertheMPLSNetwork

Another advantageof thepartner gateway is that a VMware SD-WANEdgewithbothinternetconnectivityandMPLSconnectivityfromtheserviceproviderhostingthegatewaycanhaveVCMPtunnelsterminateonboththepublicandprivatesideofthegateway.Figure8.10illustratesthisscenario.

In this sample figure, Edge11 has both internet and private MPLS underlayconnectivity. Edge11 forms a VCMP tunnel with GW11 via both underlayconnections.Thisintroducessomeadditionalbenefitsasfollows:

• Traffic between the VMware SD-WAN Edge and partner gateway canbenefitfromDMPOusingboththeinternetandprivateMPLScircuits.

• IntheeventthattheVMwareSD-WANEdgelosesinternetconnectivity,theVMware SD-WAN Edge is still manageable on the VMware SD-WANOrchestrator.ThemanagementplanetrafficisabletogothroughtheSD-WAN overlay on the private MPLS side to reach the VMware SD-WANOrchestratorthroughthepartnergateway.

Figure8.10:VMwareSD-WANEdgeformsSD-WANOverlaywithSD-WANpartnergatewayontheprivateMPLS

VMwareSD-WANPartnerGatewayOrder

When assigning the partner gateway to the VMware SD-WAN Edge, themanualassignmentrequiresanordernumbertobeselected.ThescreencaptureinFigure

8.11showsGW11withanordernumber1,whileGW12hasanordernumber2.Thesmallerthenumber,thehigherpriorityitis.

Thereareafewpointstohighlightforthemeaningoftheordernumber:

• WhentheVMwareSD-WANEdgelearnsprefixesfromthepartnergateway,the original BGP attributes such as AS-PATH metrics are retained. TheVMwareSD-WANEdgewillselectthepreferredpartnergatewaybasedontheprefixattribute.Iftheprefixattributeisidentical,theordernumberwillbe the tiebreaker, that means the partner gateway with order 1 will beused.

• When the partner gateway is as shown Figure 8.9, the VMware SD-WANEdgewhich initiates the trafficwill always send the traffic to thepartnergatewaywithorder1 for itselfunlesstheSD-WANoverlaytothatpartnergatewayisdown.

• ThepartnergatewayadvertisesprefixeslearnedfromtheVMwareSD-WANEdgestothePErouter.Duringthisrouteadvertisement,theordernumberbecomes the metric. This means that partner gateway with order 1willadvertisetheprefixwithmetric1tothePE.Partnergatewaywithorder2willadvertise theroutewithmetric2 to thePE.This is trying to influencethePErouterssuchthatthepartnergatewaywithorder1isthepreferredpartnergatewaytoreachthecorrespondingprefix.

Navigateto Configure > Profile > {profile name} > Device tab >Edit Gateway Handoff Assignment

Figure8.11:VMwareSD-WANpartnergatewayordernumber

T E C H T I P

ItisnotrecommendedtoassignallpartnergatewaystoeveryVMwareSD-WAN

Edgeavailableinanenterprise.TheSD-WANOverlayfromtheVMwareSD-

WANEdgetothepartnergatewayispersistent.Assigninganexcessivenumber

ofpartnergatewaystotheVMwareSD-WANEdgewillresultinalargenumber

ofSD-WANOverlaysbeingestablishedintheSD-WANEdgebutnotinuse,

whichcreatesunnecessaryoverhead.

SymmetricRoutewithPartnerGateway

Pertainingtotheprevioussection,thepartnergatewayusestheordernumberasthemetric when advertising the route to the PE router. However, in a situationwith twopartnergatewaysusingadifferentASnumber, thePEroutermightnotconsiderthemetricnumberforrouteselection.Thiscanresult inanasymmetricpathsituation,asseeninFigure8.12.

WhentheVMwareSD-WANEdge11sendstraffictoVMwareSD-WANEdge21,theprefixes fromVMware SD-WANEdge21 get the same attribute fromGW11 andGW22, and VMware SD-WAN Edge11will send the traffic toGW11 as it is thepartnergatewaywithorder1.

The PE routers will likely ignore the metric when GW11 and GW12 are usingdifferentASnumbers,andthePEroutersarenotcomparingmetricsfromdifferentAS.The return traffic fromVMwareSD-WANEdge21maycome toGW12,whichwill then forward the return traffic toVMwareSD-WANEdge11. TheconnectivitybetweenVMwareSD-WANEdge11andVMwareSD-WANEdge21issuccessfulbutundesirablebecausethetrafficpathisasymmetric.

Figure8.12:Undesirableasymmetricsituation

To avoid this asymmetric situation, it is recommended to utilize the BGPcommunitymappingfeature.

Figure 8.13 shows a sample BGP community mapping configuration. In thissample,thepartnergatewaywithorder1willattachBGPcommunity65000:100toalltheprefixesitadvertisestothePErouter.Forpartnergatewaywithorder2, itwill attach BGP community 65000:90 to all the prefixes it advertises to the PErouter.

InthePErouter,therewillbearoute-maptosetalocalpreferenceof100whenthe prefix is received with a BGP community of 65000:100 and set a localpreferenceof90whentheprefixisreceivedwithaBGPcommunityof65000:90.With the BGP communitymapping set and corresponding route-map in the PErouters, asymmetric paths across different partner gateways will be avoided.Navigateto Configure > Customer > Configure Hand Off > CustomerBGP Priority > Enable Community Mapping

Figure8.13:AutoBGPcommunitymapping

DefaultRoutefromPartnerGateway

In cases of a gatewaywhere partner handoff has been turned off, the gatewayalwaysadvertisesadefaultroutetotheconnectedVMwareSD-WANEdge.Inthepartner gateway scenario, the service provider administrator has an option todecidewhethertohavethepartnergatewayadvertiseadefaultrouteornot,butitisnotadvertisedbydefault.AdefaultcanbeconfiguredasastaticrouteandsettoNAT on the gateway. This will allow traffic in that enterprise to use the publichandoffinterfaceofthatGatewaytosendtrafficouttotheinternet,emulatingthebehaviorofastandardVMwareSD-WANGateway.

KeyTakeaways:

• PartnergatewaysdonotformSD-WANOverlaysamongstthemselves.IftrafficfromonepartnergatewayneedstoreachanotherpartnergatewayintheMPLSbackbone,thisisbasedonBGProuting.

• The assignment of partner gateway to the VMware SD-WAN Edgesgoverns what the traffic flow will be between the VMware SD-WANEdges.

• ItispossibleandrecommendedtoformaprivateSD-WANOverlayfromtheVMwareSD-WANEdge to thepartnergateway if theVMwareSD-WANEdge comeswithaprivateMPLS circuit connected to theMPLSbackbone.

• Measures should be taken to ensure the traffic going out from thepartner gateway to the MPLS backbone return on the same partnergatewaytoavoidasymmetricrouting.

To learn more, visit:

www.velocloud.com

VMware, Inc. 3401 Hillview Avenue Palo Alto CA 94304 USA Tel 877-486-9273 Fax 650-427-5001 www.vmware.com. Copyright © 2019 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at

http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. and its subsidiaries in the United States and other jurisdictions. All other marks and names mentioned herein may be trademarks of

their respective companies. Item No: vmw-wp-temp-word 2/19