Cloud Platform Integration Framework
i
Modern Datacenter - Architecture Patterns - Infrastructure
Prepared by Kunal Bhatia, Robert Larson, Michael Lubanski, Tom Shinder and Michael Withrow
“Hybrid Networking”
Hybrid Networking
25-Mar-15
Version 1.0
Prepared by
Kunal Bhatia, Robert Larson, Michael Lubanski, Tom Shinder and Michael Withrow
Modern Datacenter
Architecture Patterns -
Infrastructure
Cloud Platform Integration Framework
ii
Modern Datacenter - Architecture Patterns - Infrastructure
Prepared by Kunal Bhatia, Robert Larson, Michael Lubanski, Tom Shinder and Michael Withrow
“Hybrid Networking”
Revision and Signoff Sheet
Change Record
Date Author Version Change Reference
11/26/2014 Microsoft 1.0 Release
Cloud Platform Integration Framework
iii
Modern Datacenter - Architecture Patterns - Infrastructure
Prepared by Kunal Bhatia, Robert Larson, Michael Lubanski, Tom Shinder and Michael Withrow
“Hybrid Networking”
Table of Contents
1 Overview .................................................................................................................................................. 4
1.1 Technology Background ........................................................................................................................... 4
1.2 Pattern Requirements................................................................................................................................ 6
1.2.1 Connecting On-Premises Networks to Azure ........................................................... 7
1.2.2 Extending Hybrid Networks Across Azure Virtual Networks .................................... 9
1.2.3 Extending Hybrid Networks Across Azure Datacenter Regions ............................... 9
1.2.4 Extending Hybrid Networks Across Azure Subscriptions ........................................ 10
1.2.5 Providing developers remote access to the hybrid network ................................... 10
1.2.6 Additional recommended requirements .................................................................. 10
2 Architecture Pattern ........................................................................................................................... 11
2.1 Pattern Dependencies ............................................................................................................................. 14
2.2 Azure Services .......................................................................................................................................... 14
2.3 Pattern Considerations ............................................................................................................................ 14
3 Interfaces and End Points ................................................................................................................ 15
4 Availability and Resiliency ................................................................................................................ 16
5 Scale and Performance ..................................................................................................................... 17
6 Cost .......................................................................................................................................................... 17
6.1 Cost Factors .............................................................................................................................................. 18
6.2 Cost Drivers .............................................................................................................................................. 18
7 Operations ............................................................................................................................................. 20
8 Architecture Anti-Patterns ............................................................................................................... 23
Cloud Platform Integration Framework
Page 4
Modern Datacenter - Architecture Patterns - Infrastructure
Prepared by Kunal Bhatia, Robert Larson, Michael Lubanski, Tom Shinder and Michael Withrow
“Hybrid Networking”
1 Overview
The Hybrid Networking design pattern details the Azure features and services required to deliver
network functionality that can provide predictable performance and high availability across
geographic boundaries. A full list of Microsoft Azure regions and the services available within
each is provided within the Microsoft Azure documentation site. This document provides an
overview of Microsoft Azure networking capabilities for hybrid environments. Microsoft Azure
Virtual Networking enables you to create logically isolated networks in Azure and securely
connect them to your on-premises datacenter over the Internet or using a private network
connection. In addition, individual client machines can connect to an isolated Azure network
using an IPsec VPN connection.
The Hybrid Networking architecture pattern includes the following focus areas:
Connecting on premises networks to Azure
Extending Azure virtual networks across regions
Extending Azure virtual networks between subscriptions
Providing developers remote network access
1.1 Technology Background
In order to understand the Hybrid Networking Architectural Pattern, a basic knowledge of
Microsoft Azure networking is required. The following table defines terms that will be used in
the Hybrid Networking architectural pattern.
Technology Description
Virtual
Network
An address space that can be established in Azure that can be subdivided to logically
separate virtual machines and cloud services.
Local
Network Site
One or more address spaces that represent on-premises routable subnets that you
want to communicate with across a gateway
Subnet A defined set of addresses from the Azure virtual network address pool that you want
to logically separate for other addresses within the same network ID
Cloud Platform Integration Framework
Page 5
Modern Datacenter - Architecture Patterns - Infrastructure
Prepared by Kunal Bhatia, Robert Larson, Michael Lubanski, Tom Shinder and Michael Withrow
“Hybrid Networking”
DNS Server Server that provides domain name resolution to machines in Azure virtual networks.
One or more DNS servers can be added to an Azure subscription and then assigned to
a virtual network. The DNS server will be assigned to the virtual machines on the virtual
network through DHCP during the boot process.
Azure
Gateway
This is a virtual VPN gateway device that is created on request in an Azure virtual
network. It is designed to support routable connectivity between virtual networks, on
premises locations, and single remote access VPN client machines. The gateway comes
in three configurations: Dynamic, Static and Dedicated. Gateway uses IPSec and pre-
shared keys to establish connectivity and encrypt data in motion
On Premises
Gateway
This is a physical gateway that is established at a customer on premises location. In
addition, this must be a supported device to be used with Microsoft Azure.
vNet-to-vNet
connectivity
This is the capability to have one or more virtual networks in a routable configuration.
This is accomplished using IPSec based gateways and pre-shared keys. To accomplish
vNet-vNet routing across the Azure backbone, you must configure vNet-to-vNet multi-
hop routing.
vNet Multi-
hop Routing
Virtual Network multi-hop routing requires the Azure Gateway device to be configured
with vNet-to-Local Network Site routing where the local network site definition has all
of the routable subnets on the other side of the connection.
Daisy Chain A virtual network multi-hop routing configuration where the virtual networks connect
in an end-to-end fashion
Hub and
Spoke
A virtual network multi-hop routing configuration where the virtual networks connect
to a central hub virtual network and multiple hubs connect together for expansion.
vNet1 vNet2 vNet3
vNet-vNet vNet-vNet
vNet4
vNet-vNet
vNet5
vNet-vNet
Hub
vNet2 vNet3 vNet4 vNet5
Hub2
vNet-vNet
vNet9 vNet X
Cloud Platform Integration Framework
Page 6
Modern Datacenter - Architecture Patterns - Infrastructure
Prepared by Kunal Bhatia, Robert Larson, Michael Lubanski, Tom Shinder and Michael Withrow
“Hybrid Networking”
Full Mesh A Virtual network routing configuration where the virtual networks connect to every other virtual network over a single hop connection. While this allows routing access to every subnet, it limits scalability due to the virtual network connection limit for an Azure gateway
Site-to-Site
VPN
Azure Virtual Network to on premises connection utilizing Azure Dynamic or Static
Routing Gateway and On Premises Gateway to establish a routable communications
path between subnets on both sides. The connection is an IPsec tunnel over the
Internet using pre-shared keys to establish the connection and perform encryption.
Point-to-Site
VPN
Client computer to Azure virtual network connection utilizing an Azure Gateway and a
VPN client. The connection is established as an IPSec VPN using certificates.
ExpressRoute Azure Virtual Network to on premises connection utilizing an Azure Dedicated Routing
Gateway and a private network path to the on premises environment. No traffic flows
across the Internet. Implementation of the ExpressRoute connection can be over a
multi-protocol label switching (MPLS) circuit or dedicated connection.
1.2 Pattern Requirements
The following are general requirements of the Hybrid Networking Infrastructure architectural
pattern:
An active Microsoft Azure account
An active Microsoft Azure Subscription within the account
An active Internet connection to access the Microsoft Azure portal or the service
management API to configure Microsoft Azure subscription networking
IP address pool for establishing Virtual Networks in Microsoft Azure that is non-
overlapping with on premises
IP Subnets from the IP Address pool for point-to-point connectivity: A non-overlapping
IP address block (/28 subnet) to configure IP addressing for the Site-to-Site or
ExpressRoute circuits
vNet1 vNet2
vNet3
vNet-vNet
vNet4
vNet-vNet
Cloud Platform Integration Framework
Page 7
Modern Datacenter - Architecture Patterns - Infrastructure
Prepared by Kunal Bhatia, Robert Larson, Michael Lubanski, Tom Shinder and Michael Withrow
“Hybrid Networking”
Once these requirements are satisfied, the following actions can be performed to provide
network access to the organization’s Azure subscription:
1. Connecting on-premises networks to Azure
2. Extending on-premises networks across Azure Virtual Networks
3. Extending networks across Azure datacenter regions
4. Extending networks across Azure subscriptions
5. Providing developers remote access to the Azure network
The following sections will describe each of the scenarios outlined above.
1.2.1 Connecting On-Premises Networks to Azure
Establishing a hybrid networking connection to Microsoft Azure can be accomplished using two
approaches. The first involves utilizing an IPsec based Virtual Private Network (VPN) between
Microsoft Azure and an on-premises network infrastructure and is often referred to as a “Site-to-
Site” (S2S) connection. This approach uses the Internet as the communication medium and
therefore is subject to random changes in latency and bandwidth availability. The second
approach is to use a dedicated network connection between Microsoft Azure and an on-
premises network infrastructure and is referred to as an ExpressRoute connection. This approach
uses dedicated private circuits or private network clouds for the connection. Unlike site-to-site
connections, these private network paths do not utilize the Internet and are not subject to the
random changes in latency and bandwidth of an Internet based connection.
Cloud Platform Integration Framework
Page 8
Modern Datacenter - Architecture Patterns - Infrastructure
Prepared by Kunal Bhatia, Robert Larson, Michael Lubanski, Tom Shinder and Michael Withrow
“Hybrid Networking”
IPsec VPN Connectivity
Site-to-Site communications utilize an IPsec tunnel established between two gateway devices
(one hosted within Azure and one on-premises). The IPsec tunnel uses pre-shared key approach
for establishing the connection and encrypting the data flowing over the connection. This allows
data to flow across the Internet in a secure manner. The gateway within Azure is a virtual
gateway implemented as a small virtual machine. When the gateway is created, you must select
either a dynamic routing or a static routing gateway. Note that the on-premises gateway device
must be implemented using either a supported model or compatible with a supported model.
Details about the supported on premises gateways can be found in the article About VPN
Devices for Virtual Network.
To establish the Site-to-Site connection between the two gateways, you must provide a pre-
shared key and configure both gateways to utilize the key. The pre-shared key can be generated
by the Azure portal, or it can be created and set on the connections using PowerShell. A single
gateway may now establish multiple site-to-site connections to different on-premises locations.
This enables redundancy for site-to-site communications. In this configuration, a separate on-
premises gateway is required to establish the alternate connection.
ExpressRoute Connectivity
ExpressRoute allows direct site-to-site connectivity between an on-premises datacenter and
Microsoft Azure, thus bypassing traditional, shared network connections through the Internet. It
allows for flexibility of choice in the network performance desired. ExpressRoute allows two
types of private connectivity to Azure - Through an ExpressRoute location (Internet Exchange
Provider facility or IXP) or directly from an existing MPLS VPN network provided by a network
service provider (NSP).
The following summarizes the requirements of the Hybrid Networking Infrastructure
architectural pattern for connecting on-premises networks to Microsoft Azure:
S2S Internet connectivity
Azure routing gateway (Static or dynamic)
On-premises routing gateway device that is on the Microsoft approved list of
gateway devices for Site-to-Site connections.
Potential secondary on-premises routing devices if the pattern requires multiple on-
premises site connections
Pre-shared key for each connection path for establishing connection and encryption
of data
Cloud Platform Integration Framework
Page 9
Modern Datacenter - Architecture Patterns - Infrastructure
Prepared by Kunal Bhatia, Robert Larson, Michael Lubanski, Tom Shinder and Michael Withrow
“Hybrid Networking”
ExpressRoute Connectivity
Routing gateway device that is on the Microsoft approved list of gateway devices for
ExpressRoute connections.
An existing business relationship with an Internet Exchange Provider (IXP) or a
Network Service Provider (NSP) who are on Microsoft's supported list of vendors for
facilitating ExpressRoute connectivity
Connectivity to an IXP: Network connectivity from customer datacenter to a
supported IXP Datacenter (e.g. Equinix), along with an Ethernet cross-connect to IXP's
Layer 2 peering infrastructure
Connectivity to an NSP: MPLS Layer 3 VPN connectivity through at least one of
customer sites/datacenters to a supported NSP network (e.g. - AT&T NetBond)
1.2.2 Extending Hybrid Networks Across Azure Virtual Networks
Hybrid networks can be established from an on-premises environment to Azure virtual networks.
By default, these connections only allow the on-premises network to talk to the virtual network
that the connection is established with (the first hop virtual network). In order to extend the
connection from the first hop virtual network to additional virtual networks in Azure, you make
configuration changes to establish multi-hop routing.
Establishing a hybrid network connection across Azure Virtual Networks requires an Azure
gateway to be established and connected to the on-premises gateway. In the Site-to-Site
scenario, an Azure static or dynamic routing gateway is required. If multi-hop virtual network
routing is desired, then the dynamic routing Azure gateways must be utilized. In the
ExpressRoute scenario, an Azure dedicated gateway is required. It is not possible to combine
multi-hop routing and ExpressRoute connections currently. To obtain access to multiple Azure
virtual networks in the ExpressRoute scenario, a separate ExpressRoute connection must be
established with each virtual network.
1.2.3 Extending Hybrid Networks Across Azure Datacenter Regions
Hybrid network connections can span Azure datacenter regions to allow traffic to route between
regions. This can be accomplished either using virtual networks in different regions connected
using IPsec tunnels or ExpressRoute connections independently established to virtual networks
in separate regions. These two methods cannot be combined.
Cloud Platform Integration Framework
Page 10
Modern Datacenter - Architecture Patterns - Infrastructure
Prepared by Kunal Bhatia, Robert Larson, Michael Lubanski, Tom Shinder and Michael Withrow
“Hybrid Networking”
Establishing a hybrid network connection to Azure Virtual Networks in different regions requires
an Azure gateway be established in each virtual network. These gateways will then connect to
each other to establish the link between Azure Virtual Networks. An on-premises gateway is
required to establish a hybrid network connection to an on-premises datacenter. In the Site-to-
Site scenario, an Azure dynamic routing gateway and multi-hop routing is required for cross
region connections. In the ExpressRoute scenario, an Azure dedicated gateway is required. For
ExpressRoute to obtain access to Azure virtual networks in different regions, a separate
ExpressRoute connection must be established with each virtual network.
1.2.4 Extending Hybrid Networks Across Azure Subscriptions
Hybrid network connections can span Azure subscriptions to allow traffic to route between
virtual networks in each subscription. This can only be accomplished using multi-hop routing of
virtual networks in different subscriptions connected using IPsec tunnels. ExpressRoute
connections currently are not supported across subscriptions.
Establishing a hybrid network connection to Azure Virtual Networks in different subscriptions
requires an Azure Gateway be established in each virtual network and connected. In the Site-to-
Site scenario, an Azure dynamic routing gateway and multi-hop routing is required.
1.2.5 Providing developers remote access to the hybrid network
In scenarios where a user does not have access to the corporate network where a Site-to-Site or
ExpressRoute hybrid network connection can provide routable access to Azure virtual networks,
it is possible to establish a temporary connection to the hybrid network by creating a Point-to-
Site (P2S) connection directly from the a Windows-based system to an Azure virtual network. To
accomplish this, an Azure virtual network must be configured to support Point-to-Site
connections (remote access VPN client connections), an address range must be established for
the machines connecting, a certificate must be created and uploaded for IPsec encryption, and
the client machine must download a VPN client to establish the connection.
1.2.6 Additional recommended requirements
The following are not required to establish hybrid networking, but are recommended for an
operational environment:
Accessible DNS servers that can be specified in Microsoft Azure for automatic
assignment to virtual machines placed on virtual networks
Cloud Platform Integration Framework
Page 11
Modern Datacenter - Architecture Patterns - Infrastructure
Prepared by Kunal Bhatia, Robert Larson, Michael Lubanski, Tom Shinder and Michael Withrow
“Hybrid Networking”
An Enterprise Certificate Authority to generate trusted certificates for Point-to-Site
connections
2 Architecture Pattern
The hybrid networking architecture pattern is complex due to the possible number of scenarios
that can be created. This architectural pattern will focus on the following four scenarios:
Site-to-Site hybrid networking with Multi-hop virtual network routing within a single
subscription and region
Site-to-Site hybrid networking with multi-hop virtual network routing across
subscriptions and regions
ExpressRoute hybrid networking using MPLS connectivity
ExpressRoute hybrid networking using IXP connectivity
Site-to-Site Hybrid Networking with Multi-Hop Virtual Network Routing within a Single
Subscription and Region
Azure Subscription w/Multihop vNet routing
Virtual NetworkVirtual Network Virtual Network
Site-to-Site
Dynamic Routing Gateway
Internet
Enterprise Datacenter
On Premises Gateway
Enterprise Datacenter
On Premises Gateway
Cloud Platform Integration Framework
Page 12
Modern Datacenter - Architecture Patterns - Infrastructure
Prepared by Kunal Bhatia, Robert Larson, Michael Lubanski, Tom Shinder and Michael Withrow
“Hybrid Networking”
Site-to-Site Hybrid Networking with Multi-Hop Virtual Network Routing across
Subscriptions and Regions
Azure Subscription A East US Region
Virtual Network Virtual Network
Site-to-Site
Dynamic Routing Gateway
Internet
Enterprise Datacenter
On Premises Gateway
Enterprise Datacenter
On Premises Gateway
Azure Subscription B West US Region
Virtual Network Virtual Network
Dynamic Routing Gateway
Azure Subscription A West Europe Region
Virtual Network Virtual Network
Dynamic Routing Gateway
vNet-vNet across Subscription
and region
Cloud Platform Integration Framework
Page 13
Modern Datacenter - Architecture Patterns - Infrastructure
Prepared by Kunal Bhatia, Robert Larson, Michael Lubanski, Tom Shinder and Michael Withrow
“Hybrid Networking”
ExpressRoute Hybrid Networking using MPLS connectivity
This option uses a Network Service Provider and is for customers who already use MPLS VPN
services through a supported telecommunications provider. In this option, Azure becomes
another "branch office" extension to the customer's existing MPLS network. This option is
relatively simple to deploy as the customer's carrier is responsible for configuring and managing
the ExpressRoute connection. This option is suitable for customers who already use MPLS VPN
services through a supported Telco. In this option, Azure becomes another "branch office"
extension to the customer's existing MPLS network. This option is relatively simple to deploy as
the customer's carrier is responsible for configuring and managing the ExpressRoute connection.
This is illustrated in the diagram below:
MPLS VPN Network
ExpressRoute Hybrid Networking using IXP Connectivity
This option uses a Network Service Provider and allows customers to directly peer with Azure by
collocating customer premises equipment at an IXP (e.g. - Equinix). This option provides more
granular routing control to the customer and also enables higher bandwidth connections to
Azure. It is suitable for enterprises that require a high-throughput connection to Azure or run
Cloud Platform Integration Framework
Page 14
Modern Datacenter - Architecture Patterns - Infrastructure
Prepared by Kunal Bhatia, Robert Larson, Michael Lubanski, Tom Shinder and Michael Withrow
“Hybrid Networking”
low latency hybrid workloads, Media services, storage/backup in Azure. This option requires the
customer to have in-house capabilities for configuring and managing this network
infrastructure. This option is illustrated in the diagram below:
Customer DatacenterExpressRoute Partner Location
2.1 Pattern Dependencies
Being an infrastructure pattern, there are no dependencies on other patterns, however
dependencies with the customer’s on-premises network environment and the network service
provider used within the organization.
2.2 Azure Services
The following Azure Services are required for this architecture pattern as outlined above:
Virtual Networks
Virtual Network Gateways
2.3 Pattern Considerations
For each component of Microsoft Azure, a series of subscription and service limits are defined
by the service. These limits are subject to change and are published at the following link.
Microsoft Azure limits fall into the categories of default and maximum limits. Default limits are
those which exist on every Azure subscription and can be increased through a request to
Microsoft support whereas maximum limits define the upper boundary of a given service or
capability within Azure. Limits can be raised by contacting Microsoft support as outlined
Cloud Platform Integration Framework
Page 15
Modern Datacenter - Architecture Patterns - Infrastructure
Prepared by Kunal Bhatia, Robert Larson, Michael Lubanski, Tom Shinder and Michael Withrow
“Hybrid Networking”
through the Azure portal as outlined in this article, however the request cannot exceed the
maximum limits outlined for each Azure service listed above. The following considerations must
be taken into account during architectural planning of the Hybrid Networking pattern:
Currently there are hard and soft limits on the number of Virtual Networks per Azure
subscription. The default limit is 10, while the maximum limit is 100.
Azure currently supports up to 3000 customer routes per ExpressRoute BGP peering.
Quality of Service (QoS):
- Customer traffic inbound to Azure: The NSP may provide QoS treatment of traffic
traversing over its network towards Azure; this is dependent on the service Offering
procured from the carrier.
- Customer traffic outbound from Azure: This traffic is provided default QoS treatment
as Microsoft routers today do not mark DSCP/CoS (Class of Service) on this traffic.
A virtual network in Azure can have a maximum of a single routing gateway.
A routing gateway can only be in a single state at a time (static or dynamic).
A dynamic routing gateway in Azure has a maximum number of connections (site-to-site
or vNet-to-vNet) as identified in the Azure Subscription and Service Limits, Quotas, and
Constraints product documentation.
A Point-to-Site VPN requires dynamic routing gateway.
A Site-to-Site VPN requires a dynamic or static routing gateway.
An ExpressRoute connection requires a dedicated routing gateway.
By default a vNet-to-vNet connection can only route across a single vNet.
ExpressRoute connection cannot utilize vNet-to-vNet routing to pass traffic across the
Azure backbone.
3 Interfaces and End Points
As outlined earlier, the Hybrid Networking pattern utilizes different types of end points to
establish connectivity between the on-premises network and the Azure network infrastructure
for each subscription. For the purposes of hybrid networking, these endpoints can be divided
into physical and logical end points.
Cloud Platform Integration Framework
Page 16
Modern Datacenter - Architecture Patterns - Infrastructure
Prepared by Kunal Bhatia, Robert Larson, Michael Lubanski, Tom Shinder and Michael Withrow
“Hybrid Networking”
The following table outlines these end points by type:
End Point Type Description
Physical
IXP based connectivity - Ethernet cross connect setup between customer &
Microsoft equipment collocated in IXP facility
NSP based connectivity - MPLS based connectivity to Azure is extended from
within existing customer WAN
Logical
BGP session # 1 for connectivity to Azure Compute services within a Vnet - VM's
(IaaS) and Cloud Services (PaaS)
BGP session # 2 for connectivity to Azure Public Services - SQL, Storage, Media
Services etc
4 Availability and Resiliency
Microsoft provides clearly defined Service Level Agreements (SLAs) for each service provided
within Azure. Each architectural pattern is comprised of one or more Azure services and details
about each individual Azure service can be found on the Microsoft Azure Service Level
Agreement website. For the Hybrid Networking architectural pattern, the Azure services
required carry the following SLAs:
Azure Service Service Level Agreement
Virtual Network Gateway 99.9%
Virtual Networks 99.9%
ExpressRoute 99.9%
The composite Service Level Agreement (SLA) of the Hybrid Networking architectural pattern is
99.9% based on the services outlined above. The details behind the SLA can differ based on the
Cloud Platform Integration Framework
Page 17
Modern Datacenter - Architecture Patterns - Infrastructure
Prepared by Kunal Bhatia, Robert Larson, Michael Lubanski, Tom Shinder and Michael Withrow
“Hybrid Networking”
type of connectivity utilized. For ExpressRoute connections, the composite SLA is comprised of
the ExpressRoute SLA offered by Microsoft combined with the underlying infrastructure which is
provided by a third party service provider. For Site-to-Site connections, the composite SLA is
comprised of the VPN SLA offered by Microsoft is around the Virtual Gateway and not the
connection itself.
5 Scale and Performance
A key consideration for the performance and scale of the Hybrid Networking architectural
pattern is highly dependent on the network connection itself, which is defined by the type of
connection used. In general, ExpressRoute provides higher scale and performance (higher
throughput, lower latency, lower jitter) in comparison to an Internet-based IPSec VPN
connection (site-to-site or point-to-site). The differences in the type of connection and
expected speeds are outlined in the table below:
Type of Connection Speed
Network Service Provider 10Mbps, 50Mbps, 100Mbps, 500Mbps & 1Gbps
Exchange Provider 200Mbps, 500Mbps, 1Gbps & 10Gbps
Site-to-Site VPN Default: 100Mbps or High Performance (200Mbps)
Point-to-Site VPN 100Mbps
6 Cost
An important consideration when deploying any Solution within Microsoft Azure is the cost of
ownership. Costs related to on-premises cloud environments typically consist of up-front
investments in compute, storage and network resources, while costs related to public cloud
environments such as Azure are based on the granular consumption of the services and
resources found within them. Costs can be broken down into two main categories: cost factors
and cost drivers. Cost factors consist of the specific Microsoft Azure services which have a unit
Cloud Platform Integration Framework
Page 18
Modern Datacenter - Architecture Patterns - Infrastructure
Prepared by Kunal Bhatia, Robert Larson, Michael Lubanski, Tom Shinder and Michael Withrow
“Hybrid Networking”
consumption cost and are required to compose a given architectural pattern. Cost drivers are a
series of configuration decisions for these services within a given architectural pattern that can
increase or decrease costs. Microsoft Azure costs are divided by the specific service or capability
hosted within Azure and continually updated to keep pace with the market demand. Costs for
each service are published publicly on the Microsoft Azure pricing calculator. It is recommended
that costs be reviewed regularly during the design, implementation and operation of this and
other architectural patterns.
6.1 Cost Factors
Cost factors within the Hybrid Networking architectural pattern include the choice in
ExpressRoute providers and connectivity options (Site-to-Site vs. ExpressRoute). For Site-to-Site
connections (VPN) there is a minimal charge for the Gateway itself. The connection runs over
the open Internet. Customers can establish the ExpressRoute Service in two ways – Exchange
Provider or Network Service Provider. The associated charges for both these options differ
accordingly. ExpressRoute service using an Exchange Provider is charged based on a monthly
dual-port fee for two physical Ethernet ports (on two routers). Inbound transfer (e.g. - Customer
to Azure) is included. A finite outbound transfer (e.g. - Azure to Customer) is included with the
port speed selected. Any outbound transfer beyond that is charged based on bandwidth
consumed. Alternatively, ExpressRoute using a Network Service Provider is charged based on a
monthly dual-port fee for two physical Ethernet ports (on two routers), and there are no
additional costs related to all inbound and outbound data transfer.
6.2 Cost Drivers
As stated earlier, cost drivers consist of the configurable options of the Azure services required
when implementing an architectural pattern which can impact the overall cost of the Solution.
These configuration choices can have both a positive or negative impact on the cost of
ownership of a given Solution within Azure, however they may also potentially impact the
overall performance and availability of the Solution depending on the selections made by the
organization. Cost drivers can be categorized by their level of impact (high, medium and low).
Cloud Platform Integration Framework
Page 19
Modern Datacenter - Architecture Patterns - Infrastructure
Prepared by Kunal Bhatia, Robert Larson, Michael Lubanski, Tom Shinder and Michael Withrow
“Hybrid Networking”
Cost drivers for the Hybrid Network architectural pattern are summarized in the table below.
Level of
Impact
Cost Driver Description
High
Connectivity from customer
location to Azure
While considering the cost of connecting to Azure
using ExpressRoute, it must be kept in mind that
Network service providers/Exchange providers may
levy additional charges/fees for their services in
addition to the ExpressRoute charges levied by
Microsoft.
Cost of egress of data using
IPsec VPN connection to
Azure
An IPsec VPN based connection to Azure incurs
charges only for egress of customer data outbound
from Azure. There are no cost impacts for ingress
(inbound) of data into Azure, irrespective of the
amount of data transferred.
Medium
Data transfer over
ExpressRoute within
US/Europe Azure regions
Once a customer has established an ExpressRoute
connection to a Microsoft location within US/Europe,
data can be sent to/from any other Azure region
within that continent. For example, a US customer can
send or receive data to/from any Azure region within
US without additional charges from Microsoft. This
however, is not applicable to Asia-Pacific
Data transfer over vNet-to-
vNet connection across Azure
zones
In a VNet-to-VNet Scenario, an egress charge will be
assessed if those VNet’s are in different Azure zones.
Low Cancellation of ExpressRoute
Service
Prorated billing is applied in case the ExpressRoute
service is cancelled during a month such that the
billing would only be for the hours used and the
actual data transfer incurred.
Cloud Platform Integration Framework
Page 20
Modern Datacenter - Architecture Patterns - Infrastructure
Prepared by Kunal Bhatia, Robert Larson, Michael Lubanski, Tom Shinder and Michael Withrow
“Hybrid Networking”
7 Operations
Cloud Platform Integration Framework (CPIF) extends the operational and management
functions of Microsoft Azure, System Center and Windows Server to support managed cloud
workloads. As outlined in CPIF, Microsoft Azure architectural patterns support deployment,
business continuity and disaster recovery, monitoring and maintenance as part of the operations
of the Hybrid Networking architectural pattern.
Deployment of this pattern can be achieved through multiple methods. In addition to using the
Management portal, ExpressRoute deployment currently requires PowerShell-based
configuration tasks which need to be performed through the Service Management REST API.
The following three broad steps that need to be carried out in a proper order for a circuit to be
fully provisioned. Detailed configuration steps can be found in the article Configure an
ExpressRoute Connection through an Exchange Provider.
1. Creation of ExpressRoute circuits
2. Route configuration
3. Linking Virtual network to ExpressRoute circuit
Cloud Platform Integration Framework
Page 21
Modern Datacenter - Architecture Patterns - Infrastructure
Prepared by Kunal Bhatia, Robert Larson, Michael Lubanski, Tom Shinder and Michael Withrow
“Hybrid Networking”
These steps are illustrated in the diagram below:
Figure 1: ExpressRoute Deployment Workflow for Exchange Provider based connection
Cloud Platform Integration Framework
Page 22
Modern Datacenter - Architecture Patterns - Infrastructure
Prepared by Kunal Bhatia, Robert Larson, Michael Lubanski, Tom Shinder and Michael Withrow
“Hybrid Networking”
Azure Virtual Networks can also be created through the portal and Azure PowerShell. When
dealing with VNet-to-VNet configuration and multi-site VPN configurations, customers will be
required to author a .netcfg xml configuration file for deployment.
Monitoring of this pattern and associated resources depends on the connection type. An
ExpressRoute circuit can be managed and monitored through the Service Provider portal. A Site-
to-Site VPN connection can be managed and monitored through the Service Portal. From this
view a customer will only see the External interfaces from a connection perspective. With a Site-
to-Site VPN connection the only side which is visible in the portal is the Azure side of the
connection. In order to look at verbose logging of the connection access to the on-premises
router will be required. For VNet-to-VNet monitoring, the gateways and the connection can be
monitored solely in the Azure Portal.
Depending on the nature of the task involved, maintenance of an ExpressRoute connection can
be performed using a combination of Azure Management Portal, Service Provider portal and/or
PowerShell through the Service Management API.
From a business continuity and disaster recovery perspective, the availability constructs outlined
earlier should help make sure that solutions deployed using the Hybrid Networking architectural
pattern are designed to support continuity of operations, however it is important to understand
how this differs between ExpressRoute and Site-to-Site VPN connections.
ExpressRoute connections are typically terminated at both ends using two physical Ethernet
ports on two separate routers. This allows for a pair of simultaneously active physical
connections to carry data traffic to/from an ExpressRoute location. While Microsoft offers a
99.9% uptime SLA on the ExpressRoute service itself, it is up to the customer to appropriately
architect the network connectivity to an ExpressRoute location. This SLA will not be applicable if
the circuit is not built in an active-active configuration. Site-to-Site VPN connections are
associated to a single Azure Virtual Gateway that is redundant, but that connection is not
redundant. In the event of a lost connection on a Gateway a customer will have to establish
another connection.
Operational changes to an ExpressRoute circuit generally require modifications to both
Microsoft Azure and the underlying Service provider’s infrastructure. While Microsoft allows
customers to perform operational changes through the Service Management REST API, the
service provider generally offer a portal through which operational tasks can be carried out.
Operational changes to a Site-to-Site VPN Connection are very limited. Once the network is in
use, most changes are restricted. If you need to delete a gateway or change a gateway and
Cloud Platform Integration Framework
Page 23
Modern Datacenter - Architecture Patterns - Infrastructure
Prepared by Kunal Bhatia, Robert Larson, Michael Lubanski, Tom Shinder and Michael Withrow
“Hybrid Networking”
there are virtual machines on the Virtual Network. They will have to be removed in order to
make that modification.
8 Architecture Anti-Patterns
Establishing a Hybrid Network to Microsoft Azure where multi-hop routing will be used in a
Site-to-Site VPN scenario is not recommend. It is not recommended to utilize the “full mesh”
approach (outlined above) to perform multi-hop virtual network routing. When multi-hop virtual
network routing is configured using the full mesh routing approach, the supported number of
virtual network connections per gateway will limit the scale of the hybrid networking
infrastructure in Azure based on the gateway type selected.
The ExpressRoute connection is meant to be a Layer 3 (routed) connection between the
customer and Azure. As such, no Layer 2 connectivity is allowed over an ExpressRoute circuit.