© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.

Steve Seymour – Principal Solutions ArchitectNetworking Specialist, EMEA – Amazon Web Services

June 2017

Another Day, Another Billion Packets

@sseymour

Cloudfront

EC2Instance

Direct Connect

Availability Zone “a”

VPC subnet172.31.0.0/24

EC2Instance

Availability Zone “b”

VPC subnet172.31.1.0/24

172.31.0.0/16

YourData Center

YourUsers

Edge to Instance

Cloudfront

EC2Instance

Direct Connect

Availability Zone “a”

VPC subnet172.31.0.0/24

EC2Instance

Availability Zone “b”

VPC subnet172.31.1.0/24

172.31.0.0/16

YourData Center

YourUsers

Edge to Instance – Direct Connect

AWS Direct Connect

• Dedicated, private connection into AWS

• 1 Gbps or 10 Gbps connections

• Create private (VPC) or public virtual interfaces to AWS

• Consistent network performance

• Option for redundant connections

• Uses BGP to exchange routing information over a VLAN

AWS Direct Connect

AWS Region

Direct ConnectLocation

16 Regions - 60 Direct Connect Locations

Cloudfront

EC2Instance

Direct Connect

Availability Zone “a”

VPC subnet172.31.0.0/24

EC2Instance

Availability Zone “b”

VPC subnet172.31.1.0/24

172.31.0.0/16

YourData Center

YourUsers

Edge to Instance - CloudFront

The Amazon CloudFront Service

• Global Content Delivery Network with Massive Capacity and Scale

• Optimized for Performance and Scale

• Built in Security Features

• Self-Service Full Control Configurations

• Robust Real Time Reporting

Amazon CloudFront

• Static and Dynamic Object and Video Delivery

Edge location

AWS Region / Regional Edge Cache

Regional Edge Cache

North AmericaCities: 19PoPs: 27

Europe / Middle East / AfricaCities: 15PoPs: 24

Amsterdam, The Netherlands (2)Berlin, GermanyDublin, Ireland

Frankfurt, Germany (5)London, England (4)

Madrid, SpainMarseille, France

Milan, ItalyMunich, GermanyParis, France (2)

Prague, Czech RepublicStockholm, Sweden

Vienna, AustriaWarsaw, Poland

Zurich, Switzerland

Ashburn, VA (3)Atlanta, GA (3)

Chicago, ILDallas/Fort Worth, TX (3)

Hayward, CAJacksonville, FL

Los Angeles, CA (2)Miami, FL

Minneapolis, MNMontreal, QCNewark, NJ

New York, NY (3)Palo Alto, CA

Philadelphia, PASan Jose, CA

Seattle, WA (2)South Bend, INSt. Louis, MOToronto, ON

CloudFront Regional Edge CachesRegional Edge Caches: 11

Oregon, N. Virginia, Ohio, Frankfurt, London, Sao Paulo, Mumbai, Singapore,

Seoul, Tokyo, Sydney

Asia PacificCities: 12PoPs: 20

Chennai, IndiaHong Kong, China (3)Manila, the PhilippinesMelbourne, Australia

Mumbai, India (2)New Delhi, IndiaOsaka, Japan

Seoul, Korea (3)Singapore (2)

Sydney, AustraliaTaipei, Taiwan

Tokyo, Japan (4)

South AmericaCities: 2PoPs: 3

Rio de Janeiro, Brazil (2)São Paulo, Brazil

CloudFront Global Content Delivery Network88 Edge Locations - 77 PoPs, 11 Regional Edge Caches (20 in last 12 months)

Cloudfront

EC2Instance

Direct Connect

Availability Zone “a”

VPC subnet172.31.0.0/24

EC2Instance

Availability Zone “b”

VPC subnet172.31.1.0/24

172.31.0.0/16

YourData Center

YourUsers

Edge to Instance – Global Network

Cloudfront

EC2Instance

Direct Connect

Availability Zone “a”

VPC subnet172.31.0.0/24

EC2Instance

Availability Zone “b”

VPC subnet172.31.1.0/24

172.31.0.0/16

YourData Center

YourUsers

Edge to Instance - Region

Cloudfront

Direct Connect VPC subnet

172.31.0.0/24

VPC subnet172.31.1.0/24

172.31.0.0/16

YourData Center

YourUsers

Edge to Instance – Availability Zones

EC2Instance

EC2Instance

Availability Zone “a”

Availability Zone “b”

Cloudfront

Direct Connect VPC subnet

172.31.0.0/24

VPC subnet172.31.1.0/24

172.31.0.0/16

YourData Center

YourUsers

Edge to Instance – EC2 Instances

EC2Instance

EC2Instance

Availability Zone “a”

Availability Zone “b”

Cloudfront

Direct Connect VPC subnet

172.31.0.0/24

VPC subnet172.31.1.0/24

172.31.0.0/16

YourData Center

YourUsers

Edge to Instance – VPC

EC2Instance

EC2Instance

Availability Zone “a”

Availability Zone “b”

VPC Requirements

Customer selected IP addressesRoute aggregation for external connectivityConformance with existing network designs

172.31.0.0/18

192.168.0.0/16

Routing Table• 192.168.0.0/16: stay here• 172.31.0.0/18: AWS

172.31.1.0/24 172.31.2.0/24

172.31.1.7

172.31.1.8

172.31.1.9

172.31.2.12

172.31.2.51

Amazon Virtual Private Cloud

This Is Just Virtual Networking!

Subnet ~= VLANVPC ~= VRF (virtual routing and forwarding)But…

Scaling Challenges

VLAN ID space is constrained• 12 bits => 4096 total VLANs

VRF support is constrained• Large routers => 1-2 thousand VRFs

Fixed ratio of VLANs:VRFs

Implementation Requirements

Scale to millions of environments the size of Amazon.comAny server, anywhere in a region can host an instance attached to any subnet in any VPC

Server:Physical host in an Amazon data center

Instance:Amazon EC2 instance owned by a customer

VPC:Amazon Virtual Private Cloud owned by a customer

VPC ID:Identifier for a VPC such as vpc-1a2b3c4d

Mapping Service:Distributed lookup service. Maps VPC + Instance IP to server

Concepts

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.4

…

10.0.0.2

10.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

L2 Src: MAC(10.0.0.2)L2 Dst: ff:ff:ff:ff:ff:ff

ARP Who has 10.0.0.3?

The switch floods the ARP request out all ports

L2 Src: MAC(10.0.0.3)L2 Dst: MAC(10.0.0.2)

ARP 10.0.0.3 is at MAC(10.0.0.3)

The switch snoops the ARP response and learns the port for MAC(10.0.0.3).

L2 Src: MAC(10.0.0.2)L2 Dst: MAC(10.0.0.3)L3 Src: 10.0.0.2L3 Dst: 10.0.0.3

ICMP/TCP/UDP/…

Layer 2 (L2): Ethernet

10.0.0.2

10.0.0.3

Ethernet Switch

L2 Src: MAC(10.0.0.3)L2 Dst: MAC(10.0.0.2)

ARP 10.0.0.3 is at MAC(10.0.0.3)

Src: 192.168.0.3Dst: Mapping Service

Query: Blue 10.0.0.3

Src: Mapping Service Dst: 192.168.0.3

Reply: Host: 192.168.1.4MAC: MAC(10.0.0.3)

L2 Src: MAC(10.0.0.2)L2 Dst: ff:ff:ff:ff:ff:ff

ARP Who has 10.0.0.3?

Layer 2 (L2): VPC

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.410.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

10.0.0.2

Src: Mapping Service Dst: 192.168.1.4

Mapping valid:Blue 10.0.0.2 is at192.168.0.3

Src: 192.168.1.4Dst: Mapping Service

Validate: Blue 10.0.0.2 is at192.168.0.3

L2 Src: MAC(10.0.0.2)L2 Dst: MAC(10.0.0.3)L3 Src: 10.0.0.2L3 Dst: 10.0.0.3

ICMP/TCP/UDP/…

Src: 192.168.0.3Dst: 192.168.1.4

VPC: Blue

Server 192.168.0.3

Server 192.168.0.4

Server 192.168.1.3

Server 192.168.1.410.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

10.0.0.2

Layer 2 (L2): VPC

…

Src: 192.168.0.4Dst: Mapping Service

Query:Grey 10.0.0.3

L2 Src: MAC(10.0.0.4)L2 Dst: ff:ff:ff:ff:ff:ff

ARP Who has 10.0.0.3?

VPC Isolation

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.410.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

10.0.0.2

192.168.0.4 is not hosting any instances in VPC Blue.

Mapping DeniedAlarm Raised

L2 Src: MAC(10.0.0.4)L2 Dst: ff:ff:ff:ff:ff:ff

ARP Who has 10.0.0.3?

Src: 192.168.0.4Dst: Mapping Service

Query: Blue 10.0.0.3

VPC Isolation

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.410.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

10.0.0.2

Src: 192.168.1.4Dst: Mapping Service

Validate: Blue 10.0.0.4 is at192.168.0.4

Src: 192.168.0.4Dst: 192.168.1.4

L2 Src: MAC(10.0.0.4)L2 Dst: MAC(10.0.0.3)L3 Src: 10.0.0.4L3 Dst: 10.0.0.3

ICMP/TCP/UDP/…

VPC: Blue

Src: Mapping Service Dst: 192.168.1.4

Mapping invalid!

192.168.1.4 does not deliver the packet to the instance.

Alarm Raised.

VPC Isolation

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.410.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

10.0.0.2

L2 Src: MAC(10.0.0.2)L2 Dst: ff:ff:ff:ff:ff:ff

ARP Who has 10.0.0.1?

L2 Src: MAC(10.0.0.1)L2 Dst: MAC(10.0.0.2)

ARP 10.0.0.1 is at MAC(10.0.0.1)

L2 Src: MAC(10.0.0.2)L2 Dst: MAC(10.0.0.1)L3 Src: 10.0.0.2L3 Dst: 10.0.1.3

ICMP/TCP/UDP/…

Layer 3 (L3): IP Routing

10.0.0.2

10.0.1.3

Ethernet Switch

RouterEthernet Switch

L2 Src: MAC(10.0.1.1)L2 Dst: MAC(10.0.1.3)L3 Src: 10.0.0.2L3 Dst: 10.0.1.3

ICMP/TCP/UDP/…

L2 Src: MAC(10.0.0.2)L2 Dst: ff:ff:ff:ff:ff:ff

ARP Who has 10.0.0.1?

L2 Src: MAC(10.0.0.1)L2 Dst: MAC(10.0.0.2)

ARP 10.0.0.1 is at MAC(10.0.0.1)

Src: 192.168.0.3Dst: Mapping Service

Query: Blue 10.0.0.1

Src: Mapping Service Dst: 192.168.0.3

Reply: Host: GatewayMAC: MAC(10.0.0.1)

Layer 3 (L3): VPC

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.410.0.1.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

10.0.0.2

Src: Mapping Service Dst: 192.168.0.3

Reply: Host: 192.168.1.4MAC: MAC(10.0.1.3)

Src: 192.168.1.4Dst: Mapping Service

Validate: Blue 10.0.0.2 is at192.168.0.3

L2 Src: MAC(10.0.0.2)L2 Dst: MAC(10.0.0.1)L3 Src: 10.0.0.2L3 Dst: 10.0.1.3

ICMP/TCP/UDP/…

L2 Src: MAC(10.0.1.1)L2 Dst: MAC(10.0.1.3)L3 Src: 10.0.0.2L3 Dst: 10.0.1.3

ICMP/TCP/UDP/…

Src: Mapping Service Dst: 192.168.1.4

Mapping valid:Blue 10.0.0.2 is at192.168.0.3

Layer 3 (L3): VPC

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.410.0.1.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

Src: 192.168.0.3Dst: Mapping Service

Query: Blue 10.0.1.3

10.0.0.2

VPC: Blue

Src: 192.168.0.3Dst: 192.168.1.4

Caching

Server 192.168.0.3

Server 192.168.0.4

…

Server 192.168.1.3

Server 192.168.1.4

…

10.0.0.2

10.0.0.3

10.0.0.4

10.0.0.4

10.0.0.2

10.0.0.5

10.0.0.3

Mapping Service

L2 Src: MAC(10.0.1.1)L2 Dst: MAC(10.0.1.3)L3 Src: 10.0.0.2L3 Dst: 10.0.1.3

ICMP/TCP/UDP/…

10.0.0.0/18

172.16.0.0/16

10.0.0.0/24 10.0.1.0/24

10.0.0.7

10.0.0.8

10.0.0.9

10.0.1.12

10.0.1.51

Getting Home – Or Anywhere, Really

VPC: Blue

Src: 192.168.0.3Dst: ???

L3 Src: 10.0.0.7L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

Src: 192.168.0.3Dst: 192.168.4.3

L3 Src: 10.0.0.2L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

L3 Src: 10.0.0.2L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

Edges

Server 192.168.0.3

Server 192.168.0.4

…

Edge 192.168.4.3

Edge 192.168.4.4

10.0.1.3

10.0.0.4

10.0.0.2

Mapping Service

10.0.0.2

VPC: Blue

Host 10.0.0.4 è 192.168.0.4Host 10.0.1.4 è 192.168.0.4…172.16.0.0/16 è Edge 192.168.4.3…

Edges: VPN

Edge 192.168.4.3VPC: Blue

Src: 192.168.0.3Dst: 192.168.4.3

L3 Src: 10.0.0.2L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

IPSEC Stuff

Src: 54.68.100.245Dst: 205.251.242.54

L3 Src: 10.0.0.2L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

VPN

Edges: Direct Connect

Edge 192.168.4.3VPC: Blue

Src: 192.168.0.3Dst: 192.168.4.3

L3 Src: 10.0.0.2L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

802.1Q VLAN Tag

Src: 54.68.100.245Dst: 205.251.242.54

L3 Src: 10.0.0.2L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

AWSDirect Connect

Edges: Internet (IGW)

Edge 192.168.4.3VPC: Blue

Src: 192.168.0.3Dst: 192.168.4.3

L3 Src: 10.0.0.2L3 Dst: 176.32.96.190

ICMP/TCP/UDP/…

L3 Src: 10.0.0.2L3 Dst: 176.32.96.190

ICMP/TCP/UDP/…

Internet

54.148.157.46

Edges: Recap

VPNEdge 192.168.4.3

VPC: Blue

Src: 192.168.0.3Dst: 192.168.4.3

L3 Src: 10.0.0.2L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

IPSEC Stuff

Src: 54.68.100.245Dst: 205.251.242.54

L3 Src: 10.0.0.2L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

Direct ConnectEdge 192.168.4.3

VPC: Blue

Src: 192.168.0.3Dst: 192.168.4.3

L3 Src: 10.0.0.2L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

802.1Q VLAN Tag

Src: 54.68.100.245Dst: 205.251.242.54

L3 Src: 10.0.0.2L3 Dst: 172.16.14.17

ICMP/TCP/UDP/…

InternetEdge 192.168.4.3

VPC: Blue

Src: 192.168.0.3Dst: 192.168.4.3

L3 Src: 10.0.0.2L3 Dst: 176.32.96.190

ICMP/TCP/UDP/…

L3 Src: 54.148.157.46L3 Dst: 176.32.96.190

ICMP/TCP/UDP/…`

172.31.0.0/18

172.31.1.0/24 172.31.2.0/24

172.31.1.7

172.31.1.8

172.31.2.12

172.31.2.51

VPC As A Platform

Cloudfront

EC2Instance

Direct Connect

Availability Zone “a”

VPC subnet172.31.0.0/24

EC2Instance

Availability Zone “b”

VPC subnet172.31.1.0/24

172.31.0.0/16

YourData Center

YourUsers

Edge to Instance

Cloudfront

EC2Instance

Direct Connect

Availability Zone “a”

VPC subnet172.31.0.0/24

EC2Instance

Availability Zone “b”

VPC subnet172.31.1.0/24

172.31.0.0/16

YourData Center

YourUsers

Edge to Instance

Thank you!

@sseymour