Implementing an IPv6 Enabled Environment for a Public Cloud Tenant

Implementing an IPv6 Enabled Environment for a Public Cloud Tenant Sharmin Choksey (Cisco), Anik Mazumder (Cisco), Shixiong Shang (Nephos6)

* OpenStack is a trademark of OpenStack Foundation

Introduction

2

Sharmin Choksey

Technical Leader Cisco Systems Inc. Email: [email protected]

Shixiong Shang

Chief Technology Officer Nephos6 (A Cloud and IPv6 Company) Twitter: @shshang

Email: [email protected]

Anik Mazumder

Architect Cisco Systems Inc. Email: [email protected]

mailto:[email protected]

mailto:[email protected]

AgendaIPv6 in Public Cloud Overview

Architecture and Design

Challenges

Scalability and Performance

Lessons and Learns

Next Steps

3

IPv6 in Public CloudOvercome public IP (v4) exhaustion problem by moving to IPv6

Build a public cloud without the complexities of overlapping IP address space or NAT

Telcos and Mobile providers need IPv6 to cloud enable their services

Facilitate cloud adoption of highly distributed services like IOE/IOT

Facilitate adoption of IPv6 in public clouds

Allow tenants to embrace IPv6 for their business needs

Facilitate cloud adoption of network centric services

Increasing demand from Asia

4

IPv6 in Public Cloud

5

Logical Architecture – Icehouse OSP 5 on RHEL 7

6

M M M

Storage)Cluster

RBD)Computes)Nodes

Local)Storage

Local)Compute

Network)Nodes

Logical Scope

7

Neutron Network Node


Open vSwitch (br-‐int)

tap-‐ interface

qdhcp namespace

dhcp agent

metadata agent

ovs agent Open vSwitch (br-‐ex)

eth1

Nova Compute Node

Nova Compute Node

Dual Stack VM

vnic (ipv4 and ipv6)


Open vSwitch (br-‐ex)

eth1

VLAN Trunking

HSRP

ovs agent

IPv6 Router Advertisement: IPv6 Prefix Default GW LLA

A=1, M=0, O=1 (dhcpv6 stateless)

Linux Bridge (qbr)

DHCP

v6 In

fo-‐Req

uest

DNSMASQ

dhcpv6 stateless server ipv4 dhcp server

DHCPv6 Info-‐Reply (from dnsmasq)

Provider Networks

HSRP

Challenges - Security: Reconnaissance Attack

8

VLAN Trunking

Nova Compute Node

eth1

Linux Bridge (qbr)

qbrd94e5c3e-‐94: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet6 fe80::493:bff:fead:c085 prefixlen 64 scopeid 0x20<link> ether f2:de:90:e5:04:bf txqueuelen 0 (Ethernet)

ping6 -‐I eth0 ff02::1 64 bytes from fe80::493:bff:fead:c085: icmp_seq=1 ttl=64 time=0.046 ms

SLAAC auto-‐configures various ports (qbr-‐*, qvb-‐*, qvo-‐*, int-‐br-‐ex, phy-‐br-‐ex) on Compute node with IPv6 Link Local Address



Hello, my neighbors!

Hacker can try to gain the access to the hypervisor via the IPv6 Link Local Address

ssh root@fe80::493:bff:fead:c085%eth0

echo 'net.ipv6.conf.default.disable_ipv6=1' >> /etc/sysctl.conf echo 'net.ipv6.conf.all.disable_ipv6 = 1' >> /etc/sysctl.conf sysctl -‐p

Challenges - Security: RA Guard

9



tap-‐ interface

qdhcp namespace

DNSMASQ


Nova Compute Node

Dual Stack VM




VLAN Trunking

Linux Bridge (qbr)

Nova Compute Node



Linux Bridge (qbr)

Bogus IPv6 RA

(i.e. Blackhole tenant IPv6 traffic!)

Legitimate IPv6 RA

(from upstream router)

eth1 eth1


HSRP

Challenges - Security: DHCPv6 Guard

10

Neutron Network Nodeqdhcp namespace

eth1

Nova Compute Node

Dual Stack VM


eth1

VLAN Trunking

Nova Compute Node

DHCP

v6 In

fo-‐Req

uest

Bogus DHCPv6 Info-‐Reply

(i.e. Poison tenant nameserver entry)

DHCPv6 Info-‐Reply (from dnsmasq)

ff02::1:2 udp/547

DNSMASQ



tap-‐ interface



Linux Bridge (qbr)



Linux Bridge (qbr)


HSRP

Changes on Network Infrastructure

11

vlan 803 name customer-‐1 vlan configuration 800 no ip igmp snooping optimise-‐multicast-‐flood

vpc domain 5 peer-‐switch role priority 1 system-‐priority 100 peer-‐keepalive destination 1.1.1.2 source 1.1.1.1 vrf vpc peer-‐gateway auto-‐recovery ip arp synchronize ipv6 nd synchronize

interface Vlan803 mtu 9216 vrf member customer no ip redirects ip address 192.168.254.2/24 ipv6 address 2001:db8:cafe:b::2/64 ipv6 nd other-‐config-‐flag ipv6 nd prefix 2001:db8:cafe:b::/64 no ipv6 redirects ip arp timeout 1740 hsrp version 2 hsrp 803 …

hsrp 803 ipv6 authentication md5 key-‐chain hsrp_auth preempt delay minimum 600 priority 110 forwarding-‐threshold lower 0 upper 0 ip autoconfig track 1 decrement 20 no shutdown

Operational Challenges

12

Image support (SLAAC + dhcpv6 Stateless) – CentOS (6,7) RHEL (6,7) W2K8 Ubuntu(Trusty) – SLAAC supported by all – DHCP client behavior needed fix for CentOS,

RHEL and Ubuntu

IPv6 guest enablement criteria

– Incorrect usage of host level sysctl flags – Gating criteria has no effect on IPv6 traffic

forwarding

IPv6 system of record inconsistencies

– Dual-stacking an existing IPv4-only network – IPv6 SLAAC on the pre-existing IPv4 VMs

– SOR inconsistencies + missing IPv6 fire-walling

Subnet validation between v4 and v6

– force_gateway_on_subnet flag

– Incorrect validation for IP in Subnet of an IPv4 scheme against an IPv6 address

Compute host – Stability failures e.g. tap interfaces DOWN

“Use the force, Read the source”

Scale Testing Tools and Process

13

Objectives

– Test bed of 4K interfaces for IPv4 &IPv6 each – Generate ICMP/ICMP6 traffic for the interfaces – Test dhcp-agent (dnsmasq) resiliency – Test metada-agent resiliency – Test bed stability over a period of time

Scenarios

– 2K dual stack, dual vNic VMs – 50 concurrent VM boots/reboots – Total of 8K interfaces across 3 networks

performing dhcp offers/acks, dns-info, dhcp renewals

– ICMP/ICMP6 across all 8K interfaces Process/Tools

– iPerf scripts – In-house scale test python package – Supports concurrent operations – VM boot, reboot, icmp/6, discovery, console_log – verification for cloud-init customizations

Scale Testing Topology / Hardware Specs

14

OpenStack Controllers ✓ 35 Controller VMs on Service Cloud C240 servers

Compute Capacity (approx single core vCPU per Node) ✓ 2 x B200s (2x10 Physical CPU, 256G Mem) ✓ 16 x C220 (2x10 Physical CPU, 256G Mem) ✓ Oversubscription ratios (4.0x cpu, 1.5x ram)

Network Nodes ✓ 4 x B200M3 (all neutron agents) ✓ DHCP agents per network 2 ✓ 3 provider vlan networks (IPv4, IPv6)

Ceph Cluster (Shared) ✓ 3 x C220 for Mon ✓ 3 x C220 Rados GW ✓ 6 x C240 OSDs

Test VM Configuration ✓ Flavor Specs (1 vCPU, 1G RAM, 5G Disk) ✓ Image (cirros-‐0.3.3-‐x86_64-‐disk) ✓ 2 vNics per Dual Stack VM

Scale Testing - VM distribution / Boot times

15

Performance and Scale - PING

16

IPv4 Min Response Time

Res

pons

e Ti

me

(ms)

0

0.065

0.13

0.195

0.26

Number of VMs0 25 50 75 100

IPv4 Avg Response Time

Res

pons

e Ti

me

(ms)

0

0.15

0.3

0.45

0.6


IPv4 Max Response Time

Res

pons

e Ti

me

(ms)

0

2.5

5

7.5

10


IPv6 Min Response Time

Res

pons

e Ti

me

(ms)

0

0.065

0.13

0.195

0.26


IPv6 Avg Response TimeR

espo

nse

Tim

e (m

s)

0

0.15

0.3

0.45

0.6


IPv6 Max Response Time

Res

pons

e Ti

me

(ms)

0

2.5

5

7.5

10


IPv4 v.s. IPv6 Throughput Within A Compute Node

17

Note: All tests were run for 200 secs based on 1470 payload size

Average IPv4 TCP throughput*: 13.1 Gbits/sec


IPv4 UDP Throughput

Thro

ughp

ut (M

bits

/sec

)

0

175

350

525

700

Number of Samples0 25 50 75 100

IPv6 UDP Throughput

Thro

ughp

ut (M

bits

/sec

)

0

175

350

525

700


Average IPv4 UDP throughput*: 648 Mbits/sec



IPv6 TCP Throughput

Thro

ughp

ut (G

bits

/sec

)

0

4

8

12

16


IPv4 TCP Throughput

Thr

ough

put (

Gbi

ts/s

ec)

0

4

8

12

16


IPv4 v.s. IPv6 Throughput Between Two Compute Nodes

18

IPv4 TCP Throughput

Thro

ughp

ut (G

bits

/sec

)

0

2.5

5

7.5

10


IPv6 TCP Throughput

Thro

ughp

ut (G

bits

/sec

)

0

2.5

5

7.5

10





IPv4 UDP Throughput

Thro

ughp

ut (M

bits

/sec

)

0

200

400

600

800


IPv6 UDP Throughput

Thro

ughp

ut (M

bits

/sec

)

0

200

400

600

800





Value Adding to Icehouse Release

19

Filled Feature Gaps

– API validation on Neutron Server is not adequate

– DHCPv6 Guard

Bridged Gaps on Unit Test – Dnsmasq process launch for IPv6 subnet in

SLAAC mode was not included

– Security group and ip6table rules were not verified for DHCPv6 Stateless mode

Fixed bugs

– DHCPv6 Stateful and DHCPv6 Stateless modes are treated the same

– DHCP agent cannot reload dnsmasq process properly during subnet addition/deletion

– IPv6 default route is still statically inserted

Enhanced IPv6 Testing Capability – A total of 22 Tempest functional/API/negative

test cases

– A total of 14 Rally scalability/performance test scenarios

We will contribute back to the community!

Lessons and LearnsCommunity provides good reference architecture to the adopter. However, customization maybe be required

Security, performance/scalability and operations should be taken into the consideration as part of the design

Process doesn’t always introduce overhead. Right SDLC process can provide the quality assurance

We need think about how IPv6 solves a problem, NOT how to solve the problem of IPv6

We invite YOU to share YOUR lessons and learns, instead of features and functionalities, to accelerate the adoption of IPv6

20

Next StepsOpenStack tenant networking

Native L3 connectivity between tenant networks without the need for NAT

Direct routing to Internet from Tenant networks without the need for NAT

Allow tenant to choose between deploying IPv6 only, IPv4 only or Dual Stack networks

Allow cloud provider to centrally manage tenant IPv6 address space – we do not have the problem of overlapping IP address space with IPv6

Allow multiple prefixes

Allow private interconnects between tenant network in cloud and tenant’s own enterprise network

21

Thank you!

Date post:	08-Aug-2015
Category:	Internet
Upload:	shixiong-shang
View:	162 times
Download:	0 times