Date post: | 23-Dec-2014 |
Category: |
Documents |
Upload: | openstack-foundation |
View: | 551 times |
Download: | 1 times |
Openstack@eBayPractical SDN Deployment using Quantum
Not a public cloud, but …
Copyright eBay Inc. 2012 2
QADEV
Prod
PCI
QA DEV
DEVQA
Secure
Prod
Copyright eBay Inc. 2012 3
Principles Any Application Anywhere
Dedicated physical environments cause fragmentation Soft Cabling
Datacenter reconfiguration is costly and cannot be automated Shared Standardized Infrastructure
Simplifies automation and improves supply chain efficiency Virtualize everything
White space between applications and infrastructure helps agility Automate everything
Automation helps agility and efficiency
Copyright eBay Inc. 2012 4
Class of Service
• Translation of physical environment properties into configurations• Assigned to projects (logical environments), drives scheduling and policies
• For example, network selection
Obligations Restrictions Capabilities
QA Approved Builds No Login Access Core DB access
Prod OS version No Corp Access 24/7 Incident Mgt
Monitoring No QA Access Site traffic Access
Production
Obligations Restrictions Capabilities
Certified OS versions Limited Prod Access
Full root
Limited QA Access
No site Traffic Filtered Internet
DEV
Obligations Restrictions CapabilitiesNo Prod Access Private DB
Certified OS Versions No Corp Access 24/7 Incident Mgt
Monitoring No QA Access Site traffic Access
External
Copyright eBay Inc. 2012 5
Infrastructure designed for scale
Spine
Leaves
Core
M servers2x1Gb
N leaves
4 spines
Flat L3 (all switches are routers too)Line rate from any server to any server (oversubscription = 48/40)OSPF/ECMP to advertise routes
(48x1Gb)
(Nx10Gb)
48 -> N “½ racks”
Copyright eBay Inc. 2012 6
Isolation options: L2
Production QA
VLAN trunk
vlan 1
vlan nQA
Prod
Dedicated Network VLAN Based
- physical network build out- Fragmentation- coarse grained isolation
- Limited scale (n = 4096) - Large fault domain (STP)
+ Physical isolation+ fool proof
+ L2 isolation+ somewhat soft Cabling
Copyright eBay Inc. 2012 7
Isolation options: L3 with Security Groups
Security Groups or Virtual Firewall
- Difficult to combine provider policies and user policies- Management of rules- Impact of group membership modification- Aggregation/summarization difficult/impossible
+ no/minimal infrastructure requirement+ good for user policies (ip tables)
Copyright eBay Inc. 2012 8
Isolation Options: Virtual L2 Networks
Cloud Fabric
QA
ProdOther
Networks
+ L2 isolation+ compatible with large scale networks+ can be fully automated+ firewall can be interposed betweenvirtual networks
Virtual Networks using Software Defined Networks
+ Can complement L3 isolation+ large number of networks (n>4096)- Tunnel overhead- L2 size limited by # of tunnels and their mgt
Overlay 1
Overlay n
Copyright eBay Inc. 2012 9
ControllerThe Switch/Router
All you need to know about SDN
The Network
Routing/switching engine
Logic
controls
Netw
ork protocols
Traditional SDN
The Network
The Switch/Router
Routing/switching engine
Logic
controls
Netw
ork protocols
API
Copyright eBay Inc. 2012 10
SDN ‘levels’
Nerdy
Wizard
Ninja
Virtual SwitchesOverlay Networks
Physical SwitchesTraffic Engineering
ARP + L2 protocols
OSPF/ECMP,…
Virtual + Physical switchesOverlay Networks
Copyright eBay Inc. 2012 11
Dev Cos : aka Dev Cloud A logical environment defined as a class of service on top of shared infrastructure
Self Service VM for developers. Access must be similar to their desktops (access to QA, Corp, …) Should allow collaboration
Implemented as a set of L2 networks (/24) with in a given L3 (/20) No private networks : all developers on same shared networks No private IP space: traffic is routed within core, no need for floating Ips
Isolated from infrastructure Overlay network using OpenVswitch / STT tunneling Nicira NVP controllers integrated with Quantum (Essex) Routed out through perimeter firewall
Copyright eBay Inc. 2012 12
Niciracontrollers
NiciraServiceNodes
N
vswitch
gtw-xxxxM
Active Gateway
vswitch
HypervisorC
Eth1/vlan 1
vif
Eth0/vlan 2
S Q
Eth1/vlan 1
Eth0/vlan 2
N
vswitch
gtw-xxxx
gtw-xxxxM
Standby Gateway
10.9.0.110.9.0.1010.9.1.0/24
10.9.2.0/24
10.9.1.1
10.9.2.1
NiciraServiceNodes
Niciracontrollers
Corp
Internet
QA
Dev Cloud : 10.9.0.0/20
trunk
10.9.0.0/20 ->10.9.0.10From 10.9.1.0/24 default->10.9.0.1From 10.9.2.0/24 default->10.9.0.1
default->10.9.2.1
K AN:Nova-network+dnsmasqC:Nova-computeS:Nova-schedulerM:Metadata
K:Ubuntu + KVMA:Nova-apiQ:Quantum
Infrastructure/Internal
Infrastructure/External
Virtual network
Copyright eBay Inc. 2012
13
eBay IaaS
eBay Cloud Portal
Create instance (COS,OS, size)
DNSManagement
Create DNS (A,PTR)
Nova API
Boot Instance(Image ID,Flavor, NIC)
Get Free Networks
1
2
34
Nova Network
Nova-manage
Create network(project = admin,Cidr=10.9.x.0/24)
novadb
Quantum
Nicira Controller
Create lswitch
Create port
Nova Compute
Nova Scheduler
Create routes
Gateway
Create gtw-xxxx
AdminDeveloper
Get IPCreate port
Copyright eBay Inc. 2012 14
0
50
100
150
200
250
0102030405060708090100
SuccessFailedrate
Instance Requests
15
What works/What doesn’t
Good Perimeter firewalls configured once, not
dependent on the instance creation/deletion/movement
Network are pre-created using nova-manage, good for provider networks
Can be extended with other COS using same pattern
Stability of both Nicira NVP and Openstack + Ubuntu + KVM
Looking forward to new features in Folsom – Quantum v2
Bad No capacity/policy based assignment of
networks – had to be implemented outside. Moving it to nova scheduler.
One network flavor supported in Essex. Cannot have, e.g., one gateway per network, with different behavior (dhcp)
Scale out requires bigger links out of the gateway, or more gateways
Upset the separation of concern requirement: Netsec + Networking + Sys Admins in same box = ‘interesting’
16
What’s Next New classes of service
External : private networks + VIP and Floating IP on the Internet Production : Bridged network
Scale out 80 today, going to a lot more More gateways/10Gb
Folsom upgrade L3 Routers Load Balancers
Cleaner Openstack integration Network Allocation DNS configuration AuthN/AuthZ
Copyright eBay Inc. 2012 17
We are Hiring !
http://www.ebaycareers.com/