www. chameleoncloud.org
AP RIL 6 , 2 0 1 8 1
EXPERIMENTING WITH VIRTUAL SDXS USING CHAMELEON AND EXOGENI
Paul RuthRENCI – University of North Carolina
www. chameleoncloud.org
OUTLINE
� Background� ExoGENI testbed (wide footprint edge cloud)
� NSF Cloud Chameleon testbed (mid-scale cloud)
� Experiments Spanning Testbeds � Inter-slice stitching
� Campus stitching
� Inter-testbed stitching
� Software Defined eXchange (SDX) Experiments� SAFE SDX (RENCI, DUKE, US DOE/Esnet)
� SciDAS (Clemson, RENCI, Washington State University)
www. chameleoncloud.org
OUTLINE
� Background� ExoGENI testbed (wide footprint edge cloud)
� NSF Cloud Chameleon testbed (mid-scale cloud)
� Experiments Spanning Testbeds � Inter-slice stitching
� Campus stitching
� Inter-testbed stitching
� Software Defined eXchange (SDX) Experiments� SAFE SDX (RENCI, DUKE, US DOE/Esnet)
� SciDAS (Clemson, RENCI, Washington State University)
www. chameleoncloud.org
NSF GLOBAL ENVIRONMENT FOR NETWORK INNOVATIONS (GENI)
Virtual laboratory for networking and distributed systems research and education
www. chameleoncloud.org
GENI FEDERATION� Federated identity
� InCommon
� X.509 identity certificates
� Common APIs� Aggregate Manager
� Clearinghouse
� Agreed upon resource description language� RSpec
� ExoGENI translates relevant portions from NDL-OWL to RSpec and back as needed
� Several major portions� ExoGENI, InstaGENI, WiMax, Internet2 AL2S, ESnet
� Federation with EU FIRE effort
www. chameleoncloud.org
Cloud Providers
Virtual Compute and Storage Infrastructure
Network Transit Providers
Cloud APIs (Amazon EC2 ..) Network Provisioning APIs (DOE ESNetOSCARS, Internet2, OESS, OGF NSI …)
Virtual Network Infrastructure
EXOGENI
www. chameleoncloud.org
Mutually Isolated Virtual Networks
VM VM
VM VM
VM VM
VM VM
VM VM
VM VM
VM VM
VM VM
VM VM
VM VM
VM VM
VM VM
Edge Providers(Compute Clouds and Network Providers)
Mutually Isolated Slicesof Virtual Resources
Workflows
EXOGENI
www. chameleoncloud.org
EXOGENI
� Relationship to GENI� One of two computational testbeds built for GENI
� Implements GENI API
� Accepts GENI users
� Notable features:� Wide scale footprint (20 sites)
� Edge clouds (OpenStack)
� Dynamic layer 2 circuits between sites
� Stitchports: layer 2 connections to external resources
� Limitations� Small scale computational sites
� No core network control
www. chameleoncloud.org
EXOGENITOPOLOGY
www. chameleoncloud.org
EXOGENITOOLS
www. chameleoncloud.org
EXOGENI: STITCHING
www. chameleoncloud.org
OUTLINE
� Background� ExoGENI testbed (wide footprint edge cloud)
� NSF Cloud Chameleon testbed (mid-scale cloud)
� Experiments Spanning Testbeds � Inter-slice stitching
� Campus stitching
� Inter-testbed stitching
� Software Defined eXchange (SDX) Experiments� SAFE SDX (RENCI, DUKE, US DOE/Esnet)
� SciDAS (Clemson, RENCI, Washington State University)
www. chameleoncloud.org
CHAMELEON PHASE 1 IN A NUTSHELL� Deeply reconfigurable: “As close as possible to having it in your lab”
� Deep reconfigurability (bare metal) and isolation
� Power on/off, reboot from custom kernel, serial console access, etc.
� But also – modest KVM cloud for ease of use
� Large-scale: “Big Data, Big Compute research”
� ~650 nodes (~15,000 cores), 5 PB of storage distributed over 2 sites connected with 100G network…
� …and diverse: ARMs, Atoms, FPGAs, GPUs, etc.
� Blueprint for a sustainable production testbed: “cost-effective to deploy, operate, and enhance”
� Powered by OpenStack with bare metal reconfiguration (Ironic)
� Open production testbed for Computer Science Research
� Project started in 10/2014, testbed available since 07/2015
� Currently 1,600+ users, 300+ projects
www. chameleoncloud.org
CHAMELEON: PHASE 1 HARDWARE
SCUs connect tocoreandfullyconnected toeachother
HeterogeneousCloudUnits
ARMs,Atoms,lowpowerXeions, FPGAs,GPUs,SSDs, etc.
SwitchStandardCloudUnit42compute4storagex10
Chicago
To UTSA, GENI, Future Partners
AustinChameleonCoreNetwork
100Gbps uplink publicnetwork(eachsite)
CoreServices3.6PBCentralFileSystems, FrontEndandDataMovers
CoreServicesFrontEndandData
MoverNodes 504x86ComputeServers48Dist.StorageServers102HeterogeneousServers16Mgt andStorageNodes
SwitchStandardCloudUnit42compute4storagex2
www. chameleoncloud.org
NEW HARDWARE� 4 new Standard Cloud Units (32 node racks in 2U chassis)
� 3x Intel Xeon “Sky Lake” racks (2x @UC, 1x @TACC)
� 1x future Intel Xeon rack (@TACC) in Y2
� Corsa DP2000 series switches� 2x DP2400 with 100Gbps uplinks (@UC)
� 1x DP2200 with 100Gbps uplink (@TACC)
� Each switch will have a 10 Gbps connection to nodes in the SCU
� Optional Ethernet connection in both racks
� More storage configurations� Global store @UC: 5 servers with 12x10TB disks each
� Additional storage @TACC: 150 TB of NVMes
� Accelerators: 16 nodes with 2 Volta GPUs (8@UC, 8@TACC)
� Maintenance, support and reserve
www. chameleoncloud.org
NEW HARDWARE� 4 new Standard Cloud Units (32 node racks in 2U chassis)
� 3x Intel Xeon “Sky Lake” racks (2x @UC, 1x @TACC)
� 1x future Intel Xeon rack (@TACC) in Y2
� Corsa DP2000 series switches� 2x DP2400 with 100Gbps uplinks (@UC)
� 1x DP2200 with 100Gbps uplink (@TACC)
� Each switch will have a 10 Gbps connection to nodes in the SCU
� Optional Ethernet connection in both racks
� More storage configurations� Global store @UC: 5 servers with 12x10TB disks each
� Additional storage @TACC: 150 TB of NVMes
� Accelerators: 16 nodes with 2 Volta GPUs (8@UC, 8@TACC)
� Maintenance, support and reserve
www. chameleoncloud.org
CORSA DP2000 SERIES SWITCHES� Hardware Network Isolation
� Sliceable Network Hardware
� Tenant controlled Virtual Forwarding Contexts (VFC)
� Software Defined Networking (SDN)� OpenFlow v1.3
� User defined controllers
� Performance� 10 Gbps within a site
� 100 Gbps between UC/TACC (Aggregated)
www. chameleoncloud.org
StandardCloudUnit
NETWORK HARDWARE
Chicago
Internet 2 AL2S, GENI, Future Partners
Austin
ChameleonCoreNetwork100Gbps uplink publicnetwork
(eachsite)
StandardCloudUnit
Corsa DP2400Corsa DP2400
StackedSwitches(LogicallyOne)
StandardCloudUnit
Corsa DP2200
100Gbps(Aggregate)
100Gbps(Aggregate)
www. chameleoncloud.org
ISOLATED VIRTUAL SDN SWITCH� Isolated Tenant Networks
� BYOC– Bring your own controller: isolated user controlled virtual OpenFlowswitches (coming soon)
StandardCloudUnit
Corsa Switch
ComputeNode
(TenantA)
ComputeNode
(TenantA)
ComputeNode
(TenantB)
ComputeNode
(TenantB)
VFC(TenantA)
VFC(TenantB)
OpenFlowController(TenantB)
OpenFlowController(TenantA)
Ryu
www. chameleoncloud.org
StandardCloudUnit
CHAMELEON: SDN EXPERIMENTS
� Chameleon Networking
� RENCI added to the team� Hardware Network Isolation
� Corsa DP2000 series
� OpenFlow v1.3 � Sliceable Network Hardware� Tenant controlled Virtual Forwarding
Contexts (VFC)
� Isolated Tenant Networks� BYOC – Bring your own controller
� Wide-area Stitching
� Between Chameleon Sites (100 Gbps)� ExoGENI� Campus networks (ScienceDMZs)
CorsaDP2400Switch
Internet 2 AL2S, GENI, Future Partners
Chicago
Austin
ComputeNode
(TenantA)
OpenFlowController(TenantB)
OpenFlowController(TenantA)
Ryu
VFC(TenantA)
ComputeNode
(TenantA)
ChameleonCoreNetwork100Gbps uplink publicnetwork
ComputeNode
(TenantB)
ComputeNode
(TenantB)
VFC(Tenantb)
www. chameleoncloud.org
OUTLINE
� Background� ExoGENI testbed (wide footprint edge cloud)
� NSF Cloud Chameleon testbed (mid-scale cloud)
� Experiments Spanning Testbeds � Inter-slice stitching
� Campus stitching
� Inter-testbed stitching
� Software Defined eXchange (SDX) Experiments� SAFE SDX (RENCI, DUKE, US DOE/Esnet)
� SciDAS (Clemson, RENCI, Washington State University)
www. chameleoncloud.org
EXOGENI: INTER-SLICE STITCHING
www. chameleoncloud.org
EXOGENI: INTER-SLICE STITCHING
www. chameleoncloud.org
Public Internet
EXOGENI: INTER-SLICE STITCHING
www. chameleoncloud.org
EXOGENI: INTER-SLICE STITCHING
Starlight
www. chameleoncloud.org
EXOGENI: INTER-SLICE STITCHING
Starlight
Service Slice Client Slice
www. chameleoncloud.org
OUTLINE
� Background� ExoGENI testbed (wide footprint edge cloud)
� NSF Cloud Chameleon testbed (mid-scale cloud)
� Experiments Spanning Testbeds � Inter-slice stitching
� Campus stitching
� Inter-testbed stitching
� Software Defined eXchange (SDX) Experiments� SAFE SDX (RENCI, DUKE, US DOE/Esnet)
� SciDAS (Clemson, RENCI, Washington State University)
www. chameleoncloud.org
EXOGENITO CAMPUS STITCHING
IPcore(L3) Circuit fabric
providers
SDSN
GENI
Phys
DMZDTN
IPegress
BEN
I2/A2LS
ESnet
OtherGENIsitesOthercampusesOtherfacilities
ControlplaneAPIs
Dukecampusboundary
L2egress
e.g.GENI-APIe.g.OSCARSe.g.Plexuse.g.ORCA
Duke University Software Defined Science Network (SDSN)Science DMZ
www. chameleoncloud.org
EXOGENITO CAMPUS STITCHING
Stitchport: Named meeting point linking a layer 2 circuit between ExoGENI and
external resources.
Stitchport Duke SDSN
www. chameleoncloud.org
MULTI-TESTBED EXPERIMENTS
Starlight
Client SliceService Slice
Client Campus
www. chameleoncloud.org
OUTLINE
� Background� ExoGENI testbed (wide footprint edge cloud)
� NSF Cloud Chameleon testbed (mid-scale cloud)
� Experiments Spanning Testbeds � Inter-slice stitching
� Campus stitching
� Inter-testbed stitching
� Software Defined eXchange (SDX) Experiments� SAFE SDX (RENCI, DUKE, US DOE/Esnet)
� SciDAS (Clemson, RENCI, Washington State University)
www. chameleoncloud.org
CHAMELEON TO EXOGENI STITCHING
• Dynamic VLANs• Connectivity to
ExoGENI Stitchport
• ExoGENI slice• Dynamic Chameleon
Stitchport
Stitched L2 path
www. chameleoncloud.org
CHAMELEON TO EXOGENI STITCHING
StitchPort
Stitching between ExoGENI and Chameleon nodes
www. chameleoncloud.org
INTER-TESTBED EXPERIMENTS
Starlight
Client SliceService Slice
Client Campus
www. chameleoncloud.org
INTER-TESTBED EXPERIMENTS
Starlight
Service Slice Client Slice
Client CampusClient Slice
www. chameleoncloud.org
INTER-TESTBED EXPERIMENTS
Starlight
Service Slice Client Slice
Client CampusClient Slice
www. chameleoncloud.org
INTER-TESTBED EXPERIMENTS
Starlight
Virtual SDXService Slice Client Slice
Client CampusClient Slice
www. chameleoncloud.org
OUTLINE
� Background� ExoGENI testbed (wide footprint edge cloud)
� NSF Cloud Chameleon testbed (mid-scale cloud)
� Experiments Spanning Testbeds � Inter-slice stitching
� Campus stitching
� Inter-testbed stitching
� Software Defined eXchange (SDX) Experiments� SAFE SDX (RENCI, DUKE, US DOE/Esnet)
� SciDAS (Clemson, RENCI, Washington State University)
www. chameleoncloud.org
CICI SAFE PROJECT
“Creating Dynamic Superfacilities the SAFE Way”Paul Ruth, Cong Wang, Mert Cevik, RENCI
Jeff Chase, YuanjunYao, Qiang Cao, Victor Orlikowski. Charley Kneifel, Duke Univeristy
Nick Buraglio, ESnet
NSF CICI Award #1642142
www. chameleoncloud.org
SUPERFACILITY
� Definition� Two or more existing facilities (e.g. instruments, compute resources, data repositories) using
high-performance networks and data management software in order to increase scientific output.
� Currently manually created � Superfacilities are purpose-built manually for a specific scientific application or community.
� Trust: “handshake model”
� Ideally automated� Advanced Science DMZs and federated Infrastructure-as-a-Service provide the technical
building blocks to construct dynamic superfacilities on demand.
www. chameleoncloud.org
SUPERFACILITY
� Definition� Two or more existing facilities (e.g. instruments, compute resources, data repositories) using
high-performance networks and data management software in order to increase scientific output.
� Currently manually created� Superfacilities are purpose-built manually for a specific scientific application or community.
� Trust: “handshake model”
� Ideally automated� Advanced Science DMZs and federated Infrastructure-as-a-Service provide the technical
building blocks to construct dynamic superfacilities on demand.
Trust also needs to be automated
www. chameleoncloud.org
SUPERFACILITIES THE SAFE WAYDuke Science DMZ Other Campus
ExoGENI Slice
IDS IDS IDS IDS
DTN
Virtual SDX
• Automating Superfacilites– Multiple domains– Friction free L2 paths
• Naked L2 paths are not secure– Handshake model of trust is not possible
• Virtual SDX (vSDX)– Distributed– Enforces SDX connectivity policy– Enforces client’s forwarding policy
(security, BGP, etc.)– Intrusion Detection System (Bro)
• SAFE: Secure Authorization for Federated Environments– Isolates applications from logic concerns
• Certificate discovery (DAGs)• Logic inference• Cryptography
– Logic scripting language• Slang (SAFE Language)• Based on Datalog
– Shared certificate repository• Stores statements and DAGs
www. chameleoncloud.org
SCIDAS
1PBStge/FIONA 1PBStge./FIONA 1PBStge./FIONA
Cost-AwareOptimize
iRODSShim (aaS)
API
PerfSONARShim (aaS)
API PerfSONARmapping
Requester
Orchestrator
Network
www. chameleoncloud.org
SCIDAS
Automated vSDX superfacility