Automated Platform for Testing VNF Performance and Interoperability with
Variable Flavors
Takashi Torii NEC CorporationTatsuya Tohyama, Okinawa Open Laboratory/OKIT
Vision: Okinawa Smart Hub
• Okinawa Prefectural Government vision and policy for industrial development and promotion • Attract companies, talents and knowledge from other prefectures in Japan and overseas,
especially Asian countries, and promote new value creation in Okinawa• Form major international ICT hub in Asia
Okinawa Smart Hub and OOL
• Contribute to realizing Okinawa Smart Hub concept• Based in Okinawa with support from Okinawa Prefectural Government, attracting leading
companies, organizations, educational institutions, and research bodies from other prefectures in Japan and overseas, especially Asia, and promoting international R&D activities
• Contributes to promoting concentration of ICT industries and advancement of regional companies
SDN technology research and standardization base
OSS cloud development community
Leading Linux and OSS organization
OSS SDN development project
Okinawa Open Laboratory
Practical-use research base that integrates cloud technology and SDN technology
International R&D organization
Leading companies and organizationsfrom other prefectures in Japan
and overseas, especially Asia
Technology cooperation
Cooperation, attraction
Activity support
・Support for R&D activities・Promotion of cloud DC facility use
Mission
• Research and development activities for practical application and dissemination of advanced Information Communications Technologies (ICT), especially next-generation ICT platform technologies, such as
• Software-Defined Networking (SDN)• Cloud Computing using Open Source Software (OSS, especially Open Stack)
• Open Innovations
• Collaboration, Joint Work
• Bridging the gaps between users and providers
Activity Model
• OOL connects OSS development communities and market via use cases and reference models to expand user base of next-generation ICT platform technologies in cooperation with MEMBER and partners
• OOL Value Chain: Use Case à Verification à Ref. Model à Promotion• Feedback Loops: (1) OOL ó Market, (2) OOL ó OSS Dev. Communities
OSS Bodies (Community/Consortium)
Develop
Use Case Verify Promote
Issues/Needs
Apply/Use
OOL and its Members
Market (Providers and End Users)
Expand User Base
ReferenceModel
Testbed (OpenStack with SDN)
Project Formations
FullTimeEngineers
Organizer
CompaniesResearchInstituteAcademia
• Planningprojectsandgoals• Communicatewithcompanies• CommunicatewithBoards,Staffsand
Stakeholders(Governmentetc)• Managetheactivities• Promoteandcollaborate
Staffs Board
Dev Project Example: OF-Patch
• Patch panel function implemented using OpenFlowtechnology
OFPMUser
Change network topology by remote operation
OF-PatchGUI
OFC
OFS Patch by flow
Host1 Host2 Host3
OPNFV Test Lab
• VNF Test Automation
• Service Function Chaining
• Joint POC with oversea members
Internet
JumpServer
POD#1(Fuel)
POD#2(RDO)
Background
• Market of VNF is now emerging• ETSI proposed NFV architecture• Open Source VNF
• VyOS (Vyatta)
• Open vSwitch…etc
• Vender VNF• Cisco CSR1000V• Juniper FireFly, vSRX• Brocade Vyatta Core
…etc
• But we don‘t have test automation platform of VNF
Background
• Problem of VNF testing• There is no standard• Performance and stability is depend on flavors• Many test patterns are required than H/W appliance
• Impossible to cover all test patterns by manual• Risk of human error or miss configuration
• Configurations are depend on vendors• Need support from venders or skilled engineers in all phase
• To solve these problems, we have developed VNF automation testing system
How to test VNF?
• Workflow• Deploy
• Deploy VNF on VIM with flavor• Initialize VNF
• Configuration• Configure and confirm• Check the network connectivity with other devices
• Run• Subscribe test patterns and run on traffic generator.• Check VNF status and behavior
• Reporting• Writing report based on result
Test workflow
• Benefit of automation• Engineer can focus on troubleshooting• Engineers who don’t have much skills can operate the tests• High repeatability
Repeat
TestDeployVNFInitialize Initialize
VNFConfig Delete
VNFLog
ArchiveAnalysisReportCheck
ArchitecturePoC User
Orchestrator
Web-Portal
Web-Portal
Report generationManagement of Test
REST API
Interoperability Traffic Test
REST API
Virtual Appliance Ctr(Deploy of VNF appliance)
NW Device Ctr(Insert/delete of configure)
Tester Ctr(Control of IXIA)
ArchitecturePoC User
Orchestrator
Web-Portal
Web-Portal
Report generationManagement of Test
REST API
Interoperability Traffic Test
REST API
Virtual Appliance Ctr(Deploy of VNF appliance)
NW Device Ctr(Insert/delete of configure)
Tester Ctr(Control of IXIA)
Web Portal
• Set the test parameters• Select the test scenario from drop down lists
Orchestrator
Web-Portal
REST API
PoC User
Test Parameter
ArchitecturePoC User
Orchestrator
Web-Portal
Report generationManagement of Test
REST API
Interoperability Traffic Test
REST API
Virtual Appliance Ctr(Deploy of VNF appliance)
NW Device Ctr(Insert/delete of configure)
Tester Ctr(Control of IXIA)
Web-Portal
Orchestrator
• Controller management• Controls each components based on the configuration of test scenarios.
• Creating test scenario• Create test scenarios by GUI
• Reporting• Output PDF format
Scenario generate tool
ArchitecturePoC User
Orchestrator
Web-Portal
Report generationManagement of Test
REST API
Interoperability Traffic Test
REST API
Virtual Appliance Ctr(Deploy of VNF appliance)
NW Device Ctr(Insert/delete of configure)
Tester Ctr(Control of IXIA)
Web-Portal
ArchitecturePoC User
Orchestrator
Web-Portal
Report generationManagement of Test
REST API
Interoperability Traffic Test
REST API
Virtual Appliance Ctr(Deploy of VNF appliance)
NW Device Ctr(Insert/delete of configure)
Tester Ctr(Control of IXIA)
Web-Portal
Virtual Appliance Controller
• Multi virtualized platform control• Deploy virtual appliance on selected virtualized platform.
Orchestrator
REST-API
REST-MessageOpenStack Link
Virtual Appliance Controller
REST-API REST-API
ArchitecturePoC User
Orchestrator
Web-Portal
Report generationManagement of Test
REST API
Interoperability Traffic Test
REST API
Virtual Appliance Ctr(Deploy of VNF appliance)
NW Device Ctr(Insert/delete of configure)
Tester Ctr(Control of IXIA)
Web-Portal
NW Device Controller
• Multi vendor control• Provides the ability to control multiple vendor VNF
Orchestrator
NW Device Controller
REST-API
Telnet/SSH/Netconf
REST-MessageParameter Ex
・Vender Name・Vender Version・Test scenario
other…
ArchitecturePoC User
Orchestrator
Web-Portal
Report generationManagement of Test
REST API
Interoperability Traffic Test
REST API
Virtual Appliance Ctr(Deploy of VNF appliance)
NW Device Ctr(Insert/delete of configure)
Tester Ctr(Control of IXIA)
Web-Portal
Tester Controller
• Generating Tcl script• Based on the parameters set by REST API from Orchestrator
• Ixia Driver• Load Tcl Script by IXIA API.
Orchestrator
Tester Controller
REST-MessageParameterEx
・InterfaceInformation・PacketSize(Short,Long,Imix)・Framerateother...
Test Traffic
IXIA API
REST API
Tcl Script
Test workflow
Repeat
VirtualApplianceController
TesterControllerNWDeviceController
Orchestrator
TestDeployVNFInitialize Initialize
VNFConfig Delete
VNFLog
ArchiveAnalysisReportCheck
Our environment
VyOS
vSwitch
vnic0
vSwitch
Interop
erability
L2S
W
Traffic Test L2S
W
vnic1
vnic1
OpenStack (node2)
VMware Esxi (node1)
VMware Esxi (node3)
OpenStack (node4)
vnic2
vnic3
vnic1
vnic2
vnic3
MGMT-Seg
vnic0
vnic0
vSwitch
JuniperFirefly
CiscoCSR1000v
Vyatta5600
vnic0
vSwitchvnic1
VyOS
JuniperFirefly
CiscoCSR1000v
Vyatta5600
Our environment for interoperability
VyOS
vSwitch
vnic0
vSwitch
Interop
erability
L2S
W
Traffic Test L2S
W
vnic1
vnic1
OpenStack (node2)
VMware Esxi (node1)
VMware Esxi (node3)
OpenStack (node4)
vnic2
vnic3
vnic1
vnic2
vnic3
MGMT-Seg
vnic0
vnic0
vSwitch
JuniperFirefly
CiscoCSR1000v
Vyatta5600
vnic0
vSwitchvnic1
VyOS
JuniperFirefly
CiscoCSR1000v
Vyatta5600
Our environment for interoperability
VyOS
vSwitch
vnic0
vSwitch
Interop
erability
L2S
W
Traffic Test L2S
W
vnic1
vnic1
OpenStack (node2)
VMware Esxi (node1)
VMware Esxi (node3)
OpenStack (node4)
vnic2
vnic3
vnic1
vnic2
vnic3
MGMT-Seg
vnic0
vnic0
vSwitch
JuniperFirefly
CiscoCSR1000v
Vyatta5600
vnic0
vSwitchvnic1
VyOS
JuniperFirefly
CiscoCSR1000v
Vyatta5600
VNFappliancetestenvironment
VNFappliancetestenvironment
Our environment for interoperability
VyOS
vSwitch
vnic0
vSwitch
Interop
erability
L2S
W
Traffic Test L2S
W
vnic1
vnic1
OpenStack (node2)
VMware Esxi (node1)
VMware Esxi (node3)
OpenStack (node4)
vnic2
vnic3
vnic1
vnic2
vnic3
MGMT-Seg
vnic0
vnic0
vSwitch
JuniperFirefly
CiscoCSR1000v
Vyatta5600
vnic0
vSwitchvnic1
VyOS
JuniperFirefly
CiscoCSR1000v
Vyatta5600
VNFapplianceforinteroperabilitytesting
VNFapplianceforinteroperabilitytesting
Our environment for interoperability
VyOS
vSwitch
vnic0
vSwitch
Interop
erability
L2S
W
Traffic Test L2S
W
vnic1
vnic1
OpenStack (node2)
VMware Esxi (node1)
VMware Esxi (node3)
OpenStack (node4)
vnic2
vnic3
vnic1
vnic2
vnic3
MGMT-Seg
vnic0
vnic0
vSwitch
JuniperFirefly
CiscoCSR1000v
Vyatta5600
vnic0
vSwitchvnic1
VyOS
JuniperFirefly
CiscoCSR1000v
Vyatta5600
Our environment for Traffic Test
VyOS
vSwitch
vnic0
vSwitch
Interop
erability
L2S
W
Traffic Test L2S
W
vnic1
vnic1
OpenStack (node2)
VMware Esxi (node1)
VMware Esxi (node3)
OpenStack (node4)
vnic2
vnic3
vnic1
vnic2
vnic3
MGMT-Seg
vnic0
vnic0
vSwitch
JuniperFirefly
CiscoCSR1000v
Vyatta5600
vnic0
vSwitchvnic1
VyOS
JuniperFirefly
CiscoCSR1000v
Vyatta5600
Our environment for Traffic Test
VyOS
vSwitch
vnic0
vSwitch
Interop
erability
L2S
W
Traffic Test L2S
W
vnic1
vnic1
OpenStack (node2)
VMware Esxi (node1)
VMware Esxi (node3)
OpenStack (node4)
vnic2
vnic3
vnic1
vnic2
vnic3
MGMT-Seg
vnic0
vnic0
vSwitch
JuniperFirefly
CiscoCSR1000v
Vyatta5600
vnic0
vSwitchvnic1
VyOS
JuniperFirefly
CiscoCSR1000v
Vyatta5600
VNFappliancetestenvironment
VNFappliancetestenvironment
Our environment for Traffic Test
VyOS
vSwitch
vnic0
vSwitch
Interop
erability
L2S
W
Traffic Test L2S
W
vnic1
vnic1
OpenStack (node2)
VMware Esxi (node1)
VMware Esxi (node3)
OpenStack (node4)
vnic2
vnic3
vnic1
vnic2
vnic3
MGMT-Seg
vnic0
vnic0
vSwitch
JuniperFirefly
CiscoCSR1000v
Vyatta5600
vnic0
vSwitchvnic1
VyOS
JuniperFirefly
CiscoCSR1000v
Vyatta5600
TrafficGenerator
Our environment for Traffic Test
VyOS
vSwitch
vnic0
vSwitch
Interop
erability
L2S
W
Traffic Test L2S
W
vnic1
vnic1
OpenStack (node2)
VMware Esxi (node1)
VMware Esxi (node3)
OpenStack (node4)
vnic2
vnic3
vnic1
vnic2
vnic3
MGMT-Seg
vnic0
vnic0
vSwitch
JuniperFirefly
CiscoCSR1000v
Vyatta5600
vnic0
vSwitchvnic1
VyOS
JuniperFirefly
CiscoCSR1000v
Vyatta5600
Result of test automation
• Enables multiple patterns of test runs which are difficult by manual
Automatethecombinationtestof2736patterns by40hours
【CSR1000v,Firefly,Vyatta,VyOS】VNFappliance
【VMwareESXi,OpenStack】VIM
【CPU,Mem,HDD,VNIC】
【CSR1000v,Firefly,Vyatta,VyOS】Interoperabilitywithreferencerouter
【VMwareESXi,OpenStack】VIMofreferencerouter
【BGP:5 Pattern,OSPF:7 Pattern】
【TestPattern:3Pattern,FrameRate:6Pattern】
FlavorofVNFAppliance
Routingprotocol
Traffictestpattern
Current Activities
• We have just joined OPNFV, and deploy Community Lab.
• Refactoring to align NFV Architecture of ETSI• Using TOSCA based orchestrator (we are considering Tacker)• Collaborate with Functest