Nokia Blueprint E2E Testing in Real Cloud Customer Environment Conditions
Nikos Vrakas
Test Lead, Nokia Core Engineering Solution Testing
1. The Solution2. Testing
Areas
3. Automation
The Telco Could: Network Function Virtualization
• Dedicated HW replaced by VMs
• Automated deployments and LCM
• Virtualized Network Functions (VNFs)
• Flexibility and better resource utilization
• Dynamic Scaling
• Faster time to market
The chassis-based implementation has been virtualized by turning each line card in the
system into a virtual machine
Simplified ETSI NFV MANO reference architecture
ComputingHardware
StorageHardware
NetworkHardware
Hardware resources
Virtualization LayerVirtualized
InfrastructureManager(s)
VNFManager(s)
NFV Orchestrator
NFVI
EMS 1
Service, VNF and Infrastructure Description
Virtual Computing
Virtual Storage Virtual Network
VNF 1
EMS 2
VNF 2
EMS 3
VNF 3
E2E vs E2E
RealElement
RealElement
RealElement
RealElement
RealElement
RealElement
SUT: Gateway
Phone Phone
E2E vs E2EHTTPS/JSON
A-BGWATGW
E-CSCFEATF
I-/S-CSCFBGCF
TASSCC-AS
SM-IP-GWMFRC
LIMS
Charging system
AAA
PCRF
WCDMA
GSM
LTE
DirectWAP
BSC
RNC
SMSC
S2b (GTP)
S6a
SCPSWx
P-CSCFATCF
MME
MSS
ePDG
SGW
PGW
UntrustedWAP
HSS-FE
HLR-FE
SDM
MRFP
MRF
DNSENUM
IM-MGW
MGCF
PSTN
PGW
S1-MME
S1-U
Gm
S11
S5
SWm s6a
Cx/Dx
Sh
MAPMAP
Hd
BSF/NAF
Ut/XCAP
Ut/XCAP
Sv
Rx
Gx
Mw
ISC, Ma
Iq
Mr’
MpMb
SGi
Mb
Mb
Nb
Nc
Swu
AuC
Mw
MNP DB
MAP
Mg
IxIMS ntw
I-BCF
TrGW
NetAct/COM/SAM
Traffica
Op
era
bility
iOS ES
Mx
Registers
Voice Core
EPC
SPR
Sp
I2
Mm
RCS, UMC
MSAN
AGCF
AGW
POTS/ISDN
IP PBXFixed Access
E2E vs E2E
The Cloud Core Solution
Scalability
SBC
1. The solution2. Testing
Areas
3. Automation
E2E - Voice over X Testing
VNF Functional validation
• Calls CS-VoLTE-VoWiFi
• Emergency calls
• Lawful Interception
• Handovers
Cloud Operability / Life Cycle Management
• VNFM based LCM (20+ E2E core VNFs)
• Scaling, Evacuation/Migration
4G/5G
2G/3GWi-Fi
Test LabE2E - Voice over X Testing
3G
4G
3G
4G
Performance Testing
StabilityStudy Solution’s stability with at least such traffic intensity that it is normally designed to be handled and that all the VNFs are stable when loaded for defined duration (24h/48h).
OverloadMass traffic is applied outside the required capacity until an overload situation is achieved.
ProvocativeThe solution is tested under special conditions, performing provocative actions with mass traffic ongoing.
Stability ProvocativeOverload
Tools: Simulators/Call GeneratorsPerformance Testing
500k concurrent calls
CloudCore Solution
48h stability
1M mobile phones!
Tools: Simulators/Call GeneratorsPerformance Testing
CallSimulator
CallSimulator
CallSimulator
CallSimulator
CallSimulator
500k concurrent calls
CloudCore Solution
48h stability
Simulate: UEs, Radio, WiFi Access Points
Security/Robustness Testing
Tools: DDoS OrchestratorSecurity Testing
• Internally designed and developed
• SDN datacenters can handle enormous amount of traffic
• Initiating attacks from simple VMs cannot have impact on such deployments
• Orchestrates many servers for various DDoS attacks
• Supports SDN by using the Nuage VRS-B plug-in
ReflectionSIP
TCP SYN
Flooding
Controller
Engine
DD
oS
Orc
he
stra
tor
Clo
ud
Co
re
Tools: Simulating Extreme ConditionsRobustness Testing
Deviations
CloudCore Solution
CallSimulator
80 64 e1 40
00 40 06 44X
X
X X
1
overwrite
bursts
GBps
Sec
fragmentation
Line breaks
X
1. The Solution2. Testing
Areas
3. Automation
Robot Framework
• Developed at Nokia Networks, now sponsored by Robot Framework Foundation
• Keyword driven, keywords created by python experts
• No programming skills are needed from testers
• Testers can create new higher-level keywords by composing old ones
• Clear and detailed reports in html/xml format
Test StructureRobot Framework
• Setup, Test Steps, Teardown
• Restores system configuration
• Supports tags
• Easy, clear and human readable test steps
Suite Setup
Test Setup
Test Steps
Test Teardown
Test Setup
Test Steps
Test Teardown
Suite Teardown
Test 02
Test 01
Robot Framework
‘Start ipsec tunnels’ :
• Connect to a certain network element
• Execute cli command for tunnel bring-up
• Check cli response
In case of error:
• Execution breaks
• Teardown runs for recovery
Security_IPSec_Tunnel_3DES
[Documentation]
... Establishes IPsec tunnel - 3DES-CBC - MD5
[Setup] Configure_IPsec_Tunnels_3DES
[Tags] security ipsec regression
start ipsec tunnels
show ipsec ikev1 tunnels
start traffic on simulator simulator_A5 basic_ip_trafic.conf
robot wait for seconds 10
stop traffic on simulator simulator_A5
[Teardown] Delete_Configure_IPsec_Tunnels_3DES Collect_Logs
Test Automation Framework
• For Business and stakeholders
• Pass/Fail indication
• Measurements categorized by Test Case and Feature Name
• Compares historical data
• Difficult to read by Business and Product owners
• Beautiful and very detailed graphs
• Helpful for extensive analysis of KPIs and measurement results
robot
• python 2.7• robot
framework Call KPIs & resource Graphs
Generic PipelineTest Automation Framework
CallSim.
The WAF (Wireless Automation Framework)E2E - Voice over X Testing Automation
• It is a Python Library by Apple
• Used to control R&D software equipped iPhone mobiles from a Mac
• Control functions are triggered from Robot Framework
Radio
WAF setupE2E - Voice over X Testing Automation
Attenuators
RF Boxes
WAF Macs
EthernetUSB/lightning
Coaxial
WAF PipelineE2E - Voice over X Testing Automation
Robot Framework:
Robot Framework is a generic test automation framework for acceptance testing. It is a keyword-driven testing framework that uses tabular test data syntax. All our E2E Regression DevOps tests are coded in Robot.
Jenkins:
Jenkins is a server-based system that helps to automate the non-human part of software development process, with continuous integration and facilitating technical aspects of continuous delivery.
WAF Tool:
WAF tool is developed by Apple, and it can be used only with Apple devices which run RnD software.
It is a Python library that runs only on Mac after installing from a simple installer..
WAF tool can be used to control R&D SW equipped iPhone mobiles, so all iPhones should be connected to a Mac and an RPC service should be installed and started on it to call these function from Robot Framework.
E2E Solution Environment:
An overview of the environment we use in E2E Solution Testing.
Jenkins connects to GITLAB server, where all cases are stored. They are written in Robot.
python 2.7
DDoS PipelineSecurity Testing Automation
Security Testers
commit
pullExecute Robot
Robot Dev
commit
Run Tools
CallSim.
DDoSOrch.
PublishKPI Measurements
Analyze the impact
“Run, then automate” vs “Automate before even run”
Thank you!