© 2019 Nokia1

We create the technology to connect the world

Guido BauerOctober 2019

Dealing with Telecommunication Product Verification Challenges @ Nokia Nuremberg

© 2019 Nokia2

Agenda

• Nokia in General

• Nokia in Germany

• Optics Business Unit

• System Verification Tasks

• System Verification Challenges

• Large Product Family

• Complex Products

• Release Schedule & Content

• SW Load Deliveries

• Management Interfaces

© 2019 Nokia3

Unrivalled track record of innovation

Foundations of …

• The entire electronics industry

• The internet, networking & optics

• Mobile and fixed communications

Nobel Prizes9 Turing Prizes3 Grammys2 Emmys2 Oscar1

Satellite commsTransistors Unix/C/C++Laser/fiber optics

Solar cells Charge-coupled devices

Super-resolution microscopy

Coherent optics

© 2019 Nokia4

Nuremberg Highlights

▪ Center of Excellence for High Speed Optical interface andOptical Cross Connect switching systems development

▪ Home of the PSE-3 (Photonic Service Engine)

▪ Customer Demo Center for Optical Solutions

▪ Development Hub of the Optics BU

▪ Service Integration, Field Force and Procurement on site

Nokia Nuremberg

Employees▪ ~400 employees in Nuremberg▪ Highly skilled, motivated and multicultural▪ Very experienced team with low attrition rate▪ Agile development teams▪ Fully integrated in the international Nokia community

Nokia in Germany

Stuttgart

Nürnberg

Hannover

Hamburg

Berlin

Mannheim

Leipzig

UlmMünchen

Düsseldorf

Bonn

• about 4.000 employees

• 2 large multi-functional sites, 2 specializes development sites,Nokia Bell Labs research

© 2019 Nokia5

Optics Global R&D & Support Presence

OTTAWA

MURRAY HILL

CHENGDU

NA

ASIA

EMEA

VILLARCEAUX (FRANCE)

Nuremberg / Stuttgart (Germany)

VIMERCATE (ITALY)

WESTFORD

BANGALORE

SHANGHAI

R&D Centers

ONGOING GLOBAL INVESTMENTS IN OPTICS TECHNOLOGIES

29.000+Active Patents

2000+R&D Engineers

© 2019 Nokia6

WW Competence Center for Hi-Speed Photonics, OTN & GMPLSOptics in Germany

Stuttgart

Nürnberg

~300 R&D/PLM engineers and scientists

• Product Line Management

• Bell Labs Research

• System Engineering & Architecture

• High Speed Photonics Development

• Hardware & ASIC/FPGA Development

• Software Development

• System Verification

Key competencies

• Optical Switching Systems

• Photonic Technologies (WDM)

• GMPLS Control Plane, MRN and

Transport SDN

© 2019 Nokia7

1830 PSS-24x/12x/8x OTN Switches (Core & Metro)Optical Innovation from Germany

PSE-2 powered

cards

Flexible 100G-400G Super Coherent transport

wavelengths

Optimized for 100G interface density

1830 PSS-12x

1830 PSS-8x

1830 PSS-24x

Scalability

Capacity

4.8T

48T

24T

4T

1.6T

12T

9.6T

Flexibility

Programmable

Interfaces

SPQPSK

BPSK

QPSK

8QAM

16QAM

64QAM

Efficiency

0.8W/GbSpectral Efficiency

32T

21.6T

17.6T

10.8T

© 2019 Nokia8

Ultimate performance from subsea to metro powered by PSE-3Optical Innovation from Germany

500-600GMetro

300G-450GRegional / Long Haul

200G-250G Long Haul / Subsea

© 2019 Nokia9

System VerificationTasks

© 2019 Nokia10

Large Product FamilySystem Verification Challenges

Details Addressed by

Large diverse product family• in network configurations• with network planning & network management

support• requiring interoperability• with stringent load, robustness and stability

demands• Continuously growing

HW and SW asset development and reuse across products for effort minimization, cost control and quality improvement

Be prepared for product transfers within the Optics BU• Global organizations with global discipline leads

ensuring efficient worldwide operation• Global tooling with assigned engineers for

• Ticket system• Document repositories and reviews• Test automation environment

Dedicated System Verification sub-teams for• Core test automation• Interoperability testing• Cross product Load & Robustness

© 2019 Nokia11

Large Product Family (cont’d)System Verification Challenges

Details Addressed by

See previous slide Verification in various configurations• simple functionality in single NE• Network features in multi-NE configs of single

and mixed product types

© 2019 Nokia12

Complex ProductsSystem Verification Challenges

Details Addressed by

Complex products comprising• High end HW with latest CPUs, FPGAs/ASICs

at technology limits and high speed electrical and optical connections

• Distributed, embedded SW with demanding real time requirements

• Highly sophisticated network planning tool and network management system

Invest in testers‘ system white box know-how to• be able to determine and contact correct

responsible developer• provide right log information for fast root

cause analysis in case of product issues

Investment in “faking” functionality to improve product verification coverage

• SW debug and faking (e.g. for alarm creation)

• HW fault faking by using a system boundary scan

• Power cycle equipment

© 2019 Nokia13

Complex Products (cont’d)System Verification Challenges

Details Addressed by

Demanding Lab Environment• Space• Power supply

• AC for measurement equipment and IT HW• 48/60V DC for network elements (NEs)• PSS-24x in 9T6 configuration: at maximum

14.4kW and typical < 8kW• Cooling• Noise

Extensive investment in and usage of VMWare based simulation environment (HIL: HW-in-the-loop) to• Start testing very early before product HW is

available• Improve costs by reducing needed target

configurations• Increase testing speed

© 2019 Nokia14

Release Schedule & ContentSystem Verification Challenges

Details Addressed by

Very demanding release schedules regarding• number of releases per year

• release content size (amount and magnitude of new features)

• release date and quality adherence

Automation is key• Develop and use automation already for initial

new feature testing• For regression testing available latest end of

release• There is no time to add automated

regression functionality at the beginning of the next release

• Optics System Verification core automation team

• System Verification Nuremberg automation core team

• Each tester develops automation for functional area (s)he is owning• No separation between manual and

automated testing resources

Rx.0.0

May year Y

Content + bug fixes

Rx.0.1

Jul year Y

bug fixesonly

Rx.1.0

Nov year Y

Content + bug fixes

Rx.0.1

Jan year Y+1

bug fixesonly

© 2019 Nokia15

Release Schedule & Content (cont’d)System Verification Challenges

Details Addressed by

• rapidly increasing amount of “legacy features”• Testers are more and more busy with

regression test result evaluation

• Started recently investment in AI/ML based automatic test result assessment

© 2019 Nokia16

Release Schedule & Content (cont’d)System Verification Challenges

Details Addressed by

Last major release contained ~18000 test cases• ~4000 new feature test cases• ~14000 regression test cases

Ensure continuous product quality• Day-by-day

• Daily load delivery with quality verification• When approaching the release date (CP)

• Start of so-called bug scrub process 2 months before CP• Load line split (dedicated release load

line)• Control of each fix going to release

load line by System Verification team• Fix pre-test in next release or private

load before merge to release load line• Final Quality checks right before CP

• 1 week soak test• Final regression and upgrade tests

© 2019 Nokia17

SW Load DeliveriesSystem Verification Challenges

Details Addressed by

SW Load Delivery Handling• “Continuous Integration” leading to daily loads

for up to 2 load lines (main and release endgame after bug scrub start)

• Additional private loads on demand for• bug fix retest in bug scrub phase• dedicated problem investigations

• Global load build

Load healthiness assurance by Sanity testing• 2h basic verification that load is not broken• in simulation per functional area and on target

per product

Ensure “legacy” feature quality• Daily (overnight) test in simulation and on

target• Weekend Regression Test on target• Feature coverage

• functionality of previous release• already verified features of current release

• Varying content of target regression runs due to amount of tests

• Unique test case run time tracking• in August ~9000h, i.e. 375 days

© 2019 Nokia18

Management InterfacesSystem Verification Challenges

Details Addressed by

Management interface richness• Customers want to manage their products and

network differently• With Nokia management capabilities• By using own management systems

• Therefore large amount of management interfaces implemented and to be verified• SNMP

• Network Management System• Craft Terminal (WebUI)

• CLI• TL1 (for dedicated NAR customers)• NetConf & gRPC for SDN application

Test environment supporting verification of all interfaces in manual and automated operation

Integration of management interface switching into automation environment• For products owned by Nuremberg currently

CLI, TL1 and WebUI• Automated sanity and regression tests can

switch the interface run by run

© 2019 Nokia19

Thank you!