Ericsson Internal | 2016-09-20 | Page 2Complex Networks, Simple Management
Ericsson oss
›Operator Support System
›Real Time
›1 Billion Subscribers
›320 Operators
›Over 1000 Deployments
Ericsson Internal | 2016-09-20 | Page 4
›OSS-RC to ENM
›A New Strategy for a New
Product
Ericsson Athlone
Ericsson Internal | 2016-09-20 | Page 5
ENM Development
5 countries
100+ Teams
~1400 Engineers
1880M2
Data Centre
Ericsson Internal | 2016-09-20 | Page 7
Organisational Challenges
› Strong focus on individual teams
› Parallel development of
framework, platform and SW
› Communication: Vision gets lost
› Main requirement transparency
› User story size and team
capability to deliver
Ericsson Internal | 2016-09-20 | Page 8
ENM Program Structure
Administration
Program Management
Stakeholder Management
Requirement Area 1
Early Phases
Forecast &
Capability
Requirement Area 4 Requirement Area 5
Bug Follow Up
Improvements Program Communications
Operational PO
Flow Vizualisation
& Optimisation
Technical
Coordination
Integration
PlanningRelease
Validation
Maintrack
ManagementProduct
Introduction
Requirement Area 3
Quality
ManagementEngineering
Tools
Requirement Area 6Requirement Area 2
Area POs
Test
Manager
• MT Stand-up
• Outlook
• Bug Akuten
• Program
Steering
Ericsson Internal | 2016-09-20 | Page 9
Ready - Set - Go
Ready (15B)
2015-09-24
Customer Lab
5K cells (1250 Nodes) Support
LTE eRBS (G1)
• FCAPS
• AMOS
Set (16A)
2015-12-15 (PD4)
Live network (LA)
15K cells (3750 Nodes)Support
LTE eRBS (G1)
• FCAPS
• Auto Provisioning
• FMX
Go (16B)
2016-05-06 (PD4)
GA Product
40K cells (10,500 Nodes) Support
LTE eRBS (G1+G2)
• Macro/Micro/RDS/PICO
SGSN-MME/vSGSN-MME
IP Router 6000
MINI-LINK
Ericsson Internal | 2016-09-20 | Page 10
Poster Campaigns
ENM Info Session
TV
communication
Telepresence
Slide subtitle
3 week sprint calendar
Analysis
Deliver to MT
Coding
5 dayavg US
cycle time
Code review
Await code review
Transaction Cost (→ 0)
Wait time (→ 0)
Config TE & run test
5 dayavg US
cycle time
5 dayavg US
cycle time
Fast Commit
Ericsson Internal | 2016-09-20 | Page 13
› Automated test environment
spin up and test suites: From
days to <1 hr
› Technology– Docker
– J2EE UT in secs
– Arquillian
– TAF
› Coaching for 5 day User
Stories
Fast Commit Enablers
Ericsson Internal | 2016-09-20 | Page 16
Agile Quadrants
Q1
Q2 Q3
Q4
Business Facing
Technology Facing
Su
pp
ort
ing th
e T
ea
mC
ritiqu
e th
e P
rod
uct
Static Tests
Unit Tests
Basic Integration Tests
Acceptance Tests
Usability Tests
Exploratory Tests
Security Tests
Performance Tests
Load Tests
ility Tests
Automated
TAF
Automated
Dev Env
Manual and
Automated
Tools
TAF
Ericsson Internal | 2016-09-20 | Page 17
›Whole Team Approach
›Team Maturity
›Test Community
›Support the Teams to
Produce Quality
Test Role
Ericsson Internal | 2016-09-20 | Page 18
ENM Development flow
IDE
IDE
IDE
BIT
Tests
BIT
Tests
BIT
Tests
Maintrack
Entry
RFA
(Long Loop)
Instant <10 mins
Integration
<3 hour
Main Track Entry (MTE)
Tests
<8 hrs
All teams Tests
RFA Level of Test
RVB
72 hrs
RVB Level of Test
Visualisation and Reporting
Unit
Tests
Unit
Tests
Unit
Tests
seconds
Unit
TAF
Test
TAF
Test
TAF
Test
<10 mins
Functional Test
Main trackTEAM Environment
.
.
.
.
.
.
.
.
.
.
.
.
Ericsson Internal | 2016-09-20 | Page 19
Enm Maintrack Loops
Short Loop Long Loop
<3 Hr <8 Hrs
Load Test
Main track LOOPS
Sprintupgrade
Upgrade
Under load
functional
level test
5k cells,
degrade
detection
All teams Test
<24 Hrs
Cloud
Cloud
Deployment
Robustness
Customer
Upgrade
LOOP
<72 Hrs
60k Cells,
KPIS
Availability
BUR
Backup
& Restore
<1 Wk
Sprint
Release
Real NODE
Manual
E2E
Workflow
• Over 1000
Commits/
sprint
• 25 physical
ENM envs
• Daily
standup
Feedback to
Teams
24 / 7
operation
Follow the sun
for blocking
Issues
Continuous
innovation –
Shift Left
Ericsson Internal | 2016-09-20 | Page 20
MT Loop Evolution
Basic
FCAPS Upgrade
480 Node Upgrade
under load
(5K Network)
RFA250
2015 2016
Qu
ality
Bar
Degrade
Detection
Q4 Q1
Upgrade
under load (15K Network)
Q2 Q3
Availability Fast MT
Q4
Ericsson Internal | 2016-09-20 | Page 21
Test as a Service
Node Access
Cloud Based Test Environment
Simulated Network
Test Consultancy
Ericsson Internal | 2016-09-20 | Page 22
Release Validation
automated Load profiles simulate customer conditions. Used in RV but also by the teams
KPI, stability and endurance testing on the latest MT SW. RV Provides a 72 hour feedback to design
Validation of each sprint on a continuous basis using stamping tests and Clean room.
Workflow Test and usability focus
High availability testing and support to ENM teams via automated test Suite
Test Expertise, Supporting the Teams
Ericsson Internal | 2016-09-20 | Page 23
› Scrum
› Continuous Everything
› Quick to Change
› Strong, Competent Teams with Quality Focus
› Sustainable SW Producing Machine
› Working Software
› Quality First
› Deliver Value to the Customer Frequently
Agile Principles Intact!
Ericsson Internal | 2016-09-20 | Page 24
› We have been on an adventurous journey
› Scaling agile means re-thinking how you are organised
› Test in a scaled agile environment mean supporting your
team to produce quality
› We have failed many times, but we have failed fast
› We have made it work
› Deliver Value to the Customer Frequently
Conclusion