56
WIND Hellas Benefits and Lessons Learned from MYCOM OSI PrOptima™ V4.2 Kernel upgrade and IP 3.0 solution 20-21 April 2016 Spyros Kokosis Foteini Erga Wind Hellas
WIND Hellas
Benefits and Lessons Learned from
MYCOM OSI PrOptima™ V4.2
Kernel upgrade and IP 3.0 solution
20-21 April 2016
Spyros Kokosis
Foteini Erga
Wind Hellas
• Wind Hellas Company Profile and Network
• V4.2 kernel upgrade project
• Drivers
• Project Phases and Risks
• Post-upgrade issues
• Benefits
• Lessons Learned
• New Feature Requests
• Q & A Session
Agenda
COMPANY PROFILE
Network Coverage
% GSM Population Coverage 99%
% UMTS Population Coverage 72%
% Fixed Line Coverage 70%
% LTE Population Coverage 60%
Strategic partnership with Vodafone Hellas for
partial sharing of the 2G / 3G mobile network
since 2014
~ 800 2G and 3G shared sites
6
Radio Transmission IPCircuit Switch
Packet Core Fixed Access Fixed Core VAS
Vendors per Main Technology Area
NOKIA
NOKIA
NOKIA
WH Network
integration in
PrOptima™
CS Domain
PS DomainSGSN
GGSN
MGW
MSS
HLR
RNC
BSCBTS
NodeB
IP/MPLS
OSS
Circuit and Packet Core
VAS
eNodeB
BTS
CS Domain
PS DomainSGSN
GGSN
MGW
MSS
HLR
RNC
BSC
NodeB
IP/MPLS
OSS
Circuit and Packet Core vs. PrOptima™
VAS
eNodeB
LITESPAN
IP
DSLAM
WH Fixed
Transport
IP/MPL
S
INTERNET
BRAS
NGN
IP
LEX
OTE, PLMNAND OTHER OPERATORS
End User
premises
Splitter
H.248
MSAN
Fixed Network
LITESPAN
IP
DSLAM
WH Fixed
Transport
IP/MPL
S
INTERNET
BRAS
NGN
IP
LEX
OTE, PLMNAND OTHER OPERATORS
End User
premises
Splitter
H.248
MSAN
Fixed Network vs. PrOptima™
V4.2 Kernel upgrade
1. Drivers
The main reason for the V4.2 kernel upgrade
Users View Definitions
• Separate the views of users working in Radio Network sharing project (Huawei, Victus engineers)
• Partial data sharing - Confidentiality issues
• Build shared reports between Wind - Victus
The main reason for the IP 3.0 Solution
IP SLA Monitoring
• Verify network SLAs with our customers
• Offer reliable services with the requested capacities
• Metrics to measure:
– Port Speed
– Service Availability
– Port Utilization
– RTT/ Packet Loss/ Jitter
– Delay (round-trip, one-way)
– Packet sequencing
2. Project Phases
“It always takes longer than you expect,
even when you take into account Hofstadter's Law”
Douglas Hofstadter
Project Duration: 8 months
• Preliminary Phase: 2 months
– Replace old HW proptipoller servers (Month1)
– Solaris 11 upgrade on pm servers (Month2)
• Main Phase: 6 months
– Test server V4.2 & IP3.0 upgrade (Month3)
– SIT and UAT (Month4)
– Live server V4.2 Kernel upgrade (Month5)
– IP 3.0 Solution upgrade (Month6)
– Final Acceptance (Month8)
3. Risks
Project Risks
• OS Solaris 11 – prerequisite for V4.2 release - not supported by existing proptipollers HW
• replace HW
• migrate ~ 700 IP elements
• IP release currently installed was too old
• Wind Hellas was the first customer who migrated from IP1.0 to IP3.0
• The 2 step migration procedure never tested on production environment
Project Risks
• Differences between live and test environment regarding the IP network
• 700 IP NEs integrated in the live server
• only 2 test IP devices in staging server
• IP3.0 Solution would be tested on a couple of test IP devices only
Project Risks
Project Risks
• NO roll back procedure neither for V4.2 nor the IP3.0 upgrade
Project Risks
• More than 700 IP nodes should be migrated in the new device platforms concept
4. Post-Upgrade
Issues
“Anything that can go wrong,
will go wrong”Murphy's law
Post Upgrade Issues from the V4.2 upgrade
Upgrade day
• During live server V4.2 kernel upgrade the PrOptipollersstopped polling the IP devices for 6 hours.
Upgrade day
• IP Throughput counters converted from bits per sec to bytes per sec– decreased Throughput values in all IP interfaces
– generation of false PM alarms
– IP3.0 upgrade postponed until issue was resolved
Proved to be an incompatibility issue between V4.2 and IP1.0
Post Upgrade Issues from the V4.2 upgrade
The day after
• Performance degradation
• Delays in reports computation and data importation
Misconfigured parameters
Post Upgrade Issues from the V4.2 upgrade
The day after
• Impact on Pre-aggregated KPIs and Monthly aggregation.
Monthly aggregation script was accidentally left outside of the migration procedure
Misconfigured parameters led to memory allocation issue
Post Upgrade Issues from the V4.2 upgrade
Upgrade day
• The upgrade process took 8 times longer than initially estimated
Client unavailable for 1 business day
# of the IP elements (700) wasn’t taken into account when estimating the migration procedure
Post Upgrade Issues from the IP3.0 upgrade
The day after
• Memory consumption due to a wrong mapping between IP1.0 and IP3.0
Limitation on the daily concurrent users from 30 to 10 users
Post Upgrade Issues from the IP3.0 upgrade
The day after
• Data Discontinuity due to duplicated IP NEs on the NE Palette
• Tilde (~) symbol appeared at the end of the new NE names in the NE Palette
Post Upgrade Issues from the IP3.0 upgrade
5. Benefits
User specific benefits from the IP3.0 release
• Computation Time
Improved user perception especially in IP reports
less congestion during busy hours (morning reports)
User specific benefits from the V4.2 release
• Monthly Aggregation
Flat files for monthly aggregated data
Better computation time for monthly reports vs. the on the fly computation
Increased retention period for monthly data vs. the daily data retention
User specific benefits from the V4.2 release
• Multi Dimensional Analysis
Mixed network areas (2G, 3G and 4G) using the Site and/or Cell Sector new object types
Multi RAT Performance Management
User specific benefits from the V4.2 release
• Ability to import static Network Areas from a CSV file stored on a local PC
Efficiency compared to the manual procedure through the client
No need to contact the administrator
User specific benefits from the V4.2 release
• KPI cut and paste from the client
Easy re-organization of the KPI library through the client
No need to contact the administrator
• Visualization of the raised alarms in a table report
Avg duration per alarm type and Network Object type
# of acknowledged per alarm type and Network Object type
# of cleared per alarm type and Network Object type
# of raised alarm per type and Network Object type
User specific benefits from the V4.2 release
• Automatic switch between pre-agg and on the fly formula
Users don’t have to create different reports for historical reporting
Better computation time
(-) In the reports where Network Areas are used, the system switches to
the pre-agg formula and returns NaN because the Pre-agg KPIs don’t
apply to Nas
(-) Users have to manually disable this feature
User specific benefits from the V4.2 release
• User Views Filtering Victus and Huawei users can view only Radio
related NEs and statistics
User views separation to NOC personnel for the alarms monitoring per Network domain
Enhancements:
• In Alarm monitoring window, the alarms created on Network Areas to be filtered based on the user view definitions.
• When applying a user view definition per MOC, the parent counters should not be visible
• Decrease the log in time required for a user with restricted views
Admin specific benefits from the V4.2 release
BEFORE
AFTER
• Configuration importation per VTI
Autonomous configuration process
Time efficiency
Admin specific benefits from the V4.2 release
• Pre-Aggregated formulas importation from a CSV file
Straightforward and efficient way
Save time especially for bulk requests
Admin specific benefits from the V4.2 release
• Off the shelf support for vendors, models through the device platforms
• Easiest and more accurate way of integrating IP elements
• On demand Device Platform creation
Admin specific benefits from the IP3.0 release
6. Lessons Learned
Lessons learned
Things will always take longer than you expect
When in doubt, choose then validate and if needed iterate
Lessons learned
A HW and SW audit should precede an upgrade
Mirror your live and test environment as much as you can
Lessons learned
Double-check all the configuration settings on the servers
Thoroughly review the migration procedure
Lessons learned
Involve as many users as you can in the UAT / Regression tests
Document all the changes from one release to another
Lessons learned
Mitigate the risk of data loss as much as you can
Remember the lessons of the past
6. Feature Requests
Request for Features
Requests for new Features in the following release:
• Apply Pre-agg KPIs to Network Area dimension
• Enhanced Search & find function in Report explorer
• Improvements on sharing and renaming a Static Network Area
• Cell Sector and Site enhancements
Thank you
