The Transformation of QoS/QoETesting and Benchmarking

Presenter: Eng. Mohamed Hedi Jlassi

Prepared by: Dr. Irina Cotanis

Agenda

2

The transformation……

Network testing evolution with InfovistaTEMS

A view on benchmarking changes

Take away: facts and conclusions;

Agenda

3

The transformation……

Network Testing evolution with InfovistaTEMS

A view on benchmarking changes

Take away

Transformation emerging from the technology shift

Processes Automation & RCA reporting

(Reduce engineers, back office decision making)

Automatic Predictive Analysis & Insightful Recommendations

Decisions and recommendations made based on Multi-Sourced Big Data Analytics and AI, alarms when things go wrong

Support for SON and MDT functionalities (Rel. 16+)

Slice and user centric

AI/ML functionality and evaluation techniques

(AI augmented tools to detect and evaluate network AI actions)

Legacy networks

5G integration

5G slicing / new

verticals support

5G QoS/QoEContext aware

QoE

(AI/ML based per

slice, and per user

type/profile

evaluation and

optimization)

5G planning

5G deployment

New freq and antenna patterns

mmW, mMIMO/3D beamfoming

modeling

New freq, new air interface,

multi-band/mode devices

Support for technology

disruptions (e.g. device and

beam centric network)

Rethink test probes

Rethink test/evaluation procedures

Evolve meaning of

QoS/QoE

Enhance test/evaluation: automate

Big Data Analytics

enabled by data driven networks

5

Transformation emerging from AI/ML

AI Analytics for 5G+

Descript

Diagn

Predict

Prescript

Network&Device

Distributed ML/AI( domain/slice//link optimization)

Centralized ML based

SON

Intelligent Wireless

Communica5tions

AI-enabled

closed loop

network

optimization

5G

5G+

Legacy

(3G/4G/LTE)

Transformation in standards:QoS/QoE measurements, test scenarios, tools

6

Transformation in standards: ITU-T/ETSI trends on AI/ML.

Study Group 12: Performance, QoS, QoEMachine learning based QoE(e.g. P.565)InfoVista VoLTE/Vo5G voice testing sQLEAR

Study Group 12: Performance, QoS, QoEIntelligent network diagnosis(e.g. E.845 )InfoVista Analytics

Study Group 13: Future Networks – IMT 2020Focus Group ML for Future Networks including 5G(ML5G)InfovIstaAugmented Measurements

InfoVista Analytics

Agenda

8

The transformation roots

Network testing evolution with InfovistaTEMS

A view on benchmarking changes

Take away

9

Untangling the complex relationship between jitter, packet

loss and MOS

Jitter (ms) (z axis) during one 5.5s long RTP sequence (x axis)

during a call duration (y-axis)

TEMS network testing solutionsRethinking QoE modeling: machine learning required due to increased complexity of the interdependencies

Use case: sQLEAR Speech Quality machine LEARningConcept, field set-up and performance

10

Reference voice sample

QoE MOS predictor

Network centric, free of device’ impact enabling

cost efficient optimization towards network

issues rather than device’s

Network centric prediction

EVS codec / client info

Temporal distribution / DTX

Jitter, packet loss

RTP/IP packet stream

Bit rate, bandwidth, client behavior / error concealment

POLQA

LEARNING

Reference voice

Sync: Temporal distribution / DTX

P.565 based

Operator validation results

11

Real-time analytics and orchestration

TEMS Investigation TEMS Paragon TEMS Sense

Remotely scripted and controlled data collection

TEMS Pocket

Post-processing and analysis

TEMS network testing solution Automation/real time data feeding

TEMS Director TEMS Discovery

Rules based Root

Cause Analysis

Machine learning based

Predict

Real-time Test

Orchestration

TEMS Director

Edge computing

Real time data streaming

TEMS network testing solutionsRethinking testing: ML based probes and automated RCA towards prediction

1

2

3

4

RF Logs

GPS

External data

sources

(e.g. site info)

TEMS 5G Engineering Field Probes

Unattended probes

Play back

Analytics & RCA

TEMS Discovery

IV-TEMS Edge computing

(real time detection/diagnosis at the UE)

Use case: predictive probes for a Connected Car scenario

Realtime analysis

Classify in which

degree certain

limitations are

affecting the

offered service Machine learning algorithms

Classify/categorize

throughput/latency that adapt to QoE

requirements

Predictive analytics of QoE/QoS performed in the network

Edge, TEMS Measurement Agent inside car module enables car

AI

Edge computing

- Call will most likely be dropped shortly……..

- Likely driving through an area of low radio quality, download of

navigation information for that area recommended…

- Within 3 mile there is a reduced overall QoE that will affect the

autonomous system of the vehicle, video streaming will be stopped

and please pay attention to traffic.

AI

NT field

probes

Analytics

TEMS QoS / QoE testing

and prediction

Bad coverage in 20sBad QoS/QoE

Poor coverage in 30sQoS/QoE impact expected

Coverage Limited QoS/QoE in 10s

Coverage in 30sPoor QoS/QoE

No service in 10s

Found legacy network

Use cases RCA – automated VoLTE root cause determination

14

• VoLTE calls shown on a map

• Without Root Causes there is extensive

manual effort

• Root Causes automatically added for

failed calls

• Root Causes are focussed answers that

help drive recovery actions

• No expert resources needed

Use case RCA – real time events

15

• Data is streamed in real time

• Shown here are service events

• System heartbeat events use exactly the same method

• These can be used for real time

determination of measurement device

location and status

TEMS network testing solutions: Readiness to going challenges

16

New spectrum

(F1/F2 spectrum range)

New technologies

(mMIMO/3D beamforming, mmW, )Co-existence with legacy networks (“hybrid” LTE-NR)

New architecture concepts

(NFV/SDN, slicing)

New design concept

(Beam centric / Human-machine centric)

Technology adaptations

(From TCP to QUIC/UDP for the delivery of new services e.g. VR/AR, Video360)

“Traditional” testing within a transformed network

Use case: LTE-NR co-existence (EN-DC) - NSA

17

LTE + NR

Combined view in TEMS Investigation

Use case: 3D scenario for TRP beam characterization and coverage evaluation

18

UE based coverage parameters (RSRP, RSRQ,

CINR) for reference signals and performance

(throughput) in azimuth and elevation

SSS RP, SSS RQ, CI and Beam Index for strongest scanned CI

Verify overall and cell coverage as well as coverage gaps

Identify strongest beams; beam failures

Agenda

19

The transformation…….

Network testing evolution with InfovistaTEMS

A view on benchmarking changes

Take away

ITU-T/ETSI benchmarking transformation

20

Benchmark ETSI scoring transformation

21

Overall Global Network Performance

Score*

(country level)

Cities

GNPS=

=Sum(Aw*ServicesScore)Roads

Composite (hot spots,

trains)

ServicesScore=

= Sum(KPIScore)

Voice/SMS KPIs

Video streaming KPIs

(YT)

KPI Score =value−Bad limit

Good limit−Bad limit×𝒘𝒆𝒊𝒈𝒉𝒕

Area

Weightings(Aw)

Statistical inference at

country level (bootstrapping technique)

Per

area/topology aggregation

Data KPIs

(web browsing, http, SM)

KPI 2 Score Map

Dynamic adaptive

thresholds per applic/slice

Flexible/scalable

weightings

Transformable QoE

Interactivity QoE

- Real time/latency - Continuity (jitter)- Reliability (packet loss)

Benchmarking QoE/QoS transformation (ETSI)

Voic

e • SetupSuccessRatio

• CallDropRatio

• MOS

• MOS<1.6MOS

• 90th percentile of MOS

• SetUpTime

• SetUpTime>1.5s

• 10th percentile of SetUpTime

Vid

eo s

tream

ing • SuccessRatio

• MOS

• 10th percentile MOS

• Access time

• Access time>10s

Data

• Data

• TransferSuccessRatioDL

• Avg.ThroughputDL

• 10th /90th

percentileThroughputDL

• TransferSucessRatioUL

• Avg.ThroughputUL

• 10th /90th

throughputUL

• Browsing

• SuccessRatio

• Avg.Duration

• ActivityDuration>6s

• SM

• SucessRatio

• Avg.Durtaion

• ActivityDuration>15s

22

AR/Remote surgery

AR/VR/360Video

AR/GamingAR/Automotive

ITU Benchmarking in quest for CrowdSource

• Managed & limited controlled user/device

information (connected mode) geolocated,

passive/active(!)

• Network (RAN, IP) information (device dependent)

• Application information (technology agonistic)

• QoE surveys

Reliability

Score

• User centric traffic/resources for

• Trend detection (against thresholds and/or

historic behavior

• Optimization

• BMing: Geographical/demographic/operators

comparisons

Use Cases/

Applications

Application

layerRAN

layers

User/Device

CS

CS

CS

CS

IP layer

Decoding

IE/KPI/API

• Measure of spatial consistency

• Measure of temporal consistency

• Measure of fluctuations and variability

• Absolute number of measuring values

Agenda

24

The transformation….

Network testing evolution with infovistaTEMS

A view on benchmarking changes

Take away

Take away

• Technology (mmW, beam centric, user centric)

• AI/ML embedded, operational and management

The transformation

Infovista

TEMS network testing evolution

• New services/slices require new QoS/QoE dependencies

• New expected performance require dynamic adaptive quality thresholds and weightings

• New data source such as crowd source require reliability testing

A view on benchmarking

changes

25

ML based QoE / sQLEAR

Automation - RCA VoLTE use case

Predictive probes - connective car scenario

3D scenario

Thank you!www.infovista.com

26