Date post: | 29-Dec-2015 |
Category: |
Documents |
Upload: | lechihuong |
View: | 52 times |
Download: | 1 times |
Network Quality & Performance Division
“Monitoring the End User Quality of
Experience using the derivative Mean
Opinion Score (MOS) KPI”
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic
Paper Authors: Konstantinos Vlahodimitropoulos PhD, Evangelos Katsaros
Presentation: Filippos Kyriazidis
Network Quality & Performance Division
Objectives:
� Indicate the importance of MOS to the end user perceived speech quality.
� Introduce the MOS derivative Key Performance Indicator (KPI) which comes from Drive Tests Data and Network Statistics.
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic
Statistics.
� Implement a mathematical formalism in order systemize the MOS calculation procedure.
� Compare different speech codecs implementations in BSC areas using MOS distribution patterns.
Network Quality & Performance Division
Contents:
•• MOS concept & dependencies.MOS concept & dependencies.
• Average MOS calculation in Cell & BSC level combining drive tests data & OSS measurements.
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic
• Cell MOS distribution patterns for different speech codecs. EFR vs AMR codec comparison.
• Effect of MS technology on different speech codecs share.
• Evaluating speech codec effectiveness using quality and speech codec distribution patterns
Network Quality & Performance Division
Mean Opinion Score (MOS) concept description:• A common benchmark used to determine sound quality. • In voice telephony, represents a measure of human speech quality, at the
destination end of the circuit. • It is an important factor in determining the QoS.
MOS Measurement: 1. A pool of listeners rate a series of audio files using a five-grade scale ranging
from 1 to 5.2. The average or Mean Opinion Score (MOS) for each audio file is calculated.
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic
2. The average or Mean Opinion Score (MOS) for each audio file is calculated.
Conditions:• 16 or more listeners (adequate sample).• Quiet environment.
�To reduce variability, tests commonly include reference files that have "industry accepted" MOS scores.
Network Quality & Performance Division
MOS dependencies in a mobile network:
�Speech Codec �Radio Interface Conditions
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic
MOS classification according to ASCOM scale
53.5
ExcellentGood
2.72.1
FairPoor
0
Bad
1.5
Full Rate (FR)Half Rate (HR)
Enhanced Full Rate (EFR)Adaptive Multi Rate (AMR)
Signal Qualityi.e.: BER, FER
Network Quality & Performance Division
•• MOS concept & dependencies.MOS concept & dependencies.
•• Average MOS calculation in Cell & BSC level Average MOS calculation in Cell & BSC level combining drive tests data & OSS measurements.combining drive tests data & OSS measurements.
• Cell MOS distribution patterns for different speech codecs.
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic
• Cell MOS distribution patterns for different speech codecs. EFR vs AMR codec comparison.
• Effect of MS technology on different speech codecs share.
• Evaluating speech codec effectiveness using quality and speech codec distribution patterns
Network Quality & Performance Division
• Basic principle of extracting a MOS derivative KPI:
Perform Drive Tests
Process results & create MOS Quality &
Query OSS
Export Speech Codec usage Matrix
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic
create MOS Quality & Speech Codec Matrix
Matrix Export Quality class statistics
Matrix
Perform Matrix Operations to calculate
Cell/BSC MOS value
Data Merge
Network Quality & Performance Division
Drive Tests Reference Table:
Speech Codec degradation direction
(5 speech codecs)
Quality degradation direction (8 quality classes)
Speech Codec / DL Quality class 0 1 2 3 4 5 6 7AMR FR MOS 3,71 3,56 3,31 3,05 2,72 2,49 2,12 1,70AMR HR MOS 3,36 3,24 3,04 2,88 2,71 2,47 2,11 1,40
EFR MOS 3,76 3,68 3,43 3,08 2,72 2,34 2,01 1,44FR MOS 3,19 3,14 3,03 2,77 2,58 2,22 1,82 1,45HR MOS 2,92 2,83 2,68 2,59 2,43 2,13 1,66 1,41
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic
Speech Share of BSC Table
AMR FR AMR HR EFR FR HR33.95% 13.96% 39.46% 0.39% 12.23%
Speech Codec Share
Quality Class Share of BSC
DL Quality Samples ShareQUAL_0 80.17%QUAL_1 3.73%QUAL_2 3.68%QUAL_3 3.67%QUAL_4 3.22%QUAL_5 2.72%QUAL_6 1.89%QUAL_7 0.91%
OSS Derived Tables
Network Quality & Performance Division
BSC MOS calculation example:
DL Quality Samples ShareQUAL_0 80.17%QUAL_1 3.73%QUAL_2 3.68%QUAL_3 3.67%QUAL_4 3.22%QUAL_5 2.72%QUAL_6 1.89%QUAL_7 0.91%
Spech Codec / DL Quality class 0 1 2 3 4 5 6 7AMR FR MOS 3.71 3.56 3.31 3.05 2.72 2.49 2.12 1.70AMR HR MOS 3.36 3.24 3.04 2.88 2.71 2.47 2.11 1.40
EFR MOS 3.76 3.68 3.43 3.08 2.72 2.34 2.01 1.44FR MOS 3.19 3.14 3.03 2.77 2.58 2.22 1.82 1.45HR MOS 2.92 2.83 2.68 2.59 2.43 2.13 1.66 1.41
X =
Dimension: 5X8
Dimension: 8X1
Speech Codec Weighted MOS ValueAMR FR MOS 3.55AMR HR MOS 3.24
EFR MOS 3.59FR MOS 3.08HR MOS 2.82
Dimension: 5X1
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic
Speech Codec Weighted MOS ValueAMR FR MOS 3.55AMR HR MOS 3.24
EFR MOS 3.59FR MOS 3.08HR MOS 2.82
BSC MOS = X
Dimension: 5X1
Dimension: 1X5
= 3.43AMR FR AMR HR EFR FR HR33.95% 13.96% 39.46% 0.39% 12.23%
Speech Codec Share
Network Quality & Performance Division
Generalized calculation procedure:The (m x n) (m codecs, n quality classes) matrix is multiplied with the (n x1)
quality classes matrix as in (1):
(1)
...
...
...
...
......
............
............
......1
0
1
0
000
=
×
n SC
SC
Q
Q
MM
MM
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic
............
0
mm
mnm SCQMM
The product of (1), (m x 1) is further multiplied with the (1 x m) speech codecs matrix to obtain a single average MOS value, as in (2):
[ ] (2) _.........
...
...
...
0
0
MOSAverageCC
SC
SC
m
m
=×
Network Quality & Performance Division
Key issues:• Process can be easily implemented in a performance
monitoring system.• Can be applied in Cell, BSC or any other network
structure. • Cells with Poor or Bad MOS can be isolated and be further
investigated.
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic
investigated.• A single average MOS value of a cell group (i.e. BSC
area) may not be essential. The comparison of different network areas provides QoS indicative results.
Network Quality & Performance Division
• MOS concept & dependencies.
• Average MOS calculation in Cell & BSC level combining drive tests data & OSS measurements.
•• Cell MOS distribution patterns for different speech Cell MOS distribution patterns for different speech
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic
•• Cell MOS distribution patterns for different speech Cell MOS distribution patterns for different speech codecs. EFR vs AMR codec comparison.codecs. EFR vs AMR codec comparison.
• Effect of MS technology on different speech codecs share.
• Evaluating speech codec effectiveness using quality and speech codec distribution patterns
Network Quality & Performance Division
MOS distribution:•Derives from cell level MOS calculation of a group of cells
•Gives a brief description of the end user perceived speech quality in a specific network area.
60
7053.5
ExcellentGood
2.72.1
FairPoor
0
Bad
1.5
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic
0
10
20
30
40
50
2,5
2,58
2,66
2,74
2,82 2,
92,
983,
063,
143,
22 3,3
3,38
3,46
3,54
3,62 3,
73,
783,
863,
94
MOS classification
Nu
mb
er o
f ce
lls
Network Quality & Performance Division
MOS improvement pattern:•The distance between distributions shows the MOS improvement
between two different states of a group of cells.
50
60
70FR/HR codec
AMR codec (together with EFR &FR/HR)
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic
0
10
20
30
40
50
2,5
2,58
2,66
2,74
2,82 2,
92,
983,
063,
143,
22 3,3
3,38
3,46
3,54
3,62 3,
73,
783,
863,
94 >4
MOS classification
Nu
mb
er o
f ce
lls
53.5
ExcellentGood
2.72.1
FairPoor
0
Bad
1.5
Network Quality & Performance Division
EFR versus AMR:•EFR achieves highest but also lowest MOS values in comparison to AMR.
•EFR distribution has a larger standard deviation than AMR.
•AMR is subsequently, a preferable speech codec.
120
140
160
EFR codec (together with FR/HR)
AMR codec (together with EFR/FR/HR)
53.52.72.10 1.5
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic
0
20
40
60
80
100
120
2,5
2,58
2,66
2,74
2,82 2,
92,
983,
063,
143,
22 3,3
3,38
3,46
3,54
3,62 3,
73,
783,
863,
94 >4
MOS classification
Nu
mb
er o
f ce
lls
53.5
ExcellentGood
2.72.1
FairPoor
0
Bad
1.5
Network Quality & Performance Division
• MOS concept & dependencies.
• Average MOS calculation in Cell & BSC level combining drive tests data & OSS measurements.
• Cell MOS distribution patterns for different speech codecs.
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic
• Cell MOS distribution patterns for different speech codecs. EFR vs AMR codec comparison.
•• Effect of MS technology on different speech codecs Effect of MS technology on different speech codecs share.share.
• Evaluating speech codec effectiveness using quality and speech codec distribution patterns
Network Quality & Performance Division
The role of Mobile Station technology
� The following speech codecs share matrix is involved in all MOS calculations:
]..........[0 m
CCSC =
According to the previous example:
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic
� The existence of old-technology mobiles in a network area, which do not support ‘modern’ speech codecs, will affect codecs’ shares against EFR & AMR percentages. This will degrade average MOS of BTSs.
]%23.12%39.0%46.39%96.13%95.33[=SC
� Promotion of latest technology MSs to existing subscribers could be a key issue of marketing strategy in order to optimize speech quality.
AMRFR AMRHR EFR FR HR
Network Quality & Performance Division
• MOS concept & dependencies.
• Average MOS calculation in Cell & BSC level combining drive tests data & OSS measurements.
• Cell MOS distribution patterns for different speech codecs.
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic
• Cell MOS distribution patterns for different speech codecs. EFR vs AMR codec comparison.
• Effect of MS technology on different speech codecs share.
•• Evaluating speech codec effectiveness using quality Evaluating speech codec effectiveness using quality and speech codec distribution patternsand speech codec distribution patterns
Network Quality & Performance Division
Quality & Speech codec distribution patterns:
� An equivalent way of deciding speech codecs’ effectiveness is to calculate the percentage of samples in each MOS class, i.e. create a graph with the following information:
35%
40%
45%
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic
To achieve this aim drive tests data should be post-processed differently with regard to the original data.
0%
5%
10%
15%
20%
25%
30%
Excellent Good Fair Poor Bad
Sam
ple
s sh
are
Network Quality & Performance Division
Reference Table Quality & Speech codec distribution patterns:
SC MOS Class 0 1 2 3 4 5 6 7AMR FR E 87.79% 75.78% 55.38% 35.41% 15.51% 7.77% 0.00% 0.00%AMR FR G 7.33% 13.66% 22.58% 31.91% 33.16% 30.10% 19.28% 0.00%AMR FR F 4.31% 7.45% 16.94% 21.40% 32.09% 33.01% 30.12% 22.86%AMR FR P 0.57% 3.11% 4.30% 9.73% 17.11% 24.27% 31.33% 34.29%AMR FR B 0.00% 0.00% 0.81% 1.56% 2.14% 4.85% 19.28% 42.86%AMR HR E 38.71% 20.81% 15.63% 10.59% 12.50% 10.81% 0.00% 0.00%
Post-Processed Drive Tests Data
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic
AMR HR E 38.71% 20.81% 15.63% 10.59% 12.50% 10.81% 0.00% 0.00%AMR HR G 53.12% 68.79% 58.75% 54.12% 35.00% 18.92% 12.50% 0.00%AMR HR F 7.10% 7.72% 18.13% 23.53% 35.00% 29.73% 31.25% 11.11%AMR HR P 1.08% 2.68% 6.88% 11.76% 12.50% 37.84% 43.75% 11.11%AMR HR B 0.00% 0.00% 0.63% 0.00% 5.00% 2.70% 12.50% 77.78%
EFR E 91.48% 87.06% 68.26% 41.13% 15.69% 6.15% 0.00% 0.00%EFR G 7.05% 9.90% 19.90% 30.65% 39.87% 18.46% 16.67% 0.00%EFR F 1.25% 2.08% 8.82% 21.77% 29.41% 43.08% 21.43% 15.00%EFR P 0.00% 0.80% 2.77% 5.24% 7.84% 23.85% 38.10% 15.00%EFR B 0.23% 0.16% 0.25% 1.21% 7.19% 8.46% 23.81% 70.00%FR E 0.00% 0.21% 0.00% 0.00% 0.67% 0.00% 0.00% 0.00%FR G 96.14% 92.31% 84.43% 63.30% 43.33% 18.92% 4.00% 0.00%FR F 3.34% 6.44% 12.11% 25.00% 38.00% 36.94% 24.00% 10.00%FR P 0.39% 1.04% 3.11% 9.57% 16.67% 34.23% 45.33% 25.00%FR B 0.13% 0.00% 0.35% 2.13% 1.33% 9.91% 26.67% 65.00%HR E 0.16% 0.42% 1.31% 1.72% 1.32% 0.00% 0.00% 0.00%HR G 92.54% 81.89% 60.93% 49.68% 32.34% 9.83% 1.82% 0.00%HR F 5.40% 13.63% 26.24% 32.47% 39.27% 43.35% 13.64% 8.51%HR P 1.90% 3.95% 11.08% 14.41% 24.09% 37.57% 44.55% 19.15%HR B 0.00% 0.10% 0.44% 1.72% 2.97% 9.25% 40.00% 72.34%
Network Quality & Performance Division
Calculation example in a BSC area with AMR /EFR/FR/HR enabled:
QUAL_0 QUAL_1 QUAL_2 QUAL_3 QUAL_4 QUAL_5 QUAL_6 QUAL_7
� Calculation Procedure: � Drive Tests data are weighted according to quality class
share, that is the following table:
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic
QUAL_0 QUAL_1 QUAL_2 QUAL_3 QUAL_4 QUAL_5 QUAL_6 QUAL_783.58% 3.12% 2.97% 2.93% 2.52% 2.23% 1.71% 0.94%
� The resulting table is further transformed according to speech encoding share:
AMRFR 37.43%AMRHR 8.73%EFR 46.12%FR 0.44%HR 7.28%
Network Quality & Performance Division
Calculation example in a BSC area with AMR /EFR/FR/HR enabled:
� After all transformations (multiplications) the Drive tests table becomes:
SC MOS Class 0 1 2 3 4 5 6 7 TotalAMR FR E 27.213% 0.897% 0.663% 0.425% 0.160% 0.068% 0.000% 0.000% 29.427%AMR FR G 2.292% 0.159% 0.251% 0.350% 0.312% 0.251% 0.123% 0.000% 3.740%AMR FR F 1.348% 0.087% 0.188% 0.235% 0.302% 0.275% 0.193% 0.081% 2.710%AMR FR P 0.180% 0.036% 0.048% 0.107% 0.161% 0.202% 0.201% 0.121% 1.056%AMR FR B 0.000% 0.000% 0.009% 0.017% 0.020% 0.040% 0.123% 0.151% 0.361%AMR HR E 2.825% 0.057% 0.041% 0.027% 0.027% 0.021% 0.000% 0.000% 2.998%
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic
AMR HR E 2.825% 0.057% 0.041% 0.027% 0.027% 0.021% 0.000% 0.000% 2.998%AMR HR G 3.876% 0.187% 0.152% 0.138% 0.077% 0.037% 0.019% 0.000% 4.487%AMR HR F 0.518% 0.021% 0.047% 0.060% 0.077% 0.058% 0.047% 0.009% 0.837%AMR HR P 0.078% 0.007% 0.018% 0.030% 0.027% 0.074% 0.065% 0.009% 0.309%AMR HR B 0.000% 0.000% 0.002% 0.000% 0.011% 0.005% 0.019% 0.064% 0.101%
EFR E 35.261% 1.251% 0.936% 0.555% 0.182% 0.063% 0.000% 0.000% 38.248%EFR G 2.716% 0.142% 0.273% 0.414% 0.463% 0.190% 0.132% 0.000% 4.329%EFR F 0.482% 0.030% 0.121% 0.294% 0.342% 0.443% 0.169% 0.065% 1.945%EFR P 0.000% 0.011% 0.038% 0.071% 0.091% 0.245% 0.301% 0.065% 0.822%EFR B 0.088% 0.002% 0.003% 0.016% 0.083% 0.087% 0.188% 0.304% 0.772%FR E 0.000% 0.000% 0.000% 0.000% 0.000% 0.000% 0.000% 0.000% 0.000%FR G 0.354% 0.013% 0.011% 0.008% 0.005% 0.002% 0.000% 0.000% 0.392%FR F 0.012% 0.001% 0.002% 0.003% 0.004% 0.004% 0.002% 0.000% 0.028%FR P 0.001% 0.000% 0.000% 0.001% 0.002% 0.003% 0.003% 0.001% 0.013%FR B 0.000% 0.000% 0.000% 0.000% 0.000% 0.001% 0.002% 0.003% 0.007%HR E 0.010% 0.001% 0.003% 0.004% 0.002% 0.000% 0.000% 0.000% 0.020%HR G 5.634% 0.186% 0.132% 0.106% 0.059% 0.016% 0.002% 0.000% 6.136%HR F 0.329% 0.031% 0.057% 0.069% 0.072% 0.070% 0.017% 0.006% 0.651%HR P 0.116% 0.009% 0.024% 0.031% 0.044% 0.061% 0.056% 0.013% 0.354%HR B 0.000% 0.000% 0.001% 0.004% 0.005% 0.015% 0.050% 0.050% 0.125%
Network Quality & Performance Division
Calculation example in a BSC area with AMR /EFR/FR/HR enabled:
MOS Class Samples ShareExcellent 70.69%
Good 19.08%Fair 6.17%Poor 2.55%Bad 1.37%
� A grouping by the MOS class yields the MOS class distribution of samples within the BSC area:
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic
Bad 1.37%
0%
10%
20%
30%
40%
50%
60%
70%
80%
Excellent Good Fair Poor Bad
MOS Class
Sam
ple
s p
erce
nta
ge
Samples Share
Network Quality & Performance Division
Comparative evaluation of three different network states:
MOS Class ΑΜR/EFR/FR/HR EFR/FR/HR FR/HRExcellent 70.69% 56.98% 0.11%
Good 19.08% 33.65% 87.36%Fair 6.17% 5.32% 7.23%Poor 2.55% 2.47% 3.61%Bad 1.37% 1.58% 1.59%
100%
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Excellent Good Fair Poor Bad
MOS Class
Sam
ple
s p
erce
nta
ge
ΑΜR/EFR/FR/HR EFR/FR/HR FR/HR