of 19
7/31/2019 Ad PESQ VerOct2011
1/19
Speech Quality Measurement
In TEMS Investigation
7/31/2019 Ad PESQ VerOct2011
2/19
Speech quality measurement
AQM overview
How AQM works in TI Accuracy of AQM in TI
Typical AQM values in different environments
What if AQM worse than expected
7/31/2019 Ad PESQ VerOct2011
3/19
AQM overview
Estimate the user experienced quality.
ITU recommendation P.862.1 Compares a distorted speech recording to the original.
Reference speech
(original)
Convert into
psychoacoustic domain
Transmitted speech
(distorted)
Convert into
psychoacoustic domain
PESQ valueCognitive
modelling
7/31/2019 Ad PESQ VerOct2011
4/19
MOS Meaning
5 Excellent
4 Good
3 Fair
2 Poor
1 Bad
AQM results: MOS-LQO
MOS Scale results fromsubjective tests (listening tests).
MOS-LQO =Mean Opinion Score
Listening Quality Objective
using AQM 862.1
Scale is: 4.5 (best) to 1.0 (worst).
Best possible score in a live
network depends on the speechcoder used:
- Best EFR: 4.2
VoCoder Reference PESQ[MOS-LQO]
EFR 0 4.22
EFR 1 4.16
EFR 2 4.15
EFR 3 4.13
EVRC 0 3.96
EVRC 1 3.63
EVRC 2 3.82
EVRC 3 3.57
7/31/2019 Ad PESQ VerOct2011
5/19
AQM in TEMS
AQM works by sending speech references and comparing the
recording to the same known reference.
TI uses four speech references ,two female and two males eakers. Each reference contains of two sentences and each
sentence is 5.5 sec long. The references are in accordance with theITU standard P. 830, which stipulates how subjective MOS testsshould be performed.. One AQM score is given for each reference.
Sentence 0 Sentence 1
Reference 0
7/31/2019 Ad PESQ VerOct2011
6/19
Test units
The AQM algorithm isimplemented in the Call
Generator and in the AQMModule of the TEMSInvestigaton.
Call Generators can beconnected to analogue lines or
DL audio
measurement
to ISDN. Call Generators and AQM
module store the recorded audiofor later analysis.
Results from UL and DL are
joined into the database so youcan see them together in a routeor make statistics on both.
Digital CellularNetwork
Public Switched
Telephone Network(PSTN)
Call
Generators
UL audiomeasurement
7/31/2019 Ad PESQ VerOct2011
7/19
Measurement procedure DTMF syncs the call
Half duplex
Start when last. . .
TIRecord 5.5 sec
and
synch to the CG
Play 5.5 sec speech Record 5.5 sec
and
synch to the CG
AQM DL AQM UL AQM DL
Send DTMFto order call.Start afterlast digit.
digit received
FreqAQM FreqAQM FreqAQMFreqAQM
FreqAQM FreqAQM
7/31/2019 Ad PESQ VerOct2011
8/19
Accuracy of AQM in TI
Three steps of verification:
1. ITU standardizationChose the algorithm which compares best to listening
tests.
3) Live testsprove PESQ
in TA system
2) Conformance tests
prove the PESQ algorithms
2. Conformance testsWhen a measurment tool implements the
in TA
1) ITU has held a competition were PESQ was proven to bethe best speech quality algorithm
a gor m mus pass e con ormance
test. The test consists of 1800 speechsentences and only three scores may differ
more than 0.05 MOS from the ITU version.
3. Live testsBoth the audio collection parts of a
measurement tool and the handset candegrade the recorded audio. To verify that thedegradations are insignificant we compare live
tests to simulations and listening tests.
7/31/2019 Ad PESQ VerOct2011
9/19
Live tests
TEMS has a lab network with base station and BSC,
which makes it possible to test with controlled radioenvironment. Results are then compared tosimulations.
AMR
Codec rate
PESQmeasured on
live network on
DL
PESQmeasured on
live network on
UL
PESQmeasured on
simulation
12.2 4.11 4.08 4.17
7.95 3.81 3.84 3.84
5.90 3.50 3.50 3.50
4.75 3.21 3.18 3.26
7/31/2019 Ad PESQ VerOct2011
10/19
Typical AQM values
Environment AQM
Best EFR / AMR FR (12.2) 4.15Best AMR HR (7.95) 3.85
Best CDMA EVRC 3.8
es . = .
Best 3G (AMR 12.2) BLER = 1% UL 3.85
Typical 3G (AMR 12.2) BLER = 1% 3.8
Good 3G/2G >= 3.5
Acceptable >=3.0
7/31/2019 Ad PESQ VerOct2011
11/19
AQM measurements
PESQ algorithm (ITU Recommendation P.862) Estimates the listening speech quality
Speech-quality score based on the comparison betweenthe original speech and the speech as degraded bythe network
AQM in TEMS Investigation p on n nves ga on .
MO PESQ voice calls (command sequence)
UL and DL MOS score
Frequent AQM
Presentation of AQM scores in TEMS Investigation/Data Collection
TEMS Investigation/TEMS Discovery
Supported terminals Sony Ericsson K790i, K790a
Sony Ericsson K800i
LG CU-500
7/31/2019 Ad PESQ VerOct2011
12/19
TEMS Investigation Test TerminalAQM Module
AQM measurements
Transmitted audio
Received degraded audio(from Call Generator)
Reference audio
PESQ DL Score
Command sequence
MOBILE-TO-FIXED AQM/PESQ MEASUREMENTS
TEMS InvestigationRoute Analysis (Converter)
Call Generator(with PESQ module)
Merged log file containing bothPESQ DL and UL scores
Log file containingPESQ DL scores
PESQ UL scores in XML files(copy/download)
Voice call
PESQ UL Score
Reference audio
7/31/2019 Ad PESQ VerOct2011
13/19
AQM/PESQ MEASUREMENTS TEMS Investigation supports AQM/PESQ measurements
- Mobile-to-mobile and mobile-to-fixed
AQM is more than PESQ
TEMS Investigations AQM/PESQ solution
- TEMS Investigation with AQM license option
[ SERVICE QUALITY TESTING ]
- - Sony Ericsson W995, W995a, and W995 EDGE
- Kyocera KX5
- Samsung SPH-M320 and M330
- AQM module including cable kit
- Call Generator and/or Mobile Receiving Unit(MRU)
- Stand-alone or mounted in backpack or
equipment case
Full reference speech quality measurements
7/31/2019 Ad PESQ VerOct2011
14/19
TEMS Investigation Test terminalAQM Module
Command sequence
Initiate AQM call
Received degraded audio(from Mobile Receiver Unit)
PESQ DL Score
MOBILE-TO-MOBILE AQM/PESQ MEASUREMENTS
[ SERVICE QUALITY TESTING ]
Mobile Receiver Unit(MRU)
TEMS InvestigationExport functionality
Logfile containing PESQ DL andpositioned DL Frequent AQM Scores
Logfile containingPESQ DL scores Voice call
Answer callReference audio
TEMS Investigation
7/31/2019 Ad PESQ VerOct2011
15/19
AUDIO QUALITYMEASUREMENTS
Use TEMS Investigation to monitor end-to-end speech quality
Perform AQM/PESQ measurements with a Call Generator, mobile receivingunits, or a mix of both
- Mobile-to-fixed and mobile-to-mobile
Note the difference between the two solutions, in areas like measurements,
, ,
- PESQ UL and DL vs. PESQ DL- One radio link affecting each PESQ score vs. two radio links
affecting each PESQ score
- Impact of PSTN vs. impact of MRU location
- Capacity of Call Generator vs. number of MRUs
- Workflow and control
Use SQI measurements as a complement, when troubleshooting
Create your own solution
7/31/2019 Ad PESQ VerOct2011
16/19
AQM is worse is it bad radio on DL?
1. Check the radio for the DL: GSM: SQI low (< 30)? Analyze further:
Check no of HO;s. Each HO gives an avg drop of~0.8 MOS.
RxQual (5-7) and FER (>0),
3G: SQI low? Analyze further: Avg BLER < BLER target. Check BLER target. A
BLER target of 1% is default and gives AQM =~3.8MOS.
Is it AMR HR? Check no of Inter RAT HO.
7/31/2019 Ad PESQ VerOct2011
17/19
AQM is worse is it bad radio on UL?
2. Check the radio for the UL:1. All: Use UE traffic recording.
2. All:Look at Frequent AQM does it ever reach maximum? Check codec used: Is it AMR HR, FR or HR?
3. GSM: Look at Frequent AQM does it vary a lot? Analyzeur er:
Check DL radio. Bad DL is clue to bad UL. HO directly affect UL.
4. 3G: Remember for 3G there is always BLER on the UL!Will give varying AQM and Frequent AQM scores.
7/31/2019 Ad PESQ VerOct2011
18/19
AQM is worse not radio
No or too few AQM scores. Reason No synch.:Voice message. Analyze by setting CG to record no sync .
Round trip delay too long. CG trace parameter Start ofRecording > 2600 samples?
Bad AQM due to low or high volume: , - . -
6dB. Check UL volume, should be > -15dB and < 2dB. Typical -
2dB.
Bad AQM due to background noise:
Record DL or UL audio. Look and listen at audio in silentparts in the beginning and end.
7/31/2019 Ad PESQ VerOct2011
19/19
Some questions from customers How do you know you show the correct AQM scores?
See Accuracy of AQM in TA
Agilent / QVoice shows lower AQM scores than TEMS.WHY?
The audio recordin e ui ment is not the same. Our
equipment has low background noise, very little filtering,correct volume settings from factory.
The mobiles are not the same.
Audio references are different.
Remember we have verified our scores to simulations andin a controlled test environment (Techlab).