Post on 14-Oct-2020
transcript
Standards for Speech and media Quality assessment
Jean-Yves MonfortETSI TC STQ Chair
© ETSI 2010. All rights reserved
ETSI Workshop 3GPP release 8 IMS Implementation, Deployment & Testing
Outline
� Two topics� Testing Terminal Quality to enhance the end-to-end speech Quality� Quality assessment of Multiplay services
� How testing different VoIP (or DECT, Mobile) termin al implementations to prove and improve perceived qual ity and optimize performance?
2
� From the users point of views the quality shall be assessed including terminals
� ETSI standards defining terminal requirements are i ntended to ensure best possible quality
• Test events, Plugtests,… are based on standards for quality assessment• Results of test events provide feedbacks on require ments vs real
implemenations ���� standards requirements evolvements.
� Criteria for quality assessment of multiplay servic es� Exemple of comparison of ISP VoIP: Interoperability testing� Future plans for IPTV
Part 1 : STQ Standards for speech terminals-Narrowband (300-3400Hz)- Wideband (150Hz- 7kHz)
3
Handset and headset HandsfreeTransmission requirements for VoIP terminals from a
QoS perspective as perceived by the user
Links with other standardization bodies and certification activities
� ETSI STQ has disseminated the deliverables to other bodies� TS DECT� 3GPP SA4� ITU-T SG12And also
4
And also� GCF� DECT Forum/CATIQ
� The only way to ensure a good end to end quality is � To ensure that all speech terminals (whatever acces s network) are
providing equivalent speech quality� To avoid as far as possible coding and speech proc essing
cascading/tandeming� To reduce the end-to end delay
Plugtest (STQE) Reports� At the end of the Test Event
� An anonymous report (also made available as a contr ibution to TC STQ activities)compiling the results
� A specific and confidential report given individual ly to each equipment manufacturer participating to the Plugtes t
� As ETSI deliverables:
5
� As ETSI deliverables:TR 102 648-Series : Test methodologies for ETSI test events and
results Part 1 : VoIP Speech Quality TestingOther parts : ETSI Plugtests Speech Quality Test Event Reports
Plugtest (STQE) methodologies� Test set-up for 5 th ETSI SQTE
NISTNet
Packet Loss
Delay
IP
PacketMonitor
Gateway
6
aethraD2000 PRO
Delay
Input/Output
Input/Output
IP
E1/ISDN DSS1
IP-Terminal(Phone or PC)
HEAD acoustics Test System
Plugtest (STQE) methodologies� Test set-up for 5 th ETSI SQTE
Sub-wooferSub-
woofer
IP
Packet LossDelay
“Interface” - Gateway
IPISDN
Simulator
ISDN
Simulator
NISTNet
Receive direction
Send direction
far endIP
Packet LossDelay
“Interface” - Gateway
IPISDN
Simulator
ISDN
Simulator
NISTNet
Receive direction
Send direction
far end
7
MFE
ListeningOnly Test
ETSI EG 202 396-3
S-MOS N-MOS G-MOS
unprocessed
processed
clean
processedMFE
ListeningOnly Test
ETSI EG 202 396-3
S-MOS N-MOS G-MOS
unprocessed
processed
clean
processed
Speech sample (example 1) for MOS testing
InRCV
OutRCV
InSND
OutSND
USB
MFE VI
Send direction
EC under testnear end
Test system
ACQUA
InRCV
OutRCV
InSND
OutSND
USB
MFE VI
Send direction
EC under testnear end
Test system
ACQUA
Test setup for Echo cancellor testing
Plugtest (STQE) methodologies� Variable parameters
� Packet loss %, Networtk delay
� Quality parameters� PLC (packet Loss Concealment) Performance� VAD (Voice Activity detection) performance� MOS-LQO Mean Opinion Score – Listening Speech Quality Objective
8
� MOS-LQO Mean Opinion Score – Listening Speech Quality Objective� G-MOS (Global quality with background noise –according to EG 202396-3 )
� Conversational parameters� Performance for handset use like one-way transmissi on parameter,
echo performance tests, double talk and background noise transmission tests,
� Tests covering hands-free capability ( frequency responses , echo performance tests , double talk capability and background noise transmission quality),
� Wideband tests in handset and hands-free mode using the G.722 speech coder.
Plugtest (STQE) methodology: The Pie Diagram
9
DT: Double talk
HFT: Hands Free Terminal
TCLw: Weighted terminal Coupling Loss
BGNT: Background Noise Transmission
Plugtest methodologies, results and lessonsElectrical-acoustical set-ups
� The “Quality Pies” of the tested IP phone implement ations :
G.711 3%
4
5
G.711 SND
3
4
5
WB HFT TMOS RCV
2
3
4
WB HS TMOS RCV
3
G.711 3%
4
5
G.711 SND
3
4
5
WB HFT TMOS RCV
2
3
4
WB HS TMOS RCV
3
10
G.711, Jitter
23
4
2
3
2
2
2
HFT TMOS RCV
23
45
HFT Double Talk
2c2b2a1
3
HFT Echo Performance
30%
70%
Ok
BGNT (far)
15
10
5
0
Double Talk
2c
2b
2a
1 Echo Performance
30%
70%
Ok
G.729 Jitter
23
4
G.711, Jitter
23
4
2
3
2
32
2
HFT TMOS RCV
23
45
HFT Double Talk
2c2b2a1
3
HFT Echo Performance
30%
70%
Ok
BGNT (far)
15
10
5
0
Double Talk
2c
2b
2a
1 Echo Performance
30%
70%
Ok
G.729 Jitter
23
4
SND: send RCV: Receive
Plugtest (STQE) results� The “Quality Pies” of the IP phone implementations (continued)
G.711, Jitter
G.711 3%
2
3
4
5
G.711 SND
2
3
4
5
WB HFT TMOS RCV
2
3
4
WB HS TMOS RCV
2
3
HFT TMOS RCV5
11
23
4
23
45
HFT Double Talk
2c2b2a1
3
HFT Echo Performance
30%
70%
Ok
BGNT (far)
15
10
5
0
Double Talk
2c
2b
2a
1 Echo Performance
30%
70%
Ok
G.729 Jitter
23
4
Plugtest Benefits
� For standardization bodies� Validation of test methodologies in interoperabilit y context� Validation / Update of the requirements for future versions of
standards
� For test equipment manufacturers and test laborator ies
12
� Implementation of the standards and test equipment development� Direct contact with the terminal manufacturers and test of new
terminal implementations� To provide diagnosis when identifying implementatio n not in
conformance with the standards
� For terminal equipment manufacturers� To test the new implementations both in laboratory and
interoperability contexts� To have diagnosis when no conformance for some para meters� To compare the terminal performance with other manu facturers
Part 2: STQ Standards For Multiplay services
� Voice Quality metrics Two ETSI Standards
� ES 202 765-2 QoS and network performance metrics and measurement methods; part 2: Transmission Quality Indicator
13
measurement methods; part 2: Transmission Quality Indicator combining Voice Quality Metrics
� ES 202 765-4 QoS and network performance metrics and measurement methods; part 4 : Indicators for supervision of multiplay services
Multiplay services assesments� Speech Quality Analysis. Several campaigns already done
The test methods and results are made available in� TR 102 716 Series: Guidelines, objectives and resul ts of speech
quality analysis in the context of interworking plu gtests for multiplay services
Pert1: Guidelines and objectives
14
• Pert1: Guidelines and objectives• Part 2: Results (one annex / test campaign)
� IPTV Quality� Results are expected in the next months.
Multiplay services assesments� VoIP Quality – Test set-up for interoperability assessment (6 ISP s)
IP Network of ISP 2HGW 1
HGW 2HGW 1
IP to IP intra-ISP Call
15
IP Network of ISP 1
IP Network of ISP 4
IP Network of ISP 3
HGW 1
HGW 3
HGW 4PSTNIP to IP inter-ISP Call
IP to PSTN Call
Multiplay services assesments� VoIP Quality – Metrics
� Post Dialling Delay� Listening speech quality� End to End delay� Listening speech quality stability
16
� Listening speech quality stability� Active level of speech signal at reception� Reception noise level� Speech signal attenuation (or gain) after transmiss ion� Talker Echo Delay� Talker Echo Attenuation� DTMF integrity
Multiplay services assesments (VoIP – results)� Post Dialing Delay (PDD)
PDDISP1 to ISP2
1.34
3
4
2
0PDD
ISP1 to ISP31.35
3
4
2
0
PDDISP1 to ISP4
PDDISP6 to ISP1
PDDISP7 to ISP1
1.66
3
4
2
0
PDDISP4 to ISP1
0.59
3
4
2
0PDD
ISP4 to ISP21.38
3
4
2
0
PDDISP4 to ISP3
1.49
PDDISP6 to ISP4
5.33
PDDISP7 to ISP4
2.33
3
4
2
0
PDDISP1 to PSTN
1.07
3
4
2
0
PDDISP2 to PSTN
1.68
3
4
2
0
PDDISP3 to PSTN
1.150
PDDPSTN to ISP6
1.10
PDDPSTN to ISP7
0.79
3
4
2
0
17
8
6
8
6
4 ISP1 to ISP40.72
3
86
42
0
1.33PDD
ISP1 to ISP6
3
86
42
0
1.33PDD
ISP1 to ISP7
3
8
6
4
2
0
1.47PDD
ISP2 to ISP1
3
8
6
4
2
0
0.76PDD
ISP3 to ISP1
3
8
6
4
2
0
0.6PDD
ISP4 to ISP1
3
86
42
0
ISP6 to ISP11.01
3
86
42
0
8
6
4
ISP 1
8
6
8
6
1.49
3
86
42
0
1.45PDD
ISP4 to ISP6
3
86
42
0
1.24PDD
ISP4 to ISP7
3
8
6
4
2
0
0.71PDD
ISP1 to ISP4
3
8
6
4
2
0
1.12PDD
ISP2 to ISP4
3
8
6
4
2
0
5.45PDD
ISP3 to ISP4
3
86
42
0
5.33
3
86
42
0
8
6
ISP 4ISP to PSTN
8
6
8
6
4
3
8
6
4
2
PDDISP4 to PSTN
1.0938 6 4 2 0
1.19PDD
ISP6 to PSTN
3
8
6
4
2
0
1.38PDD
ISP7 to PSTN
3
8
6
4
2
0
0.99PDD
PSTN to ISP1
3
8
6
4
2
02.21PDD
PSTN to ISP2
3
8
6
4
2
0
1.17PDD
PSTN to ISP3
3
8
6
4
2
0
0.72PDD
PSTN to ISP4
3 86420
3
8
6
4
2
8
6
4
Multiplay services assesments (VoIP – results)� Listening Quality (MOS-LQON)
MOS-LQONISP1 to PSTN
4.29
44
5
MOS-LQONISP2 to PSTN
4.38
44
5
MOS-LQONISP3 to PSTN
4.395
MOS-LQONPSTN to ISP6
4.535
MOS-LQONPSTN to ISP7
4.42
44
5
MOS-LQONISP1 to ISP2
4.14
44
5MOS-LQONISP1 to ISP3
4.1
44
5
MOS-LQONMOS-LQON
MOS-LQONISP67 to ISP1
4.34
44
5
MOS-LQONISP7 to ISP1
4.32
44
5MOS-LQONISP7 to ISP2
4.45
MOS-LQONISP6 to ISP7
4.355
18
ISP 1 ISP 6ISP to PSTN
3 3
4
3
4
5
MOS-LQONISP4 to PSTN
4.3743 4 5
4.39MOS-LQON
ISP6 to PSTN
4
3
4
5
4.42MOS-LQON
ISP7 to PSTN
4
3
4
5
4.44MOS-LQON
PSTN to ISP1
4
3
4
54.41
MOS-LQONPSTN to ISP2
4
3
4
5
4.36MOS-LQON
PSTN to ISP3
4
3
4
5
4.4MOS-LQON
PSTN to ISP4
4 345
4
3
4
5
33
4
3
4 MOS-LQONISP1 to ISP4
4.09
4
3
4
5
4.06MOS-LQONISP1 to ISP6
4
3
4
5
4.06MOS-LQONISP1 to ISP7
4
3
4
5
4.34MOS-LQONISP2 to ISP1
4
3
4
5
4.32MOS-LQONISP3 to ISP1
4
3
4
5
4.25MOS-LQONISP4 to ISP1
4
3
4
5
MOS-LQONISP6 to ISP1
4.27
4
3
4
5
4
3
4
3
4
3
4 MOS-LQONISP7 to ISP3
4.39
4
3
4
5
4.33MOS-LQONISP7 to ISP4
4
3
4
5
4.35MOS-LQONISP7 to ISP6
4
3
4
5
4.07MOS-LQONISP1 to ISP7
4
3
4
5
4.37MOS-LQONISP2 to ISP7
4
3
4
5
4.38MOS-LQONISP3 to ISP7
4
3
4
5
MOS-LQONISP4 to ISP7
4.26
4
3
4
5
4
3
4
Multiplay services assesments (VoIP – results)� End to End Delay
� Improvements are needed: • The end to end delay is in most cases too long (eg ISP1 and some
ISP/PSTN configurations)
� It is possible to reach shorter delays (See ISP3)Delay
ISP1 to ISP2235
0Delay
ISP1 to ISP3Delay
ISP67 to ISP1
DelayISP3 to ISP1
357
DelayISP1 to PSTN
152 DelayDelay
19
ISP 1 ISP 3ISP to PSTN
150
400
300
200
100
0 ISP1 to ISP3252
150
400
300
200
100
0
DelayISP1 to ISP4
165
150
400300
200100
0
218Delay
ISP1 to ISP6
150
400300
200100
0
219Delay
ISP1 to ISP7
150
400
300
200
100
0
244Delay
ISP2 to ISP1
150
400
300
200
100
0
346Delay
ISP3 to ISP1
150
400
300
200
100
0
195Delay
ISP4 to ISP1
150
400300
200100
0
DelayISP6 to ISP1
317
150
400300
200100
0
ISP67 to ISP1298
150
400
300
200
100
0
357
150
400
300
200
100
0Delay
ISP3 to ISP2268
150
400
300
200
100
0
DelayISP3 to ISP4
308
150
400300
200100
0
67Delay
ISP3 to ISP6
150
400300
200100
0
217Delay
ISP3 to ISP7
150
400
300
200
100
0
241Delay
ISP1 to ISP3
150
400
300
200
100
0
206Delay
ISP2 to ISP3
150
400
300
200
100
0
255Delay
ISP4 to ISP3
150
400300
200100
0
DelayISP6 to ISP3
71
150
400300
200100
0
DelayISP7 to ISP3
211
150
400
300
200
100
0
152
150
400
300
200
100
0
DelayISP2 to PSTN
204
150
400
300
200
100
0
DelayISP3 to PSTN
175
150
400
300
200
100
0
DelayISP4 to PSTN
158150400 300 200 100 0
168Delay
ISP6 to PSTN
150
400
300
200
100
0
118Delay
ISP7 to PSTN
150
400
300
200
100
0
194Delay
PSTN to ISP1
150
400
300
200
100
0141
DelayPSTN to ISP2
150
400
300
200
100
0
109Delay
PSTN to ISP3
150
400
300
200
100
0
146Delay
PSTN to ISP4
150 4003002001000
DelayPSTN to ISP6
95
150
400
300
200
100
0
DelayPSTN to ISP7
70
150
400
300
200
100
0
Multiplay services assesment (IPTV)
Internet
Internet service
20
IP core network
Audiovisual Service
platforms
VoIP Service
platforms
Accessand
Backhaul
networks
Game service
platforms
VoIP service
IPTV and VoD services
Game service
Multiplay services assesments� IPTV Quality – Metrics
� Channel Availability � Service Group Channel Availability� Video Quality ( no standardized algorithm )� Audio Quality
21
� Audio Quality� "Black Screen" Occurrences� Blockiness Occurrences� Frozen Picture Occurrences� Lip Desynchronization Occurrences� Zapping Delay� Transmission Delay� IPTV service boot delay
Multiplay services assesments� IPTV Quality –
Metrics – Example
� No result currently available
Channel 1Availability
98.2
98
97
98
99
100Channel 2Availability
100
9898
99
100
Channel 3Availability
Zapping delay
Boot delay
7893
10000
13334
6667
0
22
available
95
96
95
96
97Availability
99.2
98
9596
9798
99100
96.7Channel 4Availability
98
9596
9798
99100
2Audio Quality
10
21
14
7
0
13.7Blockiness
Occurrences
10
21
14
7
0
0.2Black ScreenOccurrences
10
21
14
7
0
2.8Frozen PictureOccurrences
10
2114
70
4225
3000
80006000
40002000
0
20001
13334
Thanks
�For additionnal information
23
�For additionnal information �Jeanyves.monfort.6@orange.fr
Acknowledgments
�Thanks for their contributions to
24
�Thanks for their contributions to�H.G. Gierlich, Head Acoustics�Jean-Yves Le Saout, France Telecom
Annex: ETSI Standards listed in the presentation� Speech terminals
� ETSI ES 202 740 V1.3.2 (2010-09) Title: Speech and multimedia Transmission Quality (STQ);Transmission requirement s for wideband VoIP loudspeaking and handsfree terminals from a Qo S perspective as perceived by the user
� ETSI ES 202 739 V1.3.2 (2010-09) Title: Speech and multimediaTransmission Quality (STQ);Transmission r equirements
25
multimediaTransmission Quality (STQ);Transmission r equirements for wideband VoIP terminals (handset and headset) f rom a QoS perspective as perceived by the user
� ETSI ES 202 738 V1.3.2 (2010-09) Title: Speech and multimedia Transmission Quality (STQ);Transmission requirement s for narrowband VoIP loudspeaking and handsfree terminal s from a QoS perspective as perceived by the user
� ETSI ES 202 737 V1.3.2 (2010-09) Title: Speech and multimedia Transmission Quality (STQ);Transmission requirement s for narrowband VoIP terminals (handset and headset) fro m a QoS perspective as perceived by the user
Annex: ETSI Standards listed in the presentation
� Multiplay services
� ETSI ES 202 765-2 V1.1.4 (2010-07) Title: Speech an d multimedia Transmission Quality (STQ);QoS and network performa nce metrics and measurement methods;Part 2 : Transmission Quali ty Indicator combining Voice Quality Metrics
26
combining Voice Quality Metrics � ETSI ES 202 765-4 V1.1.1 (2010-10) Title: Speech an d multimedia
Transmission Quality (STQ);QoS and network performa nce metrics and measurement methods;Part 4: Indicators for supe rvision of Multiplay services
� ETSI deliverables may be downloaded fromhttp://pda.etsi.org/pda/queryform.asp