World Class Standards
Applications of objective quality assessment models in IPTV
Applications of objectives quality assessment model s in IPTV (Akira Takahashi)
Akira TakahashiNTT Service Integration Labs.
© ETSI 2007. All rights reservedEFFECTS OF TRANSMISSION PERFORMANCE ON MULTIMEDIA QoS
World Class Standards
Agenda
� Introduction
� Scenarios of QoE evaluation
� Classification and application of objective QoE mod els
Applications of objectives quality assessment model s in IPTV (Akira Takahashi) 2
� Classification and application of objective QoE mod els� Media-layer models� Parametric packet-layer models� Bitstream models� Parametric planning models� Hybrid models
� Concluding remarks and future work
World Class Standards
What is QoE?
� The overall acceptability of an application or service, as perceived subjectively by the end user.
NOTES1 Quality of Experience includes the complete end-to-end system effects
(client, terminal, network, services infrastructure, etc).2 Overall acceptability may be influenced by user expectations and context.2 Overall acceptability may be influenced by user expectations and context.
~From ITU-T Recommendation P.10/G.100~
Applications of objectives quality assessment model s in IPTV (Akira Takahashi) 3
World Class Standards
Factors affecting QoE of IPTV services
4
Vide
o
Audi
o
Text
Gra
phic
s
Con
trol f
unct
ion
EPG
Met
adat
a
Brow
ser
Acce
ssib
ility
Prici
ng
Supp
ort
QoE of IPTV services
Applications of objectives quality assessment model s in IPTV (Akira Takahashi)
World Class Standards
ApplianceTelephone
PC
QoS &QoS &QoS &QoS & QoEQoEQoEQoE
TV
Storage serverIP transmitter
Homenetwork
Core/accessnetwork
Transmissionperformance
QoSQoSQoSQoS
STB
TV
Decoder/PLC
performance
Display QualitySourcequality
QoEQoEQoEQoE
Encoder
Coding quality
5Applications of objectives quality assessment model s in IPTV (Akira Takahashi)
World Class Standards
Why QoE evaluation?
� Recent survey indicates that….� 84% of IPTV providers think video quality monitorin g is critical or
very important� 77% say video quality is a main reason for customer churn� 78% say video quality problems are a main reason fo r high volumes
of customer service calls
Applications of objectives quality assessment model s in IPTV (Akira Takahashi) 6
of customer service calls~From http://telephonyonline.com/iptv/news/iptv-video-qua lity-0219/ ~
� There surely is a market need for QoE evaluation me thods/tools!
World Class Standards
Quality monitoring as well as quality design
� Conventional telephone services over PSTN are relat ively stable after a call is connected. � In PSTN services, service providers established tel ephone systems
with static quality design , � then, monitor/manage the “availability” and “functiona lity”
� IP networks (core/access/home -NWs) and IP -based terminals are
Applications of objectives quality assessment model s in IPTV (Akira Takahashi) 7
� IP networks (core/access/home -NWs) and IP -based terminals are time-variant systems.� In IPTV services, media quality varies depending on coding
performance at H/E, traffic conditions in networks, and terminal performance
World Class Standards
What to measure?
� Pre-transmission domain• Encode quality• System failure (blackout, frame freezing etc.)
� Network domain• Packet loss• Delay• Delay• Delay jitter
� Post-transmission domain• Error recovery performance such as FEC (Forward Err or Correction)• Decode quality (including PLC: Packet Loss Concealm ent)• D/A conversion performance• Color management performance• Display performance
Applications of objectives quality assessment model s in IPTV (Akira Takahashi) 8
World Class Standards
When to measure?
� QoE measurement is necessary in the following phase s:
� Before the service is developed:• Individual quality factors (coding distortion etc.)• Quality planning
Applications of objectives quality assessment model s in IPTV (Akira Takahashi) 9
� After the service is developed, but before the service is delivered:• Off-service quality check
� After the service is delivered:• In-service quality monitoring at user end• In-service network mid-point monitoring• On-site quality check (on claim)
World Class Standards
Where to measure?
� Draft ITU-T Recommendation G.1081 ((((IPTV-PMP)))):PrePre--transmission qualitytransmission quality Network qualityNetwork quality
PostPost--transmission transmission
Applications of objectives quality assessment model s in IPTV (Akira Takahashi) 10
PostPost--transmission transmission qualityquality
World Class Standards
How to measure?
� Subjective quality assessment is time consuming and expensive.
� It cannot be applied to on-site and/or real-time measu rement , which is essential in quality monitoring.
Applications of objectives quality assessment model s in IPTV (Akira Takahashi) 11
� Objective quality assessment , which estimates QoE by subjective testing, is the solution!
World Class Standards
Overview of objective quality assessment models
� From the viewpoint of input information into a mode l, objective quality assessment models can be categorized as fol lows:
� Media-layer models
� Parametric packet -layer models
Applications of objectives quality assessment model s in IPTV (Akira Takahashi) 12
� Parametric packet -layer models
� Bitstream-layer models
� Parametric planning models
� Hybrid models
World Class Standards
Media-layer models
Source signal
System under test
Source video
Model
Degradedsignal
Full-reference (FR) models
Source video Degraded
Example:• Codec optimization• Off-service quality check• Content-encoding quality monitoring at head end
• On-site quality check
Example:
Applications of objectives quality assessment model s in IPTV (Akira Takahashi) 13
System under test Model
No-reference (NR) models
System under test Model
Reduced-reference (RR) modelsSide info.
Source video Degradedsignal
Source video Degradedsignal
Example:• In-service quality monitoring at user end
• Content-encoding quality monitoring at head end
Example:• In-service quality monitoring at user end
World Class Standards
Current standards of media-layer models (video)
� ITU-T standardized Rec. J.144, which recommends four full-reference models for Standard Definition TV , in 2004.� Coding distortion only (no transmission errors such as packet loss)
� VQEG finalized its evaluation on full-reference and reduced-reference models applicable to QCIF ~ VGA. The draft Recommendation has been consented at ITU -T SG9 as Rec. J.247,
Applications of objectives quality assessment model s in IPTV (Akira Takahashi) 14
Recommendation has been consented at ITU -T SG9 as Rec. J.247, which includes four models, and Rec. J.246, respectively in May 2008.� Packet-loss degradation as well as coding distortio n is in the scope
of Recommendation.
� ITU-T SG9 considered that current no-reference tech nologies seemed to be premature in terms of quality-estimati on accuracy.
World Class Standards
Parametric packet-layer models
� Only uses packet-header information.� Light-weight implementation.� Can be applied even if payload is encrypted.
Example:• In-service quality monitoring Decoded media Media-layer
model
Applications of objectives quality assessment model s in IPTV (Akira Takahashi) 15
• In-service quality monitoring at user end
• Network mid-point monitoring
Cannot consider…•coding distortion•quality dependence of QoE on content
IPUDPRTP
TS, MP4
Decoded media
Parametric packet-layer
model
model
ES (bitstream)
World Class Standards
Bitstream-layer models
� Use up to coded bitstream (payload), but not decode r output.� Can take into account the content dependence of the impact of
packet loss.
Example:• In-service quality monitoring
Decoded mediaMedia-layer
model
Applications of objectives quality assessment model s in IPTV (Akira Takahashi) 16
• In-service quality monitoring at user end
• Computational load• Cannot be applied if
payload is encryptedIP
UDPRTP
TS, MP4
Decoded media
Parametric packet-layer
model
Bitstream-layer model
model
ES (bitstream)
World Class Standards
Standardization activities for packet-layer and bitstream-layer models
� Parametric packet-layer models� ITU-T SG12 Q.14 & VQEG
• P.NAMS (non-intrusive parametric model for the asse ssment of performance of multimedia streaming)
Applications of objectives quality assessment model s in IPTV (Akira Takahashi) 17
� Bitstream-layer models� ITU-T SG12 Q.14 & VQEG
• P.NBAMS (non-intrusive parametric bitstream model f or the assessment of performance of multimedia streaming)
World Class Standards
Parametric planning models� Use quality planning parameters as input.� ITU-T Recommendation G.1070 for videophone applicat ions.� ITU-T SG12 Q.13 is working on G.OMVAS (opinion mode l for video
and audio streaming applications).
1) Speech quality Speech-related quality parameters.
Quality estimation model defined in Quality estimation model defined in Quality estimation model defined in Quality estimation model defined in ITUITUITUITU----T Recommendation G.1070T Recommendation G.1070T Recommendation G.1070T Recommendation G.1070
Coding scheme, bitrate
Applications of objectives quality assessment model s in IPTV (Akira Takahashi) 18
1) Speech quality estimation model
(E-model)
2) Video qualityestimation model
(proposed by NTT)
Video-related quality parameters.
EstimatedMOS
3) Multimedia qualityintegration model(proposed by NTT)
� Coding scheme, bitrate� Packet-loss rate� Delay� Loudness, echo
� Coding scheme, bitrate� Packet-loss rate� Delay� Image resolution, display size Example:
• Quality planning
World Class Standards
Concluding remarksMarket trend:� QoE evaluation technologies are needed in IPTV mark et.� Testing equipment market will shift from R&D usage to
maintenance and monitoring applications.
Application scenarios:Application scenarios:� There are various applications of objective quality models in
different quality design/management phases.� Standardization activities are quite active in ITU- T SG9, SG12, and
ITU Video Quality Experts Group (VQEG).
Future issues:� Real-time quality estimation for monitoring� Light-weight, but accurate models for end user moni toring
Applications of objectives quality assessment model s in IPTV (Akira Takahashi) 19