Objective and Subjective Evaluation for
Telecommunications Services and Video Quality
Arthur Webster, NTIA/ITS
Rapporteur Q21/9, Co-Chair VQEG
IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 2004
Introduction
• Evaluation of Video Services Needed– Service Level Agreements, Contracts
• Subjective Most Accurate—but Expensive
• Objective Methods Needed
• Standardized Methods Critical
Standards Groups Involved in Video Quality Evaluation
• ITU-T SG9 and SG12– Q21/9 and Q4/9– G.1000, G.1010 in SG12
• ITU-R WP 6Q
• T1A1 (Committee T1)
• IEEE Broadcasting Technology
Video Quality Recommendations
• BT.500 Methodology for the subjective assessment of the quality of television pictures
• P.910 Subjective video quality assessment methods for multimedia applications
• P.911 Subjective audiovisual quality assessment methods for multimedia applications
• P.920 Interactive test methods for audiovisual communications• P.930 Principles of a reference impairment system for video• P.931 Multimedia communications delay, synchronization and frame rate
measurement• J.143 User requirements for objective perceptual video quality
measurements in digital cable television• J.144 Objective perceptual video quality measurement techniques for
digital cable television in the presence of a full reference
Standardization of Objective Video Quality Evaluation
Methods
ITU-T & ITU-R
T1A1 & IEEE VQEGValidation
Testing
Results
Industry & Academia
Standards and Reports
VQEG
• Video Quality Experts Group – Founded in 1997 as Rapporteurs’ Group– ITU-T SG12, ITU-T SG9, ITU-R WP11E(6Q)– Currently over 200 subscribers to the email list– Latest meeting in February 2002
• VQEG Website: www.its.bldrdoc.gov/vqeg
VQEG Phase I
• Ten Proponent Systems
• Over 26,000 Subjective Opinion Scores
• 20 Different Source Sequences Processed by 16 Different Video Systems
• 8 Independent Laboratories Worldwide.
VQEG Phase I Results
• No Single Winner—8 Way Tie
• Resulted in J.144 – 8 Objective Models Described in Appendix
• 80 Gbytes of Subjectively Rated Video Available to the Research Community
VQEG Phase IICurrent Work
• Full Reference TV– Inclusion of Expert Viewer Test
• Reduced Reference – No-reference TV– Most Useful for Monitoring
• Multimedia– Videoconferencing, Streaming, Webcasting
• VQEG Will Not Determine Winners—Report Results to Standards Bodies
T1A1 Current WorkFamily of 5 Technical Reports
• T1.TR.PP.72-2001 Methodological Framework for Specifying Accuracy and Cross-calibration of Video Quality Metrics
• T1.TR.PP.73-2001 Video Normalization Methods Applicable to Objective Video Quality Metrics Utilizing a Full Reference Technique
• T1.TR.PP.74-2001 Objective Video Quality Measurement Using a Peak-Signal-to-Noise-Ratio (PSNR) Full Reference Technique
• T1.TR.PP.75-2001 Objective Perceptual Video Quality Measurement Using a JND-based Full Reference Technique
• T1.TR.PP.77-2002 Data and Sample Program Code to Be Used With the Method Specified in Technical Report [72] for the Calculation of Resolving Power of the Video Quality Metrics in Technical Reports [74] and [75]
• TRs Input to the ITU for Further Validation– T1.TR-72-2001 Used to Judge Accuracy of Metrics
0.95
0.90
0.85
0.80
0.75
0.70
0.65
0.60
0.550 0.1 0.2 0.3
VQM(2) - VQM(1) (Normalized Scale)
Ave
rage
co
nfid
en
ce th
at
VQ
M(1
) i s
wo
rse
tha
n V
QM
(2)
0.4 0.5 0.6 0.7
1
Using T1.TR-72-2001 to Assess VQM Resolving Power
Using T1.TR-72-2001 To Classify Errors for a
Particular VQM
IEEE Broadcasting SocietyIEEE G-2.1.6
• P1486-D06 Draft Standard on Subjective Measurement of Visual Impairments in Digital Video Using an IEEE‑JND Scale – Based on Work Done at NASA
– Develops an Absolute Video Quality Scale
– Can Be Used to Calibrate Video Quality Metrics
• Input to the ITU for review
Conclusions
• Video QOS Evaluation Is Important
• Work Ongoing in Several Standards Bodies
• VQEG Is Conducting New Validation Tests
• Work Progresses as Resources are Committed to Projects
IP Cablecom and MEDIACOM 2004IP Cablecom and MEDIACOM 2004
