Date post: | 17-May-2015 |
Category: |
Technology |
Upload: | stephane-pechard |
View: | 2,163 times |
Download: | 4 times |
1
Quality of experiencein High Definition Television:
subjective assessmentsand objective metrics
Stéphane PéchardOctober, 2nd 2008
IVC
3
New technologies
capture compression5x SDTV (pixels)
transmission restitution
1 01
0110
01 010 1 0
101
1101
010 101
1NEW
NEWNEW
NEW
=> new distortions
5
OutlineSubjective quality
assessment1. global quality
assessment
2. comparing qualities of 2 TV services
3. towards a fine quality measurement
Objective quality metrics
1. H.264-specific metric (using prior knowledge)
2. generic metric based on spatio-temporal tubes
6
OutlineSubjective quality
assessment1. global quality
assessment
2. comparing qualities of 2 TV services
3. towards a fine quality measurement
Objective quality metrics
1. H.264-specific metric (using prior knowledge)
2. generic metric based on spatio-temporal tubes
7
What is video quality subjective assessment?
getting a mean human quality evaluation
observers environment methodology
8
Subjective quality assessment
preference between HDTV and SDTV ?
how quality is globally perceived ?
can we better understandquality judgment ?
9
OutlineSubjective quality
assessment1. global quality
assessment
2. comparing qualities of 2 TV services
3. towards a fine quality measurement
Objective quality metrics
1. H.264-specific metric (using prior knowledge)
2. generic metric based on spatio-temporal tubes
10
Suitable methodology
HDTV: high quality in a short range=> quality measure should be precise
and discriminative
+ important part of visual field excited=> how to consider this in a methodology ?
11
- random order
- only one viewing
- category scale
- no explicit reference
AbsoluteCategory Rating
Subjective AssessmentMethodology
for Video Quality
Video Quality Experts Group
European BroadcastingUnion
...Good
- user-driven order
- multiple viewing (natural?)
- continuous scale
- explicit reference
12
State of the art[Brotherton, 2006] both MOS (Mean Opinion
Score) populations correlation on CIF (352x288):CC(MOS
ACR, MOS
SAMVIQ) = 0.94
to confirm:more tests
QVGAVGA
HDTV
640320
1920
1080
480
240
13
Results
HDTV
VGA
QVGA 13°
19°
33°
0.969
0.942
0.899
6.73
9.31
14.06
visualfield
RMSDiff=correlation
ACR and SAMVIQ are equivalentup to a certain resolution
14
Accuracy vs.Number of observers
5 10 15 20 25 300
5
10
15
SAMVIQACR'
con
fide
nce
inte
rval
number of observers
SAMVIQ
24
15
OutlineSubjective quality
assessment1. global quality
assessment
2. comparing qualities of 2 TV services
3. towards a fine quality measurement
Objective quality metrics
1. H.264-specific metric (using prior knowledge)
2. generic metric based on spatio-temporal tubes
19
Quality and preference tests
I prefer much more A than B I prefer more A than B
I prefer a little more A than B I have no preference
I prefer a little less A than B I prefer less A than B
I prefer much less A than B
+3+2+10-1-2-3
preference scale
A: quality testswith SAMVIQ
of SDTV qualities(good and mid-range)
B: quality testswith SAMVIQ
of HDTV qualities
preference testsA vs. B
20
Resultspr
efer
ence
ΔQuality
ΔQuality =MOS
HD - MOS
SD
0
HD/SD Qgood
: QHD may be less than QSD,
benefit of the size
HD/SD Qmid-range
: QHD must be higher
than QSD, size becomes an enemy
isopreference
0
21
OutlineSubjective quality
assessment1. global quality
assessment
2. comparing qualities of 2 TV services
3. towards a fine quality measurement
Objective quality metrics
1. H.264-specific metric (using prior knowledge)
2. generic metric based on spatio-temporal tubes
23
blockiness
blur
Farias approach-2004 Proposed approach
distortion-based partition content-based partition
...
from disturbance functionsto global distorting system
homogeneousareas
fine texturedareas
strong textured areas
t
from spatio-temporalcategory qualitiesto global quality?
blur
Drawbackscontent dependency
coding system dependencydistortion list exhaustivity
pooling function?complex subjective assessment
24
H.264 coding
class-distorted sequencesgeneration
source
categoriesmasks sequence
partly-distorted sequencesusable for subjective tests
spatio-temporal classification
… …
spatio-temporal segmentation(tube creation
along local motion)tube classification
C5
C4
C3
C2
C1
25
Local to global?
MOS(Ci): partly-distorted sequence qualitiesrelated to global MOS
G: f(MOS(Ci)) = MOS
G ?
several relation tested:up to CC(f(MOS(Ci)), MOS
g) = 0.95
YES! It's possible to relate spatio-temporalcategory qualities to global quality
26
blockiness
blur
Farias approach-2004 Proposed approach
distortion-based partition content-based partition
...
from disturbance functionsto global distorting system
homogeneousareas
fine textured areas
strong textured areas
t
blur
Drawbackscontent dependency
coding system dependencydistortion list exhaustivity
pooling function?complex subjective assessment
Advantagesgeneric methodology
simple pooling functionreal distortions
classical subjective assessment
27
OutlineSubjective quality
assessment1. global quality
assessment
2. comparing qualities of 2 TV services
3. towards a fine quality measurement
Objective quality metrics
1. H.264-specific metric (using prior knowledge)
2. generic metric based on spatio-temporal tubes
28
distortedsequence
What are objective quality metrics?
system
reference reducedreferenceextraction
objectivescores
FR metricRR metricNR metric
MOS from subjectiveassessments
evaluationperformance
criteria(CC, RMSE, OR,
difference signifiance)
29
Usual approaches
signalapproach
perceptualapproach
PSNR low levelHVS models
structuralmodels
high level distorstionsmeasurement models
VSSIM [2004]
VQM [2002]
30
Performances on HDTV
metric CC RMSE ORVSSIM 0.790 11.27 0.55VQM 0.898 8.09 0.40PSNR 0.543 15.43 0.61
168 sequences
31
OutlineSubjective quality
assessment1. global quality
assessment
2. comparing qualities of 2 TV services
3. towards a fine quality measurement
Objective quality metrics
1. H.264-specific metric (using prior knowledge)
2. generic metric based on spatio-temporal tubes
33
ST contentanalysis
referencesequence
distortedsequence
modelparametersprediction
qualitymodel
bitrate B
offset, slope
global motion Mproportions P
i
quality score Q
34
ST contentanalysis
referencesequence
distortedsequence
modelparametersprediction
qualitymodel
bitrate B
offset, slope
quality score Q
use of the spatio-temporalsegmentation
class proportions Pi
60%
10%
5%
20%
mean sequencemotion M
global motion Mproportions P
i
35
ST contentanalysis
referencesequence
distortedsequence
modelparametersprediction
qualitymodel
bitrate B
offset, slope
quality score Q
offset parameter:temporal complexity estimation
related to motion Mi
slope parameter:spatial complexity estimationrelated to class proportions P
i
global motion Mproportions P
i
36
Performancesmetric CC RMSE ORVSSIM 0.791 11.90 0.45VQM 0.892 8.79 0.40
proposed 0.901 8.47 0.36
pros cons
H.264-dependentfaster than VQM
reduced referencemetric (6 parameters)
equal performances
37
OutlineSubjective quality
assessment1. global quality
assessment
2. comparing qualities of 2 TV services
3. towards a fine quality measurement
Objective quality metrics
1. H.264-specific metric (using prior knowledge)
2. generic metric based on spatio-temporal tubes
38
Visual inspection (gaze fixation)spatially localized
duration (200-300 ms)smooth local motion tracking
Interesting HVS features for this metric
some of them have been used in part 1
39
spatio-temporalsegmentation
referencesequence
distortedsequence
featuresextraction
qualityscore Q
featuresextraction
featuresdifference
long-termtemporalpooling
tubes
spatio-temporalsegmentation
short-termspatio-temporal
pooling
40
spatio-temporalsegmentation
referencesequence
distortedsequence
featuresextraction
qualityscore Q
featuresextraction
featuresdifference
temporalpooling
tubes
spatio-temporalsegmentation
short-termspatio-temporal
pooling
a tube t
41
spatio-temporalsegmentation
referencesequence
distortedsequence
featuresextraction
qualityscore Q
featuresextraction
featuresdifference
temporalpooling
tubes
spatio-temporalsegmentation
short-termspatio-temporal
pooling
spatial information feature: fSI
temporal information feature: fTI
referencetube
distortedtube-
42
spatio-temporalsegmentation
referencesequence
distortedsequence
featuresextraction
qualityscore Q
featuresextraction
featuresdifference
tubes
spatio-temporalsegmentation
short-termspatio-temporal
pooling
5 frames
=
1 time-slot (200ms)
long-termtemporalpooling
43
spatio-temporalsegmentation
referencesequence
distortedsequence
featuresextraction
qualityscore Q
featuresextraction
featuresdifference
long-termtemporalpooling
tubes
spatio-temporalsegmentation
short-termspatio-temporal
pooling
high level HVS properties
asymetricaltemporalfiltering
mid-termnon linear
quality judgment
long-termtemporalfiltering
45
Best performancesmetric CC RMSE OR
VSSIM 0.837 10.15 0.38
VQM 0.875 8.98 0.43
fixed tubes 0.875 9.08 0.38
motion-oriented tubes 0.898 8.30 0.31
generic metric
slightly better than VQM with less features
47
better knowledge of HDTV (visual)subjective quality assessment
Subjective quality assessment
generic methodology to assess fine quality => better knowledge
of judgment construction
visual image size influences preferencebetween SDTV/HDTV services
48
Experiment effort
26 sessions (6 months)(SAMVIQ, ACR and preference)
200 observers for 600 unique sessionsin 300 hours of subjective evaluation
=> 25,000 subjective scores
more than 750 cumulative daysof H.264 coding
49
fast RR metric dedicatedto H.264 systems evaluation
generic metric based on motion-orientedspatio-temporal tubes
both performed slightly better than VQM
Objective quality metrics
50
Future works
towards a multimodal quality evaluation
considering a display model=> work in progress (Tourancheau)
adapting ACR to HDTV: more than 5 items?=> work in progress (VQEG)
55
mea
n p
refe
ren
ce
ΔMOS=MOSHD
-MOSSD
HDTV prefered
SDTV prefered
MOSHD
<MOSSD
MOSHD
>MOSSDΔMOS
0=-8
ΔMOS0=-18
large screen effect distorsions effect
56
Classes
five spatial activity levels
smooth areas edgestextured areaslow high
luminance
C1
C2
C3
C4
C5
fine strongtextures
57
Tube classification
4 spatial gradientsper tube
plot in spatial space P
C4
C1
C3
C4
C5(P')
C2
ΔV
ΔH
space P
frontiers definedto get relevantclassification