The Effectiveness of a QoE - Based Video Output Scheme for Audio-

Video IP Transmission

The Effectiveness of a QoE - Based Video Output Scheme for Audio-

Video IP Transmission

Shuji Tasaka, Hikaru Yoshimi, Akifumi Hirashima, Toshiro Nunome

Department of Computer Science and Engineering,Graduate School of Engineering, Nagoya Institute of

Technology

ACM Multimedia 2008

Shuji Tasaka, Hikaru Yoshimi, Akifumi Hirashima, Toshiro Nunome

Department of Computer Science and Engineering,Graduate School of Engineering, Nagoya Institute of

Technology

ACM Multimedia 2008

IssueIssue

Conceal the impairment typified by packet loss, error, and delay in IP networks

Two ways these impairments can be remedied at the receiver are: Error-concealment Video frame skipping

These techniques approach this issue with a unique tradeoff Temporal vs. Spatial

Typically the benefits are exclusive

Conceal the impairment typified by packet loss, error, and delay in IP networks

Two ways these impairments can be remedied at the receiver are: Error-concealment Video frame skipping

These techniques approach this issue with a unique tradeoff Temporal vs. Spatial

Typically the benefits are exclusive

SchemeScheme

Switching between error Concealment and frame Skipping (SCS) utilizes this tradeoff between spatial and temporal quality to cope with video packet loss

SCS aims to improve Quality of Experience (QoE) as it depends on both spatial and temporal quality by mixing error concealment and frame skipping

Switching between error Concealment and frame Skipping (SCS) utilizes this tradeoff between spatial and temporal quality to cope with video packet loss

SCS aims to improve Quality of Experience (QoE) as it depends on both spatial and temporal quality by mixing error concealment and frame skipping

OutlineOutline

IntroductionSCS theoryAims of studyQoE measurementQoE estimationThreshold selectionConclusions

IntroductionSCS theoryAims of studyQoE measurementQoE estimationThreshold selectionConclusions

PrinciplePrinciple

SCS switches from error-concealment to frame skipping when a percentage of video slices error-concealed in a frame (Rc) exceeds a threshold value (Th)

Frame skipping continues until a new intra-coded frame is decoded

Optimal Th is dependent on the content type

SCS switches from error-concealment to frame skipping when a percentage of video slices error-concealed in a frame (Rc) exceeds a threshold value (Th)

Frame skipping continues until a new intra-coded frame is decoded

Optimal Th is dependent on the content type

Error Concealment StrategyError Concealment Strategy

I Frames Missing block is interpolated from its

neighboring blocks in the current frameP Frames

Missing block is replaced by the corresponding block in the previous output frameInstead of motion copy

I Frames Missing block is interpolated from its

neighboring blocks in the current frameP Frames

Missing block is replaced by the corresponding block in the previous output frameInstead of motion copy

OutlineOutline

IntroductionSCS theoryAims of studyQoE measurementQoE estimationThreshold selectionConclusions

IntroductionSCS theoryAims of studyQoE measurementQoE estimationThreshold selectionConclusions

Previous LimitationsPrevious Limitations

No audio accompanies the video Output quality determined by PSNR, which is

a QoS parameter (non-perceptual) PSNR evaluates no temporal qualities No real-time estimation of QoE

Full Reference (objective QoE) models compare stream with the original though not in real-time

No audio accompanies the video Output quality determined by PSNR, which is

a QoS parameter (non-perceptual) PSNR evaluates no temporal qualities No real-time estimation of QoE

Full Reference (objective QoE) models compare stream with the original though not in real-time

AimsAims

Find QoE through subject testing Want to estimate QoE through estimation

equations, which can allow threshold values to be evaluated in real-time

See how accurate estimations are to the subjective measurements

Measure the percentage of the selected threshold by way of multiple regression lines

Find QoE through subject testing Want to estimate QoE through estimation

equations, which can allow threshold values to be evaluated in real-time

See how accurate estimations are to the subjective measurements

Measure the percentage of the selected threshold by way of multiple regression lines

OutlineOutline

IntroductionSCS theoryAims of studyQoE measurementQoE estimationThreshold selectionConclusions

IntroductionSCS theoryAims of studyQoE measurementQoE estimationThreshold selectionConclusions

SetupSetup 6 videos

Video dominant (sports) Audio dominant (music video) Lower frame rate (animation)

The second of each with greater Temporal perceptual Information (TI) value

Recorded streams output by the media recipient were used as stimuli (432 total): Six different levels of average web traffic (20, 30, 40, 50, 75, 100)

Lossy environments > 20 web client processes Four Th (100, 40, 20, 0) Three picture patterns (I, IPPPP, IPPPPPPPPPPPPPP)

6 videos Video dominant (sports) Audio dominant (music video) Lower frame rate (animation)

The second of each with greater Temporal perceptual Information (TI) value

Recorded streams output by the media recipient were used as stimuli (432 total): Six different levels of average web traffic (20, 30, 40, 50, 75, 100)

Lossy environments > 20 web client processes Four Th (100, 40, 20, 0) Three picture patterns (I, IPPPP, IPPPPPPPPPPPPPP)

SetupSetup

Based on an interval scale derived from:Rating scaleLaw of categorical judgment

Impairment rating-scale:5 Imperceptible4 Perceptible, but not annoying3 Slightly annoying2 Annoying1 Very annoying

Stimuli which gave large errors of Mosteller’s test are removed

Ensures goodness of fit (psychological scale)

Based on an interval scale derived from:Rating scaleLaw of categorical judgment

Impairment rating-scale:5 Imperceptible4 Perceptible, but not annoying3 Slightly annoying2 Annoying1 Very annoying

Stimuli which gave large errors of Mosteller’s test are removed

Ensures goodness of fit (psychological scale)

Sport 2 (I)Sport 2 (I)

Pure frame skipping achieves the highest QoE

All other contents performed similarly for I

Notice that no other frames are dropped with only (I)

Pure frame skipping achieves the highest QoE

All other contents performed similarly for I

Notice that no other frames are dropped with only (I)

Sport 2 (IPPPP)Sport 2 (IPPPP)

In almost all lossy environments, Th 20% & 40% provide higher QoE than 0%

In almost all lossy environments, Th 20% & 40% provide higher QoE than 0%

Animation 2 (IPPPP) | Music Video 1 (IPPPP) Animation 2 (IPPPP) | Music Video 1 (IPPPP)

Th 0% is better than nonzero values in lossy environments

Reason: Animation has less than 30 frames & the music video is audio dominant with low video motion

Th 0% is better than nonzero values in lossy environments

Reason: Animation has less than 30 frames & the music video is audio dominant with low video motion

Music Video 1 (IPPPPPPPPPPPPPP)Music Video 1 (IPPPPPPPPPPPPPP)

At 0%, Th is now less successful

Greater QoE at lossy envioronments with a higher Th, as frame skip loses all succeeding P frames

At 0%, Th is now less successful

Greater QoE at lossy envioronments with a higher Th, as frame skip loses all succeeding P frames

OutlineOutline

IntroductionSCS theoryAims of studyQoE measurementQoE estimationThreshold selectionConclusions

IntroductionSCS theoryAims of studyQoE measurementQoE estimationThreshold selectionConclusions

QoE EstimationQoE Estimation

Psychological scale is achieved by QoS mapping between the user-level and the application level

Mapping is accomplished via multiple regression analysis QoS parameters = independent Psychological scale = dependent

Psychological scale is achieved by QoS mapping between the user-level and the application level

Mapping is accomplished via multiple regression analysis QoS parameters = independent Psychological scale = dependent

Application level QoS parametersApplication level QoS parameters

These parameters represent both temporal and spatial quality

To best estimate QoE, the variables chosen chosen should have low cross-correlations

These parameters represent both temporal and spatial quality

To best estimate QoE, the variables chosen chosen should have low cross-correlations

QoE estimationQoE estimation

Principle component analysis allows us to find cross-correlations between introduced independent variables

Variables that correlate strongly (a cumulative contribution rate> 90%) are placed in one of the five classes (A-E)

Then we calculate a multiple regression line for every combination The line chosen is the one with the greatest

multiple correlation coefficient adjusted for degrees of freedom (R*)

Picture pattern is not taken into account One parameter from each class must be selected

Principle component analysis allows us to find cross-correlations between introduced independent variables

Variables that correlate strongly (a cumulative contribution rate> 90%) are placed in one of the five classes (A-E)

Then we calculate a multiple regression line for every combination The line chosen is the one with the greatest

multiple correlation coefficient adjusted for degrees of freedom (R*)

Picture pattern is not taken into account One parameter from each class must be selected

QoS parameter classes based on PCAQoS parameter classes based on PCA

Accuracy of estimationAccuracy of estimation

OutlineOutline

IntroductionSCS theoryAims of studyQoE measurementQoE estimationThreshold selectionConclusions

IntroductionSCS theoryAims of studyQoE measurementQoE estimationThreshold selectionConclusions

Setting the thresholdSetting the threshold

In order to maximize QoE, the appropriate threshold must be chosen

This is accomplished by implementing a “learning period”

Th is first set to 100%, while the formal Th is computed by estimating the psychological scale values for each threshold

At the end of the learning period, the Th with the max psychological value is chosen

If there are two Th’s with the same value, the larger Th is chosen

In order to maximize QoE, the appropriate threshold must be chosen

This is accomplished by implementing a “learning period”

Th is first set to 100%, while the formal Th is computed by estimating the psychological scale values for each threshold

At the end of the learning period, the Th with the max psychological value is chosen

If there are two Th’s with the same value, the larger Th is chosen

Sport 2 (I)Sport 2 (I)

Pure frame skipping is chosen in a lossy environment

This was also true in subjective tests

Pure frame skipping is chosen in a lossy environment

This was also true in subjective tests

Sport 2 (IPPPP)Sport 2 (IPPPP)

Music Video 1 (IPPPP)Music Video 1 (IPPPP)

Music Video 1 (IPPPPP…)Music Video 1 (IPPPPP…)

Limitations RealizedLimitations Realized

Content contains audio and video QoE is a perceptual QoS

Psychological scale uses QoS parameters

In QoE estimation, QoS parameters account for spatial and temporal characteristics

Estimation with learning period provide real-time QoE assessment

Content contains audio and video QoE is a perceptual QoS

Psychological scale uses QoS parameters

In QoE estimation, QoS parameters account for spatial and temporal characteristics

Estimation with learning period provide real-time QoE assessment

ConclusionsConclusions

The effectiveness of estimation and human subject testing for SCS’s was examined Subject testing of these estimated SCS should be done

With picture pattern’s I and IPPPP the measured and estimated QoE’s are quite similar to each other when utilizing nonlinear multiple regression analysis

Picture pattern I favored frame dropping, while IPPP… favored error concealment

Threshold value selection must be further investigated

The effects of motion copy should be used in future tests

Is there a need for QoE estimation?

The effectiveness of estimation and human subject testing for SCS’s was examined Subject testing of these estimated SCS should be done

With picture pattern’s I and IPPPP the measured and estimated QoE’s are quite similar to each other when utilizing nonlinear multiple regression analysis

Picture pattern I favored frame dropping, while IPPP… favored error concealment

Threshold value selection must be further investigated

The effects of motion copy should be used in future tests

Is there a need for QoE estimation?

Thank youThank you