Post on 21-Jul-2020
transcript
Extreme Experience Research Report
Extreme Experience Research Report Contents
Contents
1 Introduction.................................................................................................................................... 1
1.1 Key Findings .................................................................................................................................................... 1
2 Research Summary ....................................................................................................................... 2
2.1 Project Purpose and Contents ........................................................................................................................... 2
2.1.2 Theory Principle ...................................................................................................................................... 2
2.1.3 Research Architecture ............................................................................................................................. 3
2.1.4 Research Methodology ........................................................................................................................... 4
3 Research Result ............................................................................................................................. 5
3.1 Web Page Loading............................................................................................................................................ 5
3.1.1 Zero Waiting Time for Loading Texts and Images Is 270 ms ................................................................. 5
3.2 Video Streaming ............................................................................................................................................... 5
3.2.1 Zero Waiting Time for Video Streaming Is 80 ms ................................................................................... 5
3.2.2 Data Verification for Video Streaming Zero Waiting Time ..................................................................... 6
3.2.3 Zero Waiting Time for Video Asynchronization Is 220 ms ..................................................................... 7
3.2.4 The Extreme Experience of Definition Is Affected by Screen Sizes and Illumination Conditions ......... 7
3.3 Video Call ......................................................................................................................................................... 9
3.3.1 Zero Waiting Time for Interaction During Video Calls Is 210 ms .......................................................... 9
4 Appendix ...................................................................................................................................... 10
4.1 About This Report .......................................................................................................................................... 10
4.2 Contact Us ...................................................................................................................................................... 10
4.3 Disclaimer ...................................................................................................................................................... 10
Extreme Experience Research Report Figures
Figures Figure 2-1 Project purpose .................................................................................................................................... 2
Figure 2-2 Research principle ............................................................................................................................... 3
Figure 2-3 Research architecture ........................................................................................................................... 4
Figure 2-4 Zero waiting time for loading texts and images ................................................................................... 4
Figure 3-1 Zero waiting time for loading texts and images ................................................................................... 5
Figure 3-2 Zero waiting time for video streaming ................................................................................................. 6
Figure 3-3 Zero waiting time for video asynchronization ..................................................................................... 7
Figure 3-4 Zero waiting time for interaction during video calls ............................................................................ 9
Extreme Experience Research Report Tables
Tables
Table 3-1 Human visual perception ability ............................................................................................................ 6
Table 3-2 Materials used in the experiment of zero waiting time for video asynchronization .............................. 7
Table 3-3 Extreme experience of definition ........................................................................................................... 8
Table 3-4 Materials used in the experiment of the extreme experience of definition ............................................ 8
Extreme Experience Research Report 1 Introduction
1
1 Introduction
The experience baseline is obtained through the service tests based on the existing network.
Due to the constraints such as network environment and test methods, the test results
represent only the current status of the network and service deployment rather than the actual
experience of users.
From the perspective of human factors engineering, this research obtains the extreme
experience of human perception through professional data collection and analysis system
(data on video, audio, bioelectricity, eye movement, brain wave and pressure distribution).
This document provides guidelines for improvements on network products and solutions and
the theoretical basis for the ultimate goal of the future network development based on the data
concerning users' extreme experience.
In this report, "zero wait time" is used to describe the extreme experience when a user cannot
perceive the waiting time of the service.
1.1 Key Findings The zero waiting time for loading texts and images is 270 ms, and 80 ms for video
buffering. Users cannot perceive waiting time less than the preceding thresholds.
If audio and video are not synchronized for more than 220 ms, users will perceive the
asynchronization problem when they are making video calls or watching videos.
During a video call, if the interaction delay exceeds 210 ms, users will perceive the delay
in the conversation.
The extreme experience of video definition is affected by the screen sizes of display
devices and illumination conditions. For mainstream smart phones whose screen sizes
are between 4.3 and 6 inches, degradation of video definition is perceived when video bit
rates are ranging from 400kbit/s to 1.1 Mbit/s.
Extreme Experience Research Report 2 Research Summary
2
2 Research Summary
2.1 Project Purpose and Contents
Figure 2-1 Project purpose
Text
Refreshing
Unperceived Acceptable Unacceptable
Image
Refreshing
Extreme Experiment
on Texts
Verification Experiment
on Texts
Unendurable Experiment
on Texts
Voice
Distortion
Buffering
Continuity
Synchroni-
zation
Definition
Interaction
Webpage
Voice
Noise Level
Screen Size
Illuminance
Video
Streaming
Video Call
Extreme Experiment
on Images
Verification Experiment
on Images
Unendurable Experiment
on Images
Extreme Experiment
on Voice Distortion
Verification Experiment
on Voice Distortion
Unendurable Experiment
on Voice Distortion
Extreme Experiment
on Video Streaming
Verification Experiment
on Video Streaming
Unendurable Experiment
on Video Streaming
Extreme Experiment
on Video Continuity
Verification Experiment
on Video Continuity
Unendurable Experiment
on Video Continuity
Extreme Experiment
on Asynchronization
Verification Experiment
on Asychronization
Unendurable Experiment
on Asychronization
Extreme Experiment
on Video Definition
Verification Experiment
on Video Definition
Unendurable Experiment
on Video Definition
Extreme Experiment
on Interaction
Verification Experiment
on Interaction
Unendurable Experiment
on Interaction
The tests in this research focus on user experience of the current mainstream Internet services
(webpage, video, social networking, and VoIP). User experience can be divided mainly into
two categories: time-related (waiting time for text and image refreshing, video streaming, and
also during video calls the waiting time for attaining video continuity, seamless interaction, as
well as video and audio synchronization), and quality-related (image resolution and sound
clarity). Users' final experience is graded into three levels: unperceived, acceptable, and
unacceptable. In addition, factors such as noise level, illuminance and the screen sizes of the
test terminals are all considered in this research.
2.1.2 Theory Principle
For many years, the pupil reaction has been applied to visual studies. It has long been
believed that the diameter or the size of the pupil can be calculated by certain function using the total luminous flux that the eye has received. Recent studies show that the pupillary light
Extreme Experience Research Report 2 Research Summary
3
reflex system does not only respond to the illumination changes, but also to the spatial
distribution changes of photic stimulation. It has been proven that simple graphic changes can
trigger graphic-pupil reflex. Therefore, slight eye movements such as dilating pupils can be
used to detect photic stimulation changes in real time.
Figure 2-2 Research principle
Fluctuations in human perception can be manifested in the changes of human's pupil
diameters. This experiment captures users' extreme perceptions, such as zero-perception and
zero waiting time, by using the eye-tracking device to detect changes in their pupil sizes.
Compared with other physiological indicators, such as reaction time, galvanic skin response,
and sweat glands secretion, the measurement of pupil size is more accurate. With its error
range within 5 ms, this method can provide accurate data to measure extreme experience.
2.1.3 Research Architecture
Figure 2-3 shows the architecture of the research.
Extreme Experience Research Report 2 Research Summary
4
Figure 2-3 Research architecture
User Experience
Affect
Unperceived Acceptable Unacceptable
SatisfiedChanges in
moodsGive Up
Psychological LevelPhysiologica
l Changes
Psychological
Changes
Eye Tracking
RingBehavior Ring
Emotional
Changes
Changes of
Pupil Sizes
Ring
Method
Framework
Laboratory
Scenario ScenarioVerification
Experiment
Statistics
Statistics
Environment VariablesScreen Size/Noise/
Illuminance
Gender/Education/Age/
Income and etc.
Interviews Interviews Analyzing Root Causes
Reference Timer Reference
Eye TrackingEye treacking
Screen Size/Noise/
Illuminance
Changes of
Pupil Sizes
2.1.4 Research Methodology
To illustrate the research methodology, the extreme experiment of zero waiting time for
loading texts and images is used as an example. The procedures of the experiment are shown
in Figure 2-4. Within T time after pressing Enter (entering the URL address), the texts are
displayed on the page, while the images are still being loaded and will appear after another T
time. This process composes one segment of the experimental materials and a next segment of
the experimental material will be triggered by pressing Enter after the completion of former
segment. The length of the T time is gradually extended, ranging from 50 ms to 2000 ms.
Figure 2-4 Zero waiting time for loading texts and images
Preparing for the
experiment...
Texts are displayed on
the web browser.
Images are buffering
Both texts and
images are displayed
on the web browser
200msSTOP
Latency
Percived
T T……
Based on the original data recorded by an eye-tracking device, the corresponding T time
during which eye movements abruptly change, indicating the occurrence of extreme
experience, can be identified. The accuracy of the result can be improved by narrowing the
range of T time through multiple phases of experiment.
Indicators of the extreme experience of other services can be obtained using a similar
experimental procedure.
Extreme Experience Research Report 3 Research Result
5
3 Research Result
3.1 Web Page Loading
3.1.1 Zero Waiting Time for Loading Texts and Images Is 270 ms
This experiment studies users' perception of the waiting time for loading texts and images. If
the waiting time is below a threshold, users cannot perceive the waiting time for loading texts
and images. The threshold is termed zero waiting time.
Figure 3-1 Zero waiting time for loading texts and images
The zero waiting time is gradually narrowed down to an accurate range through three phases of
experiments. In the first phase of the experiment, the zero waiting time lies between 200–800
ms. In the second phase, it lies between 250–350 ms. In the third phase, it lies between
250–300 ms. The range of zero waiting time for loading texts and images is finalized between
250–300 ms, with its mean value being 270 ms.
3.2 Video Streaming
3.2.1 Zero Waiting Time for Video Streaming Is 80 ms
This experiment studies users' perception of the waiting time of video buffering. When the
video buffering time is lower than a threshold, users cannot perceive the waiting time for
video streaming. This experience is called zero waiting.
Extreme Experience Research Report 3 Research Result
6
Figure 3-2 Zero waiting time for video streaming
In the first phase of the experiment, the zero waiting time lies between 50–200 ms. In the
second phase, it lies between 40–160 ms (The experiment is practiced at an increment of 10
ms in the second phase). The mean value of 18 participants is 70 ms.
Since the zero waiting time for video streaming, which is 70 ms, differs significantly from
that for the texts and images, which is 270 ms, more participants are involved in the third
phase of the experiment to test the validity of the data. Eight more participants are added to
the third phase. After excluding an outlier (280 ms), the time range is finalized between
50–100 ms. Therefore, the validity of the preceding data is preliminarily verified.
Based on the results obtained in the second phase, it is concluded that the zero waiting time of
video streaming is 80 ms.
3.2.2 Data Verification for Video Streaming Zero Waiting Time
The zero waiting time of video streaming is 80 ms (ranging from 40–120 ms). Considering
that it differs significantly from that of the texts and images loading (270 ms) and that of
video calls (220 ms), the validity of the data needs to be verified.
1. Repeat the preceding experiment. Eight more participants are added to the renewed
experiment. After excluding an outlier (280 ms), the time range is finalized between 50–100
ms. Therefore, the validity of the preceding data is preliminarily verified.
2. Statistics Verification. The zero waiting time of video streaming is closely related to that
of video continuity, which is 17 frames, equaling 58–59 ms and falling into the time range of
the former data.
3. Theoretical Verification. According to the Partrick's speech at the ACM in 1997, humans'
cognitive speed at perceiving videos is 3 to 6 times faster than that at perceiving static images.
His theory can be applied to explain the difference between the zero waiting time of video
streaming and that of texts and images loading.
Table 3-1 Human visual perception ability
Objects of Visual Perception Extreme Perception (Second)
Texts 0.812
Images 0.65
Animated graphics 0.21
Videos 0.23
Source: Partrick L. & Xin F. Motion Effects towards Cognitive Speed, ACM-Graphics Open Lecture, U.S., 1997
Extreme Experience Research Report 3 Research Result
7
3.2.3 Zero Waiting Time for Video Asynchronization Is 220 ms
This experiment studies users' perception of asynchronization between sound and images
during video streaming and video calls. When the asynchronization between sound and
images exceeds certain thresholds, users would perceive that the sound and images are not
synchronized.
Figure 3-3 Zero waiting time for video asynchronization
The parameters of the video materials used in this experiment are listed in the table. In the
first phase of the experiment, the maximum time for a participant among the 11 participants to
perceive asynchronization is 1000 ms, and the minimum time is 50 ms. To narrow the range
and improve accuracy, the second phase of the experiment is conducted. Based on the
narrowed range obtained from the first phase, the extreme value of zero waiting time of the 16
participants is obtained. Based on the 3SD principle and comparison of adjacent points, it is
concluded that there is no outlier among the obtained data and the calculated mean value is
220 ms.
Table 3-2 Materials used in the experiment of zero waiting time for video asynchronization
Asyn. (ms)
50 100 150 200 250 300 350 400 500 600 700 800 900 1000 1500 2000
3.2.4 The Extreme Experience of Definition Is Affected by Screen Sizes and Illumination Conditions
This experiment studies users' perception of the video definition during video streaming and
video calls. It aims to identify the threshold of the definition where users start to perceive the
degradation of video definition.
The data of extreme experience of definition in relationship to screen sizes and illumination
conditions are listed in Table 3-3.
Extreme Experience Research Report 3 Research Result
8
Table 3-3 Extreme experience of definition
Screen
SizeEnviron.
3.5 4.3 5 6 7 7.9 9.7 14
Outdoor
Office
Night
240p_low 240p_low
240p_low
240p_low
240p_high
360p_low
360p_low 360p_low
360p_low
360p_low 360p_low 360p_low 360p_low
360p_high
480p
480p
576p
576p 576p 576p
576p720p_low
720p_high
720p_high
In terms of the screen sizes, the larger the screen size is, the higher the video bit rate will be
when users perceive the degradation of video definition. For a 3.5-inch screen, participants
perceive the degradation of video definition when the definition of the video clip is 240p_low.
For mainstream smart phones whose screen sizes are between 4.3 and 6 inches, degradation of
video definition is perceived mainly when the definitions of the video clips are between 360p
and 480p, with video bit rates ranging from 400 kbit/s to 1.1 Mbit/s.
In terms of environmental conditions, as the illumination decreases, participants will perceive
the degradation of video definition.
Note 1: The parameters of the video materials used in this experiment are listed in Table
3-4. Since the definition of a video clip is closely related to its genre, video codec, and
video bit rate, the quantitative conclusions drawn in this test have their own limitations.
Note 2: In this experiment, the illumination environment is simulated in a laboratory and
is set to three typical lighting conditions (outdoor 1000Lx, office 300-400Lx, at night
1-2lx) which are measured by the illuminometer.
Table 3-4 Materials used in the experiment of the extreme experience of definition
Experiment Material
Video Definition
Video Bit Rate (kbit/s)
Video Codec (H.264)
240p_low 426*240 250 Main@2.1
240p_high 426*240 400 High@3.0
360p_low 640*360 400 Main@2.1
360p_high 640*360 750 High@3.0
480p 854*480 1100 High@3.1
576p 1024*576 1600 High@3.1
720p_low 1280*720 1100 High@3.1
720p_high 1280*720 2400 High@3.1
1080p_low 1920*1080 2200 High@4.0
1080p_high 1920*1080 5000 High@4.0
Extreme Experience Research Report 3 Research Result
9
3.3 Video Call
3.3.1 Zero Waiting Time for Interaction During Video Calls Is 210 ms
This experiment studies the zero waiting time of interaction during video calls. If the waiting
time of interaction is below certain threshold, users cannot perceive any delay during image
transmission.
Figure 3-4 Zero waiting time for interaction during video calls
The zero waiting time is gradually narrowed down to an accurate range through three phases of
experiments. In the first phase of the experiment, the zero waiting time lies between 100–800
ms. In the second phase, it lies between 150–300 ms. In the third phase, the range of zero
waiting time is between 170–260 ms, with its mean value being 210 ms.
Extreme Experience Research Report 4 Appendix
10
4 Appendix
4.1 About This Report
This research is the first of its kind in applying human factors engineering to study user
experience. This research may be incomplete due to limited participants and experimental
materials. The results of the experiments are for reference only. More research concerning the
key indicators will be conducted in the future. Due to the length of this report, only some of
the important conclusions are listed. You can contact us to obtain the complete report and
experiment details.
To create a baseline standard for user experience regardless of RAT, you can refer to the MOS
for voice quality to establish MOS systems for web pages, video streaming and video calls. In
these MOS systems, the scores will be based only on human perceptions, in which 5 equals
zero waiting time and 1 equals the maximum level of endurance. Theoretically, according to
the network conditions, there will be certain maximum values of zero waiting time for
different RATs. Achieving the maximum value can serve as the goal of the development of
each RAT.
4.2 Contact Us
Author: Wang Bin, Email: i.wangbin@huawei.com
Contact mLAB (MBB lab): MBBlab@huawei.com
4.3 Disclaimer
This report is a product of Huawei mLAB. The information provided in this report is for
reference only. This research may be incomplete due to limited participants and experimental
materials. The rights of revision regarding the content of this report are reserved to Huawei.
Huawei holds no responsibility for the consequences of revision.
Information contained in this report, express or implied, constitutes neither any basis for investment purposes nor a warranty of any kind. mLAB may add to, correct and amend
Extreme Experience Research Report 4 Appendix
11
information in this report without notice, but does not guarantee immediate release of the
revised version. Huawei will not be liable for any direct or indirect investment profit or loss
caused thereby.
This document is an intellectual property of Huawei mLAB. No part of this document may be
reproduced or transmitted in any form or by any means without prior written consent of
Huawei. If any content of this report is released by any other party in the form of reference, it
should be noted as the intellectual property of Huawei mLAB. There shall be no citation,
deletion and modification that violate the original meaning of this report.