Date post: | 13-Jan-2017 |
Category: |
Science |
Upload: | tobias-hossfeld |
View: | 481 times |
Download: | 1 times |
Prof. Dr. Tobias Hoßfeld
Chair of Modeling of Adaptive Systems (MAS)Institute for Computer Science and Business Information Systems (ICB)University of Duisburg-Essen
www.mas.wiwi.uni-due.de
QoE++: Shifting from Ego- to Eco-System?
IFIP/IEEE QCMan 2015Ottawa, 11 May 2015
1. Current Status: Managing the QoE Ego-System
2. Some Observations on QoE
3. QoE++: The QoE Eco-System
Interest in QoE over the last 10 years
• Number of publications per year when searching for „QoE“
• Academic interest is increasing! Industrial Employment?
mas.wiwi.uni-due.de 3
200420052006200720082009201020112012201320140
50
100
150
200
250
300
350
400
450
2 5 8 1853
89
149
194
298 286
393
IEEE Xplore (Metadata)
Year
#Pub
licati
ons /
Yea
r
20042005
20062007
20082009
20102011
20122013
20140
500
1000
1500
2000
2500
3000
3500
4000
547 513709 709
11001370
1700
2190
2730
34103570
Google Scholar
Year
#Pub
licati
ons /
Yea
r
Research Communities
mas.wiwi.uni-due.de 4
0 1 2 3 4 5 61
2
3
4
5
number of stallings
MO
S
crowdsourcinglaboratoryQoE
MultimediaEncoder
Decoder
0 20 40 60 80 1000
5
10
15
20
25
30
ratio of buffering events
play
time
(min
)
Engagement
Application Control Plane Application
ControllerNetwork Control Plane
Data Plane
ApplicationNetworking
Current Status: QoE Management
mas.wiwi.uni-due.de 5
• Application level, end user site
• Within network, …
• Cross-layer approaches• Realization, e.g. SDN, …
• Parametric models
• Machine learning• …
• Subjective & objective tests • Crowdsourcing• …
Key Influence Factors
QoE Model
QoE Monitor-
ing
QoE Manage-
ment
Concept of QoE Management
Cloud / DC
Access Network
Core Network
Access Network
Cloud service providerEnd user
Cloud / DC
QoE Management requires1. QoE Model2. QoE Monitoring3. QoE Control
Network provider
mas.wiwi.uni-due.de 6
The QoE Ego-System
• Main focus– in-session– short-time scale– single user QoE– single apps– user perspective
• Typical research questions– What are the key QoE influence factors?
– How and where to monitor QoE and its influence factors?
– How to deliver contents and control traffic management?
– How to adapt contents and media to current network situation?
– How to exchange information between network and application to overcome QoE issues?
mas.wiwi.uni-due.de 7
SOME OBSERVATIONS
QoE Models: Complexity and Generic Relationships
mas.wiwi.uni-due.de 9
• Model is intended to fulfill acertain goal
$$$
• Generic relationships need to be considered, e.g. IQX
Subjective Testing
• Subjective Experiments– Quantifying QoE of improved system– Challenging: proper test design,
implementation, analysis– Limited by pool of test subjects
• Crowdsourcing– Access to large pool of humans– Challenging: remote conduction
of tests, statistical analysis
mas.wiwi.uni-due.de 10
What is ?
Crowdsourced QoE: Best Practices
Conceptual aspects
Hoßfeld, T., Keimel, C., Hirth, M., Gardlo, B., Habigt, J., Diepold, K., & Tran-Gia, P. (2014). Best practices for QoE crowdtesting: QoE assessment with crowdsourcing. Multimedia, IEEE Transactions on, 16(2), 541-558.
Pseudo reliable crowd
Lab Tester
Filtering- Demographics- Hardware requirements- Reliability- …
Training
Phase 1
QoE - Test - Software based screening mechanisms
- Content questions, reliability checks
- Incentive design, variable payments
- …
Postprocessing
Phase 2
- Statistical analysis- …
Practical aspects
Tobias Hoßfeld, Matthias Hirth, Judith Redi, Filippo Mazza, Pavel Korshunov, et al.. Best Practices and Recommendations for Crowdsourced QoE - Lessons learned from the Qualinet Task Force "Crowdsourcing, 2014. https://hal.archives-ouvertes.fr/hal-01078761/
mas.wiwi.uni-due.de 11
Do we need QoE?
Can we utilize QoE for network & service management? Is it more appropriate to consider other means?
Measurement Studies for HTTP Video Streaming
mas.wiwi.uni-due.de 13
0 1 2 3 4 5 61
2
3
4
5
number of stallings
MO
S
crowdsourcinglaboratory
QoE
0 20 40 60 80 1000
5
10
15
20
25
30
ratio of buffering events
play
time
(min
)
Engagement
Engagement data: Dobrian, F., Sekar, V., Awan, A., Stoica, I., Joseph, D., Ganjam, A., Zhan, J. & Zhang, H. (2011). Understanding the impact of video quality on user engagement. ACM SIGCOMM Computer Communication Review, 41(4), 362-373.
System Model
QoE data: Hoßfeld, T., Schatz, R., Biersack, E., & Plissonneau, L. (2013). Internet video delivery in YouTube: from traffic measurements to quality of experience. InData Traffic Monitoring and Analysis (pp. 264-301). Springer Berlin Heidelberg.
Output: stalling pattern
Input: network and video
characteristics
User Behavior and QoE
• Example: QoE and User Engagement in HTTP Video Streaming
• Different video bufferdurations investigated
• Stakeholderinterested in watch time, e.g. sellingadvertisements
• Strong relationship,but complementary approach
mas.wiwi.uni-due.de 14
What are proper QoE models?
How can we extend existing QoE models to take into account the service provider's perspective, individual user perceptions?
Beyond Mean Opinion Scores (MOS)
• MOS is one measure for QoE!
• Confidence intervals show statistical significance, but not reliability!
• Reliability metrics quantify how reliable your data is.
• Standard deviation quantifies the user diversity.
• Quantiles are of interest for service providers.
mas.wiwi.uni-due.de 16
Excellent!
Bad!
Fair!Good!
Poor!
Æ
Fair = 3
Limitations of MOS
• Results from subjective experiments on video QoE
• Service providersdefines a threshold of acceptable quality
• Probability ofdissatisfied users: .
• But: Service providerwants to satisfy majority of users e.g. quantiles
mas.wiwi.uni-due.de 17
Individual QoE Profiles per User?
mas.wiwi.uni-due.de 18
QoE Model for MOSSystem Model
Do we need user profiles?Do we need usage scenarios?
Parameterization of QoE
Impact of user profile
Consequences for QoE Management
Parameterized wrt. user profile Impact of buffer size,
video bitrate, network conditions
Talk later by Christian Moldovan:
To Each According to his Needs: Dimensioning Video Buffer for Specific User Profiles and Behaviorby C. Moldovan, C. Schwartz, T. Hossfeld
Users more or less sensitive to delays and stalling
Is context more important than QoE?
Which context factors are relevant or are such context-factors even more important for network & service management, e.g. in order to foresee and react on flash crowds?
Example: HTTP Adaptive Streaming with Context
• Use context and context predictors in adaptive streaming strategies• Predict bandwidth and buffer state based on location, connectivity state
(3G, WiFi, upcoming vertical/horizontal handovers), social (e.g. flash crowds), mobility (tunnel)
• Include context information– for buffering and quality
level selection strategy
– for caching decisions
mas.wiwi.uni-due.de 20
User performs QoE management?!
QOE++: THE QOE ECO-SYSTEM
Transition to QoE Eco-System
• QoE eco-system– in-session vs. global– short- vs. long-time scale– single vs. multi-user QoE– single vs. concurrent apps– user vs. business perspective– all key stakeholder goals
• Requirements– Extend current QoE models by the
different stakeholder perspectives of the QoE eco-system– Incorporate user behavior as part of the model– Identify and include relevant internal and external context factors
including physical, cultural, social, economic context.
Content ProviderISPs
CDNs
$$$
$$$ $$$
$$$
AdsData
analysis…
mas.wiwi.uni-due.de 22
Comprehensive Framework: QoE and User Behavior
mas.wiwi.uni-due.de 23
Reichl, P.; Egger, S.; Möller, S.; Kilkki, K.; Fiedler, M.; Hossfeld, T.; Tsiaras, C.; Asrese, A.: Towards a comprehensive framework for QoE and user behavior modelling. QoMEX 2015
An abstract view
mas.wiwi.uni-due.de 24
Quality of Experience Network Layers
ManagementApplication /
Service
Network
QoE++
Technical realization, e.g. SDNMonitoring
Model
Cross-layer approach, interaction
of control loops,economic
traffic management
Viewpoint
Top down: theoretical framework
Met
hodo
logy
Bottom up: use-case & technologydriven
Intermediate players, e.g.
cloud
……
QoE++ Research Directions
• Can we utilize QoE for network & service management? – User engagement and user behavior– Context factors
• How to realize QoE management?– Cross-layer optimization: application demands vs. network capabilities– SDN as technology path
• Can we transform QoE into business models, SLAs, etc.? – Or is it possible to 'trade' QoE? For example, offering WiFi sharing at home, a
user may get improved service delivery and QoE by its ISP.
• Do we understand QoE as well as fundamental models and natural relationships? – Extend existing QoE models – Relationship between QoE and user behavior?
• Theoretical user-centric performance evaluation approaches
mas.wiwi.uni-due.de 25
THANKS
Additional Pointers (and references therein…) for HTTP Streaming QoE
Overview on HTTP Adaptive Streaming and HAS QoE. Seufert, M.; Egger, S.; Slanina, M.; Zinner, T.; Ho0feld, T.; Tran-Gia, P., "A Survey on Quality of Experience of HTTP Adaptive Streaming," Communications Surveys & Tutorials, IEEE , vol.17, no.1, pp.469,492, 2015doi: 10.1109/COMST.2014.2360940HTTP Streaming QoE Model: Total Stalling, Stalling frequency. Hoßfeld, T., Schatz, R., Biersack, E., & Plissonneau, L. (2013). Internet video delivery in YouTube: from traffic measurements to quality of experience. In Data Traffic Monitoring and Analysis (pp. 264-301). Springer Berlin Heidelberg.
HTTP Streaming model: initial delay, : Total Stalling, Stalling frequency. Tobias Hoßfeld, Christian Moldovan, Christian Schwartz: To Each According to his Needs: Dimensioning Video Buffer for Specific User Profiles and Behavior. In: QCMAN 2015. Ottawa, Canada 2015.
Time on high layer in HAS: Subjective Study. Hoßfeld, T., Seufert, M., Sieber, C., & Zinner, T. (2014). Assessing Effect Sizes of Influence Factors Towards a QoE Model for HTTP Adaptive Streaming. In Proceedings of the 6th International Workshop on Quality of Multimedia Experience (QoMEX 2014), Singapore.
HTTP Adaptive Streaming model: Total Stalling, Stalling frequency and quality adaptation. Hossfeld, Tobias; Skorin-Kapov, Lea; Haddad, Yoram; Pocta, Peter; Siris, Vasilios A. ;Zgank, Andrej; Melvin, Hugh;: Can context monitoring improve QoE? A case study of video flash crowds in the Internet of Services. In: QCMAN 2015 - Third IFIP/IEEE International Workshop on Quality of Experience Centric Management. Ottawa, Canada 2015.
Concrete HAS Implementation. Sieber, C.; Hossfeld, T.; Zinner, T.; Tran-Gia, P.; Timmerer, C., "Implementation and user-centric comparison of a novel adaptation logic for DASH with SVC," Integrated Network Management (IM 2013), 2013 IFIP/IEEE International Symposium on , vol., no., pp.1318,1323, 27-31 May 2013
Benchmarking Framework: Optimial HAS QoE. Hoßfeld, T., Seufert, M., Sieber, C., Zinner, T., & Tran-Gia, P. (2015). Identifying QoE optimal adaptation of HTTP adaptive streaming based on subjective studies. Computer Networks, 81, 320-332.
mas.wiwi.uni-due.de 27
Literature References from the Keynote
Conceptual aspects: Crowdsourced QoE. Hoßfeld, T., Keimel, C., Hirth, M., Gardlo, B., Habigt, J., Diepold, K., & Tran-Gia, P. (2014). Best practices for QoE crowdtesting: QoE assessment with crowdsourcing. Multimedia, IEEE Transactions on, 16(2), 541-558.
Practical aspects: Crowdsourced QoE. Tobias Hoßfeld, Matthias Hirth, Judith Redi, Filippo Mazza, Pavel Korshunov, et al.. Best Practices and Recommendations for Crowdsourced QoE - Lessons learned from the Qualinet Task Force "Crowdsourcing, 2014. https://hal.archives-ouvertes.fr/hal-01078761/
HTTP Streaming model: Total Stalling, Stalling frequency. Hoßfeld, T., Schatz, R., Biersack, E., & Plissonneau, L. (2013). Internet video delivery in YouTube: from traffic measurements to quality of experience. InData Traffic Monitoring and Analysis (pp. 264-301). Springer Berlin Heidelberg.
Beyond MOS: Quantiles and SOS for Service Providers. Hoßfeld, Tobias; Heegard, Poul; Varela, Martin: QoE beyond the MOS: Added Value Using Quantiles and Distributions. QoMEX 2015, Costa Navarino, Greece 2015.
QoE and User Behavior Model - Conceptual approach. Reichl, Peter; Egger, Sebastian; Möller, Sebastian; Kilkki, Kalevi; Fiedler, Markus; Hossfeld, Tobias; Tsiaras, Christos;Asrese, Alemnew: Towards a comprehensive framework for QoE and user behavior modelling. In: QoMEX 2015. Costa Navarino, Greece 2015.
User profiles and QoE / HTTP Streaming model for initial delay and stalling. Tobias Hoßfeld, Christian Moldovan, Christian Schwartz: To Each According to his Needs: Dimensioning Video Buffer for Specific User Profiles and Behavior. In: QCMAN 2015. Ottawa, Canada 2015.
mas.wiwi.uni-due.de 28