www.giraffic.com

©2016 Giraffic Technologies Ltd. All rights reserved

AVA™ Adaptive Streaming Performance Report Q3 2016

Giraffic Insights

Giraffic Insights- AVA Adaptive Streaming Performance Report Q3 2016

2

Table of Contents

1. Abstract ............................................................................................................................................................................................. 3

2. Introduction ...................................................................................................................................................................................... 3

3. Adaptive Video Acceleration™ (AVA™) ........................................................................................................................................... 3

4. Research Goal ................................................................................................................................................................................... 4

5. Methodology .................................................................................................................................................................................... 4

6. Results and Discussion ..................................................................................................................................................................... 5

7. Conclusions ....................................................................................................................................................................................... 8

Giraffic Insights- AVA Adaptive Streaming Performance Report Q3 2016

3

1. Abstract

Consumption of Over-the-Top content (OTT) is growing and maturing. As a result, consumer sensitivity to buffering, streaming resolution, and quality of service is increasing. This paper presents mass scale production data from Giraffic’s Adaptive Video Acceleration™ (AVA™) install base on over 50 million devices worldwide, measuring the current consumers’ video streaming experience and the impact of the AVA on user experience. For this analysis, a global sampling of over 300,000 adaptive sessions with duration longer than 30 seconds have been measured with AVA enabled and disabled. Results have shown:

(1) Quality of Service: AVA’s impact on Number of Quality Switches per Minute shows 35% and 50% reduction of

switch rate session for SD content and HD content, respectively.

(2) Quality - Streaming Resolution: Over 30% improvement of the delivered quality (bitrate) in high bitrate content.

● We couldn’t identify a significant improvement in streamed bitrate for content that is available only in SD

quality

(3) Buffering: Although theoretically, buffering should not occur at all with adaptive streaming, we find that 10% of

consumers experience buffering occurrences even with adaptive protocols: HLS, SmoothStreaming and MPEG-

DASH. AVA was able to reduce 18% of buffering events for both SD and HD.

2. Introduction

Over-the-Top video streaming has been improving over the years. Today’s consumers have access to a wider availability of premium and high quality content and a surge in high quality devices with displays that can support full HD and 4K that allow more and more service providers to stream content directly to consumers. However, due to internet bandwidth limitations they still aren’t able to tap into the full potential of these content services on their device in the same manner they can enjoy a broadcast TV experience. Service providers and operators are interested in offering high-quality technology solutions to reduce buffering and mediocre quality service, and enable their users to enjoy content at an equal or better quality as their broadcast experiences and to ensure as many users as possible can download and stream their content. Prior studies have presented several barriers to adoption including tech/network and UX of the system as well as discovery and authentication, and indicated that over 45% of barriers to adoption occur due to factors such as video buffering and freezing. Common solutions available in the market that aim to address these challenges are CDN, compression, and adaptive streaming. Each of the methods addresses different parts of the barriers, however none of them provide a complete solution. For example, while CDN increases the accessibility to content via caching, it doesn’t address congestion that happens in the last mile or in the home environment.

3. Adaptive Video Acceleration™ (AVA™)

Giraffic is the inventor of Adaptive Video Acceleration™ (AVA™), a client-side network throughput optimization and video-playback shaping technology that enables consumer electronic (CE) devices and content providers to deliver High Definition, 4K and VR video without re-buffering pauses or reducing streaming resolution. With over 50 million activated devices by leading vendors such as Samsung and LG in over 150 countries, Giraffic’s AVA improves billions of video streaming experiences in a completely client-side and complementary approach to existing video delivery infrastructure. AVA resides on the device’s operating system as an HTTP proxy running as a system service or within an app as a software development kit (SDK), improving video experience by bridging the video player needs with network conditions. The AVA core technology building blocks are:

1) Real Time Analytics: Ongoing diagnostics of the network condition and quality.

2) HTTP Acceleration: Optimization and enhancement of the download throughput.

3) Playback Shaping: Robust adaptation of streaming protocols to video player, device footprint and the content

quality requirements.

Giraffic Insights- AVA Adaptive Streaming Performance Report Q3 2016

4

4. Research Goal

The current research studies the impact of AVA on the user experience in a production environment on mass scale.

5. Methodology

During Q3 2016, Giraffic conducted an experiment to test the AVA performances. We ran A/B testing on over 300,000 adaptive bitrate streaming sessions playing in approximately 2,000 randomly selected devices in real consumers’ homes while AVA is enabled and disabled. To avoid any significant impact on end users, only a subset of 2-4% of sessions were AVA disabled. As part the sampling strategy, and due to the imbalance between AVA enabled and AVA disabled, we randomly selected 5,000 streamed sessions with AVA enabled and disabled out of the 300,000 sessions across the globe. Since we did not have an equal number of samples, prior to evaluating the results we had to make sure that two populations (AVA enabled, and AVA disabled) weren’t coming from the same population. To ensure this, we ran Wilcoxon Rank-Sum for non-identical populations (do not distribute normally, nor parametric distribution). We got a p-value (<0.1%), that led to the claim that the two populations aren’t identical. Later on, due the nature of our samples and since the AVA enabled population is over 10 times larger than the AVA disabled population, we had to ensure we did not get “long tail” effects. We did this by randomly selected an equal number of samples from the AVA enabled population, yet presenting the same difference from the disabled population. The content type was progressive download, and adaptive bitrates: HLS, MPEG-DASH and SmoothStreaming. We cut out sessions shorter than 30 seconds to avoid advertisements and initiation abnormalities. We classified the samples into two groups of Max Bitrates Qualities: 0-4 Mbps and 4+ Mbps. A few samples of countries and their results measured:

Country Average Internet Bitrate Number of Unique

Sessions with AVA

Number of Unique Sessions without

AVA

Turkey 2 Mb/s 43,152 1,261

Germany 10 Mb/s 288,596 7,011

Austria 10 Mb/s 11,986 157

Brazil 4 Mb/s 3,627 157

Total 347,361 8,586

Table 1. Average Internet Bitrate and Number of Sessions per Country

Three main parameters were measured in order to evaluate user experience and the impact of AVA per session: ● Bitrate Ratio: Average Streamed Bitrate vs Max Bitrate

● Switches per Minute: Number of Switches per Minute

● Buffering Rate: Downloading Time vs. Streaming Time

We saved the relevant information into the device. Then, to avoid network overload, we uploaded <0.5 MB to Giraffic secured serves every 7 days.

Giraffic Insights- AVA Adaptive Streaming Performance Report Q3 2016

5

Configuration 1. System Architecture

End user devices: Smart TVs with AVA technology implemented. While some smart TVs have been with AVA enabled,

a few were with AVA disabled. The Smart-TVs were uploading the following parameters: Average Bitrate per Session,

Highest Bitrate, Subscribed Bitrate, File Type, Number of Switches, while dimensions were Time and Date of Session

and Country.

Server: Giraffic collects data from clients to servers on Amazon Cloud, pulls out the information from the database,

sends requested queries to the webserver and delivers data to the web-browser.

Database: The database stores sent data in Amazon Cloud as well as extracts stored information per request.

Dashboard Site Web Browser: The dashboard allows presenting data based on a set of dimensions.

6. Results and Discussion

After proving the two populations did not originate from the same population we were able to evaluate the contribution of AVA to the experience. Since the distribution function of the two populations isn’t symmetric, comparing means would not be meaningful. Therefore, we measured the percentiles for each of the parameters ranging from 1%-100%. In graph 1 we present a distribution of the highest played bitrate out of maximum available bitrate of the video, per session.

Graph 1. Bitrate Quality Ratio vs. Percentile of Sessions

0

20

40

60

80

100

0% 10% 20% 30% 40% 50% 60% 70% 80%

Bit

rate

Rat

io %

Percentile of Sessions

Percentile of Maximum Streamed Bitrate Ratio

AVA EnabledAVA Disabled

Giraffic Insights- AVA Adaptive Streaming Performance Report Q3 2016

6

From the results we can see how sessions with AVA enabled provided lower number of sessions in comparison to sessions with AVA Disabled. For example, while with AVA enabled, 7% of sessions played up to 40% of the subscribed bitrate. With AVA Disabled the number rose to 15%, hence a reduction of 50% was shown by enabling AVA. Graph 2 presents the guaranteed bitrate played per session, which is the lowest streamed bitrate over a streamed video.

Graph 2. Percentile of Guaranteed Bitrate

From Graph 2 we see that AVA “pushes” the cumulative distribution towards high bitrates (hence, more sessions are playing in high bitrate). For example, with AVA enabled more than 90% of the sessions play bitrate that is higher than 80% of the subscribed bitrate. In case AVA is disabled it reduces to 75%. So, the higher the quality of the video, the greater are the network challenges, and hence the impact of AVA. Table (2) presents the percentage of sessions having buffering events.

Quality Buffering Improvement Level

AVA Disabled AVA Enabled

HD 7% 6% 16%

SD 12% 10% 20%

Table 2. Percentage of Sessions with Buffering

In comparison to SD sessions, HD sessions have overall higher probability for buffering events. For HD, a significant improvement has been demonstrated.

Graph 3. Percentage of Buffering Events

0

20

40

60

80

100

50% 60% 70% 80% 90% 100%

Gu

ran

tee

d B

itra

te o

ut

of

Max

imu

m B

itra

te

Percentile of Sessions

Percentile of Guranteed Streamed Bitrate

AVA Enabled

AVA Disabled

0%

5%

10%

15%

20%

25%

0%

5%

10%

15%

HD SD

Bu

ffe

rin

g Ev

en

ts

Type

Percentage of Buffering Events

AVA Disabled

AVA Enabled

Improvement

Giraffic Insights- AVA Adaptive Streaming Performance Report Q3 2016

7

The following tables present the percentiles of sessions achieved for each of the parameters formerly discussed. In tables 3-5, percentiles of the various values are presented, and distinguished between AVA enabled and AVA disabled results. Table (3) presents the percentile of sessions (AVA enabled and disabled) vs maximum streamed bitrate ratio. Reduction in the percentile indicates how many streams were saved.

HD Subscription

Maximum Bitrate Ratio AVA Disabled AVA Enabled Reduction Level

10% 3% 1% 66%

15% 5% 1.5% 75%

20% 8% 3% 58%

25% 8% 4% 50%

30% 9% 5% 40%

SD Subscription

Maximum Bitrate Ratio AVA Disabled AVA Enabled Reduction Level

10% 0% 0% 0%

15% 0.1% 0% 0%

20% 0.1% 0.1% 0%

25% 0.1% 0.1% 0%

30% 0.3% 0.1% 30%

Table 3. Percentiles vs. Maximum Bitrate Ratio: Saving bad streams Low bitrate ratio may indicate a poor internet connection. While the contribution of AVA to save “almost lost” sessions in SD is negligible, for HD qualities, AVA reduces 60% sessions with low bitrate ratio. Table (4) presents the percentile of sessions (AVA enabled and disabled) vs guaranteed bitrate ratio. Growth of percentile per bitrate ratio indicates larger number of sessions having high quality streaming.

HD Subscription

Guaranteed Bitrate-Ratio AVA Disabled AVA Enabled Improvement Level

60% 75% 83% 10%

70% 69% 81% 15%

80% 65% 80% 19%

90% 61% 76% 20%

SD Subscription

Guaranteed Bitrate Ratio AVA Disabled AVA Enabled Improvement Level

60% 92% 93% 1%

70% 90% 92% 2%

80% 88% 92% 5%

90% 85% 91% 7%

Table 4. Percentiles vs. Guaranteed Bitrate Ratio Sessions with SD subscribed quality are more likely to be improved. In particular, as there is more demand for guaranteed “premium” quality, AVA is increasing the percentile of sessions with high bit-rate ratio.

Giraffic Insights- AVA Adaptive Streaming Performance Report Q3 2016

8

Table (5) presents the percentile of sessions (AVA enabled and AVA disabled) vs number of switches per min.

HD Subscription

Number of Switches/min AVA Disabled AVA Enabled Improvement Level

1 33% 17% 48%

2 22% 13% 41%

3 16% 11% 31%

4 11% 8% 27%

SD Subscription

Number of Switches/min AVA Disabled AVA Enabled Improvement Level

1 13% 5% 62%

2 8% 3% 63%

3 3% 2% 33%

4 3% 1% 67%

Table 5. Percentile of Sessions vs Number of Switches In both cases, AVA reduces tremendously the percentile of sessions with high bitrate switches (per min). We suspect that in SD subscription we get higher improvement, since streams in these qualities are being streamed in low quality network environments.

7. Conclusions

We’ve been able to measure the current state of streaming on our install base, as well as the impact of Adaptive Video Acceleration (AVA) on the user experience. For both low-quality bitrate (below 4 Mbps) as well as for a high-quality bitrate (above 4 Mbps), we were able to demonstrate a consistent improvement for samples with AVA enabled. Overall, for SD quality, a significant improvement in Quality-of-Service: reduction of amount of qualities switches. For HD, a significant improvement has been demonstrated in improving of quality (resolution and bitrate) of streaming.

(1) Quality of Service: AVA’s impact on QoS shows a reduction of 50% of switch rate sessions for SD

(2) Streaming Resolution: Over 30% improvement of streamed bitrate in high bitrate sessions. We couldn’t identify a

significant improvement in streamed bitrate for sessions in SD quality.

(3) Buffering: 18% reduction of buffering events for both SD and HD.

©2016 Giraffic Technologies Ltd. All rights reserved. For more information please contact info@giraffic.com or visit our website www.Giraffic.com