CS244 ’17: CONFUSED, TIMID, AND UNSTABLE: PICKING A VIDEO STREAMING RATE IS HARD.


Team

Vayu Kishore, Pan Hu

Goals

The original paper [1] performs a measurement study on the the bit-rate selection of the clients of major video streaming services to gain insight into their behavior. The authors also try to improve QoE using a better bit-rate selection algorithm that is less conservative and more robust to measurement noise.

[1] Huang, Te-Yuan, Nikhil Handigol, Brandon Heller, Nick McKeown, and Ramesh Johari. “Confused, timid, and unstable: picking a video streaming rate is hard.” In Proceedings of the 2012 ACM conference on Internet measurement conference, pp. 225-238. ACM, 2012. Harvard

Motivation

Video traffic comprises a majority of the Internet traffic (well over 50% during rush hours in 2012. Generally, commercial video streaming services must implement a bit-rate selection algorithm on the client side: clients constantly estimate bitrate and use this signal to adapt their rate. Choosing the right bitrate is crucial to user perceived quality of experience (QoE). Rebuffing will occur if the bit rate is too high; on the other hand, if it is too low, the user will receive a lower quality video than the network connection can support.

Results

There are two major results presented in the original paper:

  1. Inaccurate bitrate selection, especially in the presence of competing flows, can trigger a phenomenon called the “downward spiral effect”, in which the video player gets much less bandwidth than its fair share, leading to low quality of experience.
  2. The authors analyze the root cause of this effect, and propose and evaluate several remedies including average filtering on the estimation as well as less conservative bitrate selection algorithm. They also verify that large segment sizes lead to more stable video play bitrate and buffer occupancy.

Subset Goal

Our first goal is try to replicate the “downward spiral” effect in the services tested by the original authors, which is shown in the figure below (Fig 4. in the original paper):

fig4.png

We  also try to replicate the results of streaming using modified dash.js clients, which use solutions proposed by the authors for the “downward spiral”  issue. The results are presented in Figures 20 and 21 of the original paper:

fig2021.png

Subset Motivation

We chose these figures as they represent the core findings of this paper — that some video streaming clients cut their bit-rates too aggressively in the presence of competing flows, leading to playback rates and throughputs which are much lower than the fair share. We’re also interested in how modified strategy affect performance of video streaming.

Subset Results

1. Downward spiral effect with third party players

We use the setting as the original paper, and have generated graphs similar to Figure 4 of the original paper for YouTube and Vimeo:

vimeo

Figure 1: Vimeo with downward spiral effect

In the above figure, the competing flow (labelled Flow Throughput) starts at the first dotted line and terminates at the second. The blue line, which represents the video playback rate starts off at 3000kbps, but deteriorates to 1000kbps after the competing flow starts.

youtube_combined.png

Figure 2: YouTube without downward spiral effect

Experimental results show that YouTube’s player does not exhibit the downward spiral effect (video bitrate > 1400kbps) , while Vimeo’s player does (video bitrate < 1400kbps). We chose to start with examining these services, rather than Netflix or Hulu (which are the services examined in the original paper) since they are freely accessible without a subscription and there exist third party tools such as  for manipulating video downloads from these sites.

We also tried to manually force Vimeo to choose a bit-rate of 3000kbps, and we observed no rebuffering, as shown in the figure below.  The result is in accordance with Fig. 5 in the original paper.

vimeo_manual

Figure 3: Vimeo with manual rate selection

2. Testing dash.js including modifications

We first tested the unmodified dash.js client (v2.50RC4). This client uses a moving average of the past 3 samples to estimate the bandwidth of the link. Additionally, it uses a bandwidth safety factor of 0.9. We see a clear downward spiral effect, as shown below.

local_org

Figure 4: Unmodified dash.js client

We then tried to increase the number of averaged samples in the throughput estimation from 3 to 10 in dash_js/src/streaming/rules/abr/ThroughputRule.js

The result is as following:

local_10_0.9

Figure 5: dash.js client with 10 sample moving average. In this case, we see an improvement in video quality compared to the unmodified version.

These results indicate that longer averaging alone is less likely to solve the downward spiral effect. Actually, through the course of multiple experiments we noted that there are some cases where the video quality could be worse in the modified version. The result is in accordance with Fig. 20 in the original paper.

We also tried to modify the bandwidth safety factor in dash_js/src/streaming/model/MediaPlayerModel.js . However, the default safety factor is already 0.9 rather than 0.6 as described in the original paper.  We change the factor back to 0.6, and the result is as following:

local_10_0.6

Figure 6: dash.js client modified to use 10 sample moving average and a bandwidth safety factor of 0.6

The average bit rate using a factor 0.6 is slightly lower. It is required to have a more advanced algorithm to deal with the downward spiral effect.

Additionally, we note that we tend to observe more rebuffering events with the longer moving average estimate.

Challenges

Conceptual: We did not encounter any major conceptual difficulties. Overall, we found the description of the experimental setup and the evaluation in the original paper quite clear. The network topology was well-described as were the parameters for the videos tested.

Implementation: There were several issues we faced in implementing the experiment to replicate the figures:

  1. Lack of visibility into DRM-protected streams. We had initially planned to run the experiments some of the same services tested in the original paper, which included NetFlix and Hulu. However, we ran into difficulties with replicating the experiment on NetFlix for several reasons. First, the streams of these services are protected by DRM, preventing us from analyzing the contents of the streams and deducing, for example, the playback length of the chunks requested by the client. Additionally, the web client does not expose playback bitrate information, or allow the user to manually set the video quality. Due to these reasons, we were unable to replicate the results on NetFlix.
  2. Playback rate extraction. We parse HTTP traffic to extract playback rate information. We currently do this by using Firefox’s ability to export network traffic in the HTTP Archive (HAR) format, which we collect during streaming, and then process offline once the streaming is complete. In order to generate the results, we map the requests for segments to video quality deduced from the MPD manifest file. In some cases, this manual mapping is not explicit in the manifest and must be deduced by comparing streaming quality to the HTTP requests generated by the clients.
  3. Clients not fully saturating links. We noticed that some clients, such as those of NetFlix, Vimeo and dash.js do not seem to fully saturate the network link created by mahimahi. We attribute this to two reasons. First, in some cases, the clients initially download enough segments to build up a playback buffer (during which time they saturate the link), and afterwards, they download segments less frequently, leading to spikes in the link utilization graph, rather than a steady saturation of the link. Secondly, some players may be very conservative to avoid rebuffering events in exchange for lower video quality.
  4. Sensitivity to video viewport size. Video streaming clients may limit the bitrate of their playback based on the size of the viewport in which the video is displayed. For example, if the size is too small, clients may elect to pick a lower bitrate even though higher bitrates are available. In order to prevent this from affecting the experimental results, we require that the displayed video is maximized upon playback. However, this approach may not be sufficient if this experiment is run in screens that do not support sufficiently high resolutions.
  5. Noisier results when running within a VM. We have provided instructions for a VM in order to enable smooth replication of our results. However, we have noticed that the quality of the graphs produced is sensitive to whether we are running the experiment in a VM or directly on a physical machine. For example, we found that setting a limit on the downlink queue to 15000 bytes with a droptail policy when running in a physical machine allows us to generate smooth link utilization graphs. However, if we set the same queue size in a VM, our graphs become noisier, perhaps due to the network latency incurred by running within a VM. Additionally, if we allow an unbounded queue on the downlink, we get cleaner results in the VM, though not as smooth as with running on a physical machine.
  6. Unable to host virtual machines on Google cloud. There is no straight forward way to make Firefox running without any display. It is possible to run the VM in cloud, and forward the display to a  local machine, but that will be extremely slow when playing video. Our solution is setup the VM in Virtualbox and run it locally.

Critique

Overall, we found that the critique seems to hold well – we were able to observe a downward spiral effect within a web client of a popular video streaming service. Notably, we were able to replicate the results even though we are using different hardware and software to emulate network links, which indicates that the results of the original paper are robust to the network emulation layer. Additionally, like the original authors of the paper, we observed that in cases where we observed a downward spiral effect, manually setting a bit-rate allows better link utilization in the presence of a competing flow. Furthermore, based on our experience with YouTube’s client, we observe that some web clients do not exhibit this effect, indicating that their clients may be more conservative in reducing bit rate in the presence of competing flows. Since YouTube’s client was not one that was tested in the original paper, we do not know if it previously exhibited a downward spiral, or whether it was changed since the publication of the original paper.

Extensions

We experimented with generating graphs using different moving average filters. Below are the same graphs as above using a 200 ms moving average filter to smooth the throughput, instead of a 4 second moving average.

vimeo_100ms_smoothing

Figure 7: Vimeo output with shorter moving average filtering for throughput

youtube_100ms_smoothing

Figure 8: YouTube output with shorter moving average filtering for throughput

The original paper used buffers to induce latency in their connection. The results above use an infinite buffer size. We experimented with using smaller buffer sizes (50kBytes) on the downlink, and the result is shown in Figure 9 below. We found that with a downlink buffer size set too low, which causes packet loss in addition to latency, the client and the flow are both unable to saturate the link, because of which we do not see a clear spiral effect.

vimeo_low_buffer_combined

Figure 9: Vimeo output with small buffer

We experiment with the throughput rate set on the downlink to see if the downward spiral effect is sensitive to this value. The result of Vimeo throughput with 10Mbps bottleneck link is shown below:

vimeo_high_throughput

Figure 10: Vimeo output with larger link rate

From Figure 10 we could see a more pronounced drop in video quality, but it takes longer for the spiral effect to occur.

We also test the throughput rate set on a link with 4Mbps capacity. The result is shown in Figure 11:

vimeo_low_throughput.png

Figure 11: Vimeo output with smaller link rate

As shown above, with a lower link rate of 4Mbps, the player chooses the lowest bit rate available even before the video starts and does not saturate the link. As a result, we do not see a downward spiral effect when the competing flow starts.

Platform

The original implementation using NetFPGA, a programmable packet switch to simulate the bottleneck link but it is hard to get the required hardware,  so we re-implement the experimental setup using mahimahi [2] to emulate the network conditions outlined in the paper (a bottleneck link of ~5Mbps). Additionally, we use the metering functionality of mahimahi to track throughput of the video stream and the competing flow.

[2] Netravali, Ravi, Anirudh Sivaraman, Somak Das, Ameesh Goyal, Keith Winstein, James Mickens, and Hari Balakrishnan. “Mahimahi: Accurate Record-and-Replay for HTTP.” In USENIX Annual Technical Conference, pp. 417-429. 2015.

We use Firefox’s developer tools to track HTTP requests made while the player is running. We save these requests and process them to extract the video playback rate for the client. Unlike the use of mahimahi to measure throughput, which can be easily applied to new streaming services, determining the playback rate of the client requires custom parsing and analysis of HTTP requests to be performed for each streaming service independently.

To assist with this, we use youtube-dl [3], which can parse DASH manifests of some video streaming websites (though, notably, not Netflix or Hulu), which we use to map the HTTP requests made during the playback to one of the rates enumerated in the DASH manifest. We also use youtube-dl to set up a competing TCP flow to the video streaming server and directly download the video being streamed.

[3] Youtube-dl, https://rg3.github.io/youtube-dl/

We encapsulate this environment in a VM image which we will make available for download. By using software components and a VM image, we believe that our platform is straightforward to use to replicate our experimental results. Some aspects of the setup that can affect reproducibility are screen resolution within the VM and the network speed and latency within the VM.

README

Prerequisites:

  1. Install VirtualBox or another Virtual Machine Manager.
  2. Download the VM image from this link.
  3. Import the VM image (.ova file) into the VM Manager. For VirtualBox, this can be accomplished by choosing File -> Import Appliance
  4. (Optional) The VM image has 4 CPU’s allocated to it. Change this to half the number of CPU’s on the physical machine. On VirtualBox, this can be done by right clicking the machine name and choosing Settings->System->Processor and picking the number of processors
  5. Start the VM. Password is password
  6. Use xrandr to check the resolution. Ensure it is at least 1280×1024. If it is not, adjust settings in the virtual machine to adjust the resolution
  7. To run the experiments, the VM requires access to the internet with at least 5Mb/s throughput. A wired link is preferable, but not necessary

Replicating experiments

  1. Open up the LXTerminal Application and cd ~/cs244-pa3/mahimahi
  2. Follow the below instructions to replicate our figures:
    • Figure 1:
      1. ./run.sh vimeo
      2. When browser opens up, maximize the video screen and hit play
      3. Wait for at least 180 seconds. Optionally, you can use ALT+TAB to confirm that there are three downlink graphs open between before the 140 seconds time mark. These track the throughput of the browser, the direct download, and the combined link
      4. Pause the video, hit escape to exit full-screen
      5. Right click in the developer tools network section and click “Save All as HAR”
      6. Save to ~/cs244-pa3/mahimahi/data/vimeo.har  . Be sure to save in the data folder – it is easy to save to the wrong path.
      7.  Exit the browser
      8. After you exit the browser, if everything went smoothly, then the graph will be generated in the figures directory. Otherwise, examine the output for errors. One issue that we have noted with replication is that there is no internet connectivity in mahimahi shells that are newly spawned through mm-link. To mitigate the effect of this issue, we test network connectivity with ping once​ ​mm-link spawns a shell, and if we cannot establish a connection, we fail the experiment with descriptive output. If you see this happening, interrupt the execution of the experiment with CTRL+C, then run ./scripts/cleanup.sh, and then rerun the experiment to generate the figure. Additionally, make sure not to suspend or try to exit the VM while the experiment is in progress as this can affect the timing computations. The graphs should be saved in ~/cs244-pa3/mahimahi/figures/vimeo_combined.png . A shortcut to the figures folder is accessible through the desktop.
    • Figure 2: Same instructions as Figure 1, but replace vimeo with youtube in steps 1, 6, and 8. Additionally, there’s no need to hit play as the video auto-plays
    • Figure 3: Same instructions as Figure 1, but manually select 720p in step 2 before hitting play
    • Figure 4: Same instructions as Figure 1, but replace vimeo with local . Additionally, when the browser first opens, click the first link (Unmodified client)
    • Figure 5: Same as instructions as Figure 4, but pick the second link (60% Bandwidth safety limit with 10 sample moving average)
    • Figure 6: Same as instructions as Figure 4, but pick the third link (60% Bandwidth safety limit with 10 sample moving average)
    • Figures 7 and 8: Are generated as vimeo_200_ms_combined.png and youtube_200_ms_combined.pngfrom steps for figure 1 and 2.
    • Figure 9: Same as for Figure 1, but use ./run_extension_buffer.sh vimeo for step 1
    • Figure 10: Same as for Figure 1, but use ./run_extension_throughput_high.sh vimeo for step 1
    • Figure 11:  Same as for Figure 2, but use ./run_extension_throughput_low.sh vimeo for step 1

For reference, our git repository is available here.

 

Advertisements

2 responses to “CS244 ’17: CONFUSED, TIMID, AND UNSTABLE: PICKING A VIDEO STREAMING RATE IS HARD.

  1. Reproducibility: 5/5

    There were initially issues with DNS inside the VM, but those were resolved fairly quickly thanks to really rapid help from Pan and Vayu.

    Nearly all of the figures matched exactly what they had in the blog post, with the exception of Figure 2, which diverged somewhat:

    However, it still conveys the point that YouTube doesn’t suffer from the downward spiral effect, and so we still considered this a success.

  2. Thank you for your comment. Youtube may adopted a different type of TCP, such as BBR, that may explain the reason. Despite the reduced throughput, Youtube still maintains a stable playback bitrate, which is better than default dash.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s