CS244 ’15: Evaluating PCC: Re-architecting Congestion Control for High Performance


Team: Kevin Shin (hgkshin | hgkshin@cs.stanford.edu), Omar Diab (odiab | me@omardiab.com)

Key results: PCC with the default utility function indeed does provide far better throughput than TCP in suboptimal network conditions. However, it also is very unfriendly to existing TCP flows and has significant problems with operating in a virtualized setting that shares tenancy. We discuss the tradeoffs and offer our critique below.

Sources:

Introduction

The PCC paper argues that TCP and its variants inherently can’t consistently provide high performance in changing network conditions. To handle complex network behavior, TCP relies on **hardwired mappings**—that is, fixed responses (like halving cwnd) for packet-level events (like drops and RTT increases). In doing so, TCP implicitly makes faulty assumptions about the causes of and proper responses to network unknowns. This causes TCP and its variants to produce suboptimal throughput and congestion control behavior in practice.

With PCC, the client runs A/B tests to determine what rate it should send at. It sends data at multiple rates as experiments, and uses observed network behavior as input to a utility function that expresses desired network performance characteristics. It then chooses what rates to send at to produce the best output utility.

Rather than the protocol making assumptions about network behavior, the utility function—which can be set by the sender—can choose what performance characteristics to prioritize. Therefore, the authors claim it can handle complex, varied network conditions flexibly. With the right utility function for fairness and convergence, they argue that PCC consistently produces 10x faster performance than TCP, and doesn’t require hardware support or new packet formats, making it relatively easy to adopt.

Motivation

PCC is an interesting protocol to analyze because it claims to produce dramatically higher throughput than TCP under the varied network conditions that compose modern links today, while at the same time improving flexibility. TCP, for instance, penalizes flows that have long RTTs because it takes longer for the congestion control algorithm to observe packet losses and transmissions, forcing it to slow down the entire TCP cycle. Additionally, TCP penalizes links with high packet loss rates by causing cwnd to enter congestion avoidance more frequently than those with lower packet loss rates. In both cases the authors argue that PCC exhibits far more efficient use of network equipment and offers greater fairness for servicing different flows over the network that experience complex, different network conditions.

The conditions above are not farfetched scenarios these days. Long RTTs correspond to satellite link connections and packets destined across the globe, passing through multiple networks; high packet loss scenarios happen all the time in wireless networks that experience significant interference, and mobile networks. In other words, the conditions in which PCC claims to dramatically increase throughput correspond to ubiquitous scenarios in modern networks.

Additionally, if PCC’s performance guarantees are reproducible and theoretically valid, then it begs the question about the way that routers and clients on the internet ought to behave. PCC was designed to not require changes to internet infrastructure, but the utility function proposed in the paper doesn’t behave kindly to existing TCP flows, since it uses a completely different congestion control model than TCP. A PCC client can easily saturate and harm the quality of service for TCP flows (and possibly TCP-friendly PCC flows), so it begs the question of whether or not the internet’s reliance on TCP for the reliable transfer of data has been a tussle in the development of optimal data flow algorithms. We are interested in exploring the ramifications of such a finding.

Methodology

Execution environment

The authors of PCC provide the source code for their implementation of the PCC protocol that transmits over UDP. With some code changes to make it easier to collect data about the performance of PCC flows, we compiled it and ran our tests in two settings:

  • Mininet on an Ubuntu Amazon AWS VM (dedicated c3.4xlarge). This was chosen for simplicity and ease of reproducibility.
  • A real, physical topology in Emulab, a set of dedicated machines one can request for research purposes. (Thanks to Mo Dong for giving us access to the Emulab service). We found that Mininet does not give optimal performance (explained below) and chose to reproduce the results in Emulab to confirm the paper figures.

The Amazon tests can be reproduced by using our public AWS AMI (id: ami-97fec1a7). This will make it easier for others to reproduce some of the results we find during this project. We discuss the specific details to run in the README in our bitbucket repository.

Topology

As with the paper, we used a simple topology for our tests. Two hosts are connected by a single router. In the AWS scenario, this topology is created via Mininet’s Python API. In the Emulab scenario, the host machines run Fedora 15, and the bridge machine runs FreeBSD with dummynet so that it can act as a router.

We change the properties of the links and router between the hosts (link latency, packet loss rate, buffer size…) depending on the experiment; in the Mininet scenario, this is done through Mininet’s Python API; in the Emulab environment, parameters are set using the ipfw tool. For example, to set the link between the sender and receiver to be 100MB/s with 30ms RTT, 300KB network buffer and 0.3% loss, we run this command:

ipfw pipe 100 config bw 100Mbit/s delay 30ms queue 300KB plr 0.003

For Figure 6, we reproduced the paper’s topology of bandwidths of 42 Mbps, an RTT of 800 ms, and a loss rate of 1%. We changed the bottleneck buffersize to collect the data for our experiment.

For Figure 7, we reproduced the paper’s topology of bandwidths of 100Mbps, an RTT of 30 ms, and a bottleneck buffersize of 375 KB. We changed the loss rate to collect the data for our experiment.

Finally, to measure TCP friendliness, we set up a topology of bandwidths of 100 Mbps, an RTT of 30 ms, a bottleneck buffersize of 375 KB, and a loss rate of 0.

PCC utility function

For all of the tests, the PCC flow used the default utility function in the PCC source code—namely, the one that the experiments in the PCC paper used. Since PCC senders can only observe local phenomena, all utility functions in PCC will ultimately be selfish. This particular utility function optimizes for two things: low loss rates and high throughput. It is designed so that at convergence it also produces fair allocation between multiple PCC hosts. The specific function used is described in Section 2.2 of the paper.

Throughput measurement techniques

We used the average of TCP and PCC throughput over 1 minute of sending and receiving data to reproduce the figures. We talked with Mo Dong, and he confirmed that this was the methodology used in the actual PCC paper.

Figures reproduced

We replicated two of the figures (as of the 3rd version of the paper): Figures 6 and 7. Figure 6 shows that in scenarios with long RTTs, PCC maintains significantly faster throughput despite changing buffer sizes. The TCP variants used are Reno (unreproduced on Emulab due to lack of library support) Cubic, Illinois and Hybla. As the paper describes, the first two are common TCP variants while the latter two are designed to sustain losses and long links respectively. Figure 7 demonstrates that in networks with high packet loss rates, PCC maintains very high throughput while TCP suffers. The TCP variants used are Illinois, a packet loss rate optimized variant, and Cubic, a common TCP variant. We chose these figures as they show PCC performing highly even in extreme but still modern conditions. If these figures held true, then it would motivate a strong reason to use PCC for its robustness and high performance.

We also did an experiment of our own that was not covered in the paper to assess the effect of a PCC flow using the default utility function on throughput of TCP flows. Because PCC’s default utility function is not built for TCP friendliness, we expected a dramatic decline in TCP throughput when a PCC flow is present, which was confirmed.

Results

Under Emulab, we were able to reproduce the authors’ figures faithfully and attribute the utility function for the increase performance, but in the Mininet/AWS environment, we observed discrepancies.

For Figure 6, after discussion with Mo Dong directly, we believe it is likely that the virtualized environment of AWS containers degraded PCC performance. PCC relies on pacing—spreading out sending packets over time—but this relies on PCC able to send packets at precise times. The virtual machine, however, may be interrupted by the host OS, causing packets PCC would send to get delayed and sent all at once later. For small router buffer sizes, we observe that PCC exhibits poor performance because packets that ordinarily would succeed instead get dropped, causing PCC to throttle its packet sending rates frequently. This will consistently affect PCC’s collected statistics to lower sending rate, thus lowering throughput. For larger buffer sizes, while the problem isn’t as bad because the buffer is less likely to be overrun, packets may still occasionally get spuriously dropped in a consistent manner, explaining suboptimal performance compared to the Emulab results. For larger buffer sizes where the pacing issue is minimized, we see that PCC significantly outperforms TCP as expected.

For Figure 7, interestingly, TCP flows in the Mininet environment appear to do marginally better than the paper and Emulab results claim they would, and PCC appears to do worse than it ought to. While the PCC degradation is explained above due to PCC’s pacing mechanism, we investigated the increase in TCP performance further and found out that “the TCP flow exceeds desired performance at first in Mininet then degrades gradually as the background load increases” [2]. However, since we are running on dedicated AWS instances, there is no increase in background load, explaining the above optimal performance of TCP. This was an interesting side result that we discovered in the process. Regardless, however, for loss rates between 1-3%, PCC significantly outperforms the TCP variants, and the Emulab results confirm that in a physical setting with PCC’s performance optimized and TCP’s performance in a real world setting, PCC does outperform the TCP variants across all loss rates shown.

PCC paper’s figures:

pcc-figures-large

Emulab figures:

emulab_fig6emulab_fig7

Mininet/AWS figures:

buffer_logloss_log

Mininet/AWS figures, on a linear scale

When viewed on a linear scale with the same data, the difference in performance between TCP variations and PCC for Figure 6 becomes very pronounced, as indicated in the paper.

buffer loss

TCP (un)friendliness test (novel, optional experiment)

With 5 TCP flows running at the same rate, we see what we expect: the usual TCP oscillation, eventually coming closer and closer to convergence. When a single PCC flow with the default utility function runs alongside 4 TCP flows, the TCP flows experience horrendous throughput, and PCC dominates the link. Because it backs off far less than TCP does in response to packet losses, TCP just assumes that the link inherently cannot handle as much traffic, whereas PCC greedily takes as much bandwidth as it can, explaining the results. PCC’s behavior in these situations can be improved by taking into account some of the feedback mechanisms TCP does, via a utility function that has TCP friendliness baked into it.

many_pcc_inactivemany_pcc_active

Lessons learned/Challenges

Overall, we learned that reproducing network experiments and studying network performance in different scenarios can be dramatically affected by the environments that the experiments are run in.

For instance, one approach we tried to take to better isolate PCC performance during our experiments was to use likwid-pin to pin the sender and receiver threads to different cores, so that they wouldn’t interrupt each other. Unfortunately, likwid-pin didn’t interact well with shared AWS boxes, and therefore did not mitigate the problems described above. Instead, we chose a high-core, dedicated AWS machine to better isolate the sender and receiver threads on dedicated cores and also reproduced the results on Emulab for verification.

Conclusion & Critique

We see a couple major issues to the introduction of PCC into real networks:

  • PCC’s proposed utility function, as it is conceived right now, is unfriendly to TCP flows, making it difficult to introduce into today’s internet. This is significant because most of the internet relies on TCP to send information, so dropping PCC into the network would disrupt significant numbers of pre-existing flows.
  • PCC does not seem to work well in virtualized environments. Given that many web applications these days are being run on virtualized cloud services, like AWS and Azure, PCC may actually be ill-fitted for optimizing performance due to its reliance on pacing. We found this to be the most concerning result as it was not directly mentioned in the paper and most Internet applications run in virtualized environments today. In fact, the paper directly mentions that “an individual tenant can use PCC safely within its virtual network to address problems such as incast and improve data transfer performance between data centers.”

With respect to the first, we realize that while it is true that PCC is quite unfriendly to TCP flows, there are scenarios in which TCP-unfriendly behavior is accepted. For instance, BitTorrent opens many parallel connections simultaneously, and, according to Nandita Dukkipati Google and many others set cwnd to be very large upon opening TCP connections. Hence, there are likely scenarios where PCC can operate without disrupting other flows dramatically [4].

With respect to virtualization, we brought up this issue to Mo Dong. He confirmed the issue, and also explained that his group is currently working on a solution to mitigate PCC pacing issues to solve the problem and confirmed that it was not a problem inherent to PCC, but rather to the current implementation. If this solution is fixed, then PCC could have a big use case in the virtualized setting as well.

If these issues are addressed, we believe and are excited that the dramatic improvement in performance that PCC promises under stressful conditions is a strong motivator to be considered as an alternative to TCP on a large scale.

Instructions to reproduce

Mininet/AWS experiments:

Follow the instructions in our repository’s README. Note that it is important to use our public AMI since it includes a binary of PCC. We discuss the importance of the time parameter in creating reproducibility and also add that because of PCC’s pacing, the buffer size is also an equally important parameter in reproducing the results above.

Emulab experiments

Since students do not have access to emulab, Mo Dong was very generous and helped us set up our environments under his PCC experiment cluster. Unfortunately, as a result, we cannot go into full detail on how to set up the individual machines. But once you have two Fedora 15 machines and a FreeBSD machine up with dummynet installed, you should be able to clone the PCC repository and follow the directions as described in the PCC repository.

Advertisements

2 responses to “CS244 ’15: Evaluating PCC: Re-architecting Congestion Control for High Performance

  1. I’m leaving a 5 for reproducibility. While you might need to run it once or twice, the resulting graphs should match the graphs in the blog very well. To run the script faster, edit run.sh, so that you only need to run the tests you need to see. I had to re-run twice to the right loss graphs.

  2. Pingback: CS244 ’16: PCC Shallow Queues and Fairness | Reproducing Network Research·

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s