CS 244 ‘15: Reproducing the 3G/WiFi application level latency results in MPTCP


Paper: How Hard Can It Be? Designing and Implementing a Deployable Multipath MPTCP [1]

Team: Wen-Chien Chen and Kevin Han

Motivation and goals

Extending TCP to support data transmission for a single TCP connection on multiple paths would lead to benefits in robustness, performance, and load-balancing by providing redundancy and distributing traffic over less congested paths. This is realized in Multipath TCP (MPTCP), a protocol currently in the IETF standardization process. To make widespread adoption possible, MPTCP has four major goals in addition to benefiting from multipath transport:

  1. It should be usable without application modification if it is supported at both endpoints of a connection.
  2. It should work in all scenario where TCP works.
  3. It should utilize the network at least as well as TCP does, but without starving TCP.
  4. It should be implementable in operating systems without excessive memory or processing requirements.

The design and deployment of MPTCP is constrained by today’s networks, where middleboxes may modify packets in optimization attempts, and both multi-homing and NATs complicate connection setup. In this paper, the authors focus on examining the constraints that TCP’s design, existing networks, and operating systems impose on the design of MPTCP. Furthermore they evaluate the performance of their implementation of MPTCP and several adaptive mechanisms in the Linux kernel using different metrics.

Results

The authors have shown that it is possible to implement MPTCP in a way that makes it robust in the presence of middleboxes, and deployable on today’s Internet. Furthermore, the algorithms proposed effectively mitigate problems such as requiring a much larger receive buffer. Based on evaluation results, the authors conclude that MPTCP is ready for adoption.

Reproducing the 3G/WiFi application level latency results

One major usage scenario for MPTCP is in wireless networks, where it would allow devices to use both WiFi and 3G for a single TCP connection. While MPTCP enables higher throughput than using the fastest path alone, the end-to-end latency seen by an application would increase because MPTCP utilizes several subflows with different RTTs. Since latency is important in interactive applications, we are motivated to reproduce the 3G/WiFi application level latency results from the paper, which measure the latency experienced by an application sending 8 KByte blocks of data on four different connections: TCP over an emulated WiFi path, TCP over an emulated slow 3G path, MPTCP over the two paths, and MPTCP with additional mechanisms, namely opportunistic retransmission and penalizing slow subflows, enabled.

The results from the paper are captured in the following graph (Fig. 7 in the paper):

fig7MPTCP with the two additional mechanisms enabled (MPTCP + M1,2) achieves results that are both comparable to those of TCP over WiFi, and have good tail latency. The authors note that the latency of TCP over WiFi being greater than that of MPTCP + M1,2 is due to the 200KB send buffer being higher than needed for the WiFi link, meaning that data spends more time in the buffer in the former case.

Our results

ourFigure 1 – Reproduced Results

We were able to reproduce results similar to the ones presented in the paper. In particular:

  1. TCP over 3G sees significantly higher delay compared to TCP over WiFi.
  2. MPTCP with optimizations achieves latencies comparable to those of TCP over WiFi, with the improvement possibly caused by the excessive buffering for WiFi described above.
  3. The range of latency values seen under MPTCP with optimizations is narrower compared to those under MPTCP without optimizations, and has a lower average.

There are some discrepancies, notably the range of delay values for TCP over 3G and the fact that our results for WiFi and MPTCP are less smooth. We believe the former is a result of our chosen link characteristics (jitter, loss rate, etc.) being different from those in the original experiment (which are not clearly specified in the paper), and the latter is a result of differences in simulation environments and PDF estimation. Lastly, the Y-axis of our graph has a 10x difference compared to the figure, but we believe this is because the original graph was mislabeled (since the area under the each curve does not sum up to 1).

Challenges encountered

We encountered three major challenges throughout the course of the project:

I. Rebuilding the kernel and patching MPTCP

Since MPTCP is not yet implemented in any mainline Linux kernel version, we had to build a modified version of the kernel to run the experiment. Furthermore, as the mechanisms for opportunistic retransmission and penalizing slow subflows had been integrated with MPTCP permanently, we had to make use of a patch made available by one of the authors of the paper (Christoph Paasch) to add options for disabling the mechanisms. Incompatibilities between the patch and different versions of MPTCP and the Linux kernel exacerbated the problem. We eventually solved this by using a snapshot of the development of branch of MPTCP from May 17, 2014 with version 3.14 of the Linux kernel (using Ubuntu 14.04), and manually applying the patch to MPTCP.

II. Making MPTCP work correctly

Our greatest challenge during this project was solving an issue where MPTCP appeared to perform nearly identically with the additional mechanisms enabled or disabled. After many attempts changing program logic and trying different parameters, we discovered that this was caused by two distinct issues:

  1. MPTCP seemed to have been using the WiFi link exclusively. Since there was no multipath, the mechanisms made no difference.
  2. The default size of the receive buffer in the Linux kernel is large enough such that the additional mechanisms, designed to handle issues caused by congested buffers, did not have an obvious effect.

Further investigation revealed that for the 3G link to be used when the IP address of the WiFi interface is used to connect to the server, we had to first ping the 3G interface address to allow MPTCP to learn of the existence of the alternative route. Once we did this and manually set the socket’s receive buffer to 32KB, MPTCP and the additional mechanisms worked correctly.

III. Unclear link characteristics

The paper does not specify the characteristics of the links used in the experiment. To compensate, we used the link characteristics specified for an experiment earlier in the paper (8Mbps with 20ms RTT for the WiFi link, 2Mbps with 150ms RTT for the 3G link). We assumed that the link buffers, specified in time (80ms and 2s for the WiFi and 3G links, respectively), referred to the queue length of the switch divided by the link rate. Substituting 1500 bytes (MTU) for packet size, we used 54 packets as the queue length for the WiFi switch and 334 packets for the 3G switch. We also set the jitter of the 3G link to 50ms, as it’s within the range of values found in other papers (e.g. [2]) and produced results similar to the ones in the paper. The jitter of the WiFi link and the loss rates of both links were set to 0.

Critique

Based on our experiments, we believe this particular set of results from the paper holds well. However, the lack of clear link characteristic specifications made reproducing the results more difficult, and we were unable to reproduce some of the results until we manually set a smaller receive buffer size, though it’s not clear if the authors of the paper did the same for their experiment. Overall, it can still be concluded that with the optimizations enabled, MPTCP does not cause the latency to degrade too much in the face of asymmetric links.

Extensions

Given that the authors were not too clear on the exact link characteristics they used for the WiFi and 3G links, one of the things that we wanted to explore was how much of an effect the bandwidths of both links affected the results. In particular, we wanted to know if increasing the difference in speeds of both links would throw the MPTCP results off. To resolve this question, we ran a sensitivity analysis using different (higher) ratios between the link bandwidths (the original test had the WiFi link at 4 times the 3G link (8 Mbps/2 Mbps). Here are the results obtained:

s1Figure 2 – 8 times link proportions (WiFi 16Mbps, 3G 2Mbps)

s1Figure 3 – 16 times link proportions (WiFi 32Mbps, 3G 2Mbps)

s1Figure 4 – 32 times link proportions (WiFi 64Mbps, 3G 2Mbps)

It is clear from the results above that while the peak of each line remains similar for both MPTCP runs, the optimization mechanisms become less effective as the ratio between the link bandwidths increases. This is evidenced by a higher percentage of values with larger latency values for the MPTCP with optimizations case. However, MPTCP with optimizations still achieves similar latencies as TCP over WiFi, while both have lower latencies than MPTCP without optimizations.

Platform

We performed our experiments in Mininet 2.2.1 in a c3.xlarge instance on Amazon EC2 running Ubuntu 14.04 and our modified Linux kernel (version 3.14 with MPTCP compiled from the snapshot). Mininet was chosen since we were already familiar with it, and we used EC2 since the isolation and available computing power would make experiment results more repeatable. We opted to use Ubuntu 14.04 instead of trying to recreate the original test environment to reduce the likelihood of issues with package installation and updates. All of our scripts are written in Python, and we used a monotonic time module to avoid issues caused by NTP clock updates.

Our setup should be easily reproducible, as we provide the patched version of MPTCP we used for our experiment and our modified kernel in our Git repository in addition to the convenience AMI. Assuming a similar setup, we believe our results are repeatable.

Running our experiments

The code used to run our experiments can be found at the following URL:

https://bitbucket.org/iamtheone188/cs244-2015-wckh-mptcp/src

For convenience, we have packaged an EC2 AMI running the modified kernel and with all dependencies preinstalled, which is available in the US West (Oregon) region with the identifier ami-dbc9f6eb. Simply launch an instance using the AMI, clone the above Git repository, and execute run.sh as root to run the main experiment. Please refer to the README file in the repository for more detailed instructions.

References

[1] Costin Raiciu, Christoph Paasch, Sebastien Barre, Alan Ford, Michio Honda, Fabien Duchene, Olivier Bonaventure, and Mark Handley. 2012. How hard can it be? designing and implementing a deployable multipath TCP. In Proceedings of the 9th USENIX conference on Networked Systems Design and Implementation (NSDI’12). USENIX Association, Berkeley, CA, USA, 29-29.

[2] Yung-Chih Chen, Don Towsley, Erich M. Nahum, Richard J. Gibbens, and Yeon-sup Lim. 2012. Characterizing 4G and 3G networks: Supporting mobility with multipath TCP. UMass Amherst Technical Report: UMCS-2012-022.

Advertisements

2 responses to “CS 244 ‘15: Reproducing the 3G/WiFi application level latency results in MPTCP

  1. The code did not immediately run at first: we had to make one minor change in mptcp.py, line 98, to change “args.bw3G” to “args.bw3g”. With that correction, though, the code ran to completion in ~8 minutes and the results were extremely close to those reported in the original post.

    Overall reproducibility was very easy! Adding the EC2 AMI to pre-configure the kernel and Mininet environment was a nice touch!

    Reproducibility score: 5.0 / 5.0

    Sensitivity analysis seems good – it looks like the team made had good understanding of the sources of variation in their measurements. The extension to quantify the variability due to the relative performance between WiFi and 3G links supported their arguments. Nice job!

    • Thank you very much for catching the bug–my bad for not retesting after a last-minute change. I’ve fixed the code in our repository. And thank you for the evaluation.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s