by Yanshu Hong, Koki Yoshida
TCP provides in-order, reliable, two-way byte streams between applications running on hosts that are communicating over an IP network. Its security robustness is an interesting but often less explored topic than its performance metrics. We strongly felt that any discussion around the design of TCP should not only focus on its performance in terms of throughput and latency, but also on its security robustness. This is the reason that we decided to reproduce the results in the paper “TCP Congestion Control with a Misbehaving Receiver” (SIGCOMM 99′) by Savage et al.  to raise public awareness of TCP security issues. In the paper, Savage et al. found out that they could easily setup a misbehaving TCP receiver to drive a standard TCP sender (implemented per RFC 2581) arbitrarily fast without losing end-to-end reliability. They demonstrated so with three simple attacks (described in detail in Section 2).
Our motivation for reproducing these attacks is very simple. Since they are receiver-side attacks, the ramification of the vulnerabilities that they manifest is profound because any greedy Web client at the time can implement them to speed up connections and obtain an unfair bandwidth share. The attacks can even be extended into a DDoS attack: a group of malicious receivers can trigger an excessive number of aggressive senders, congesting and then eventually breaking down bottleneck links.
2. Paper Results
In this section, we will present all three attacks from the paper: ACK division attack, DupACK spoofing attack, and Optimistic ACKing attack.
2.1 ACK Division Attack
RFC 2581 specifies,
During slow start, TCP increments cwnd by at most SMSS bytes for each ACK received that acknowledges new data. […] During congestion avoidance, cwnd is incremented by 1 full-sized segment per round-trip time (RTT).
We can see that the TCP congestion control is defined in terms of segments rather than bytes. However, TCP uses byte granularity for its control protocol (sequence number / ack number). Thus, ACK division attack works as follows: upon receiving a data segment with length N bytes, the attacker splits its ACK into M number of partial ACKs, with each partial ACK acknowledging only N / M bytes. See below for an example packet sequence of the attack and the result of running this attack against cnn.com, requesting for the “index.html” page back at the time.
2.2 DupACK Spoofing Attack
Again, RFC 2581 specifies part of the fast recovery algorithm as,
(Upon receiving 3 duplicate ACKs) Set cwnd to ssthresh plus 3*SMSS. This artificially “inflates” the congestion window by the number of segments (three) that have left the network and which the receiver has buffered. […] For each additional duplicate ACK received, increment cwnd by SMSS. This artificially inflates the congestion window in order to reflect the additional segment that has left the network.
Here, we can see that there are at least two problems that could potentially be exploited: first, TCP assumes that each packet that has left the network is full-sized (SMSS); second, the sender cannot distinguish a valid duplicate ACK from a forged one. Thus, DupACK Spoofing attack works as follows: upon receiving the first data segment, the attacker sends as many spoofed duplicate ACKs as she wants, forcing the sender to inflate its congestion window in the fast recovery state. See below for an example.
2.3 Optimistic ACKing Attack
This attack is the conceptually easiest one to understand. RFC 2581 assumes that the time between a data segment sent and an ACK for that segment received is at least one RTT. However, there is no enforcement of this assumption. Thus, it is possible for a receiver to fake a shorter RTT by sending an ACK before it even receives the segment for that ACK. Hence, the Optimistic ACKing attack works as follows: after receiving the first data segment, the receiver starts sending out ACKs for future data segments optimistically. See below for an example.
3. Reproduction Results
We implemented all three attacks above, and successfully reproduced the time / sequence number plots on the right (corresponding to Fig. 4, 5, 6 in the paper). We chose to reproduce these plots because we thought they are the most straightforward graphs that illustrate both the feasibility and the severity of the attacks, against TCP implemented according to RFC 2581.
We started with implementing a standard TCP Reno client using Scapy (reno.py). Scapy is a user-space Python package that provides high-level APIs to encode / decode packets of a wide number of protocols, send them on links, capture them, etc. Then, we modified our TCP Reno client to incorporate all three attacks (attacker.py) and run the attacks on a simple Mininet topology: two hosts connected via a switch. We record the sequence number of the packets that the receiver receives and the ack number of those that it sends out. Finally, we plot these statistics.
Our reproduced plots are shown below (in comparison with the original plots). Note that since Python and Scapy are much slower than C, we use much longer RTT to accentuate the effectiveness of the attacks. This results in a different x-axis, but it should otherwise have minimal impact on the validity of our reproduction.
We used the Assignment 1 VM for our development since it requires less configuration work on the audience side when reproducing our work. The VM is preinstalled with Mininet and Scapy.
3.1 TCP Reno Implementation
We implemented most of TCP Reno, focusing on its congestion control (slow start, congestion avoidance and fast recovery). The audience should be able to start a Ping-Pong conversation between two such TCP Reno clients following the instructions in our GitHub repo. Sample sawtooth graph of such conversation is shown below. Our TCP logic is all implemented in user space because it gives us more control over TCP’s behaviors, requires no change to the OS kernel, and would be much easier for the audience to work with during reproduction.
3.2 Implementation Challenges
(1) Although Scapy is a user-space library that enables packet manipulation from a user-space program, it still uses kernel sockets for its networking APIs. Thus, initially when we implemented our baseline TCP Reno, the packets were intercepted by the kernel and a series of reset packets are injected. To circumvent this issue, we ignored all reset packets in our implementation, to prevent kernel’s interference with our TCP flow.
(2) Scapy does not queue up the packets that are not sniffed in time. Scapy exposes an API sniff() to allow programmers to capture incoming packets. However, if a packet arrives while the program is not sniffing, the packet is lost forever. To solve this problem, we had to spawn a background thread that only listens for and queues all sniffed packets to a global receiver queue. We don’t have to worry about data race due to Python GIL.
(3) Scapy is very slow when it comes to sending out packets. With a number of experiments, we observed that it roughly takes Scapy 40ms just to send 1 ACK out. This is a huge problem for performing various ACK attacks, which require quickly sending out multiple ACK packets (say, 50) so that the sender’s congestion window can be instantly inflated. However, since it takes 40ms for each ACK to be sent out, the effect of such attack is mitigated since there are time gaps between each ACK’s arrival, causing the sender’s congestion window to only grow slowly. To circumvent this issue, we had to increase our link delay so that Scapy’s overhead is relatively trivial compared to one RTT. We use 375ms (by default) per link delay, which amounts to an RTT of 1.5 seconds. After this modification, our plots look very similar to the original plots provided in the paper.
We also implemented defense to these attacks following the proposal in the original paper. In essence, the defense mechanism adds a random nonce to each outgoing data segment at the sender. Since Mininet never fragments packets, the sender can then only accepts ACKs that are complete (cover all the data bytes sent in the last segment) and have a nonce that matches one of the sent data segments’ nonce. Additionally, when a sender nonce is matched, it is discarded to prevent future reuse of the same nonce.
The above mechanism effectively prevents all three attacks demonstrated. It prevents ACK Division attack since sender now only accepts complete ACKs. It prevents DupACK Spoofing attack since each nonce can only be used once. And it prevents Optimistic ACKing attack since the chance for an attacker to correctly guess the nonce for the next data segment is negligible.
However, the above mechanism forces the sender to only accept complete ACKs, which might be an issue in real world network settings, where the IP layer might fragment packets and partial ACKs can be returned from valid receivers. To address this and to prevent the ACK Division attack at the same time, TCP logic could allow partial ACKs but should only increment the cwnd by the amount ACKed. On the other hand, the nonce mechanism involves changes to the protocol, which might be harder to be adopted in real life.
The followings are some of our interesting findings during the reproduction process:
(1) The above TCP vulnerabilities were found in RFC 2581, which was obsoleted by RFC 5681 that patches these security loopholes. Thus, the attacks presented were no longer valid on current OS networking stacks. But from our reproduction point of view, we found that the attack ideas do hold and the plots are valid given that we are attacking TCP per RFC 2581. We are sure that the paper was groundbreaking at the time of its publication (1999).
(2) Upon burst of packet loss, TCP Reno potentially needs a burst of retransmission timeouts to continue sending packets. TCP Reno’s fast recovery can only efficiently recover from single packet loss. In scenarios where two or more packets are lost, TCP Reno could resend the first missing packet upon the third duplicate ACK. Upon receiving the ACK that acknowledges all packets up to the second missing packet, the sender might still have an outstanding window size larger than the halved cwnd. However, since there are no more packets in the network and no more ACKs, the sender is forced to timeout (even multiple times) to resend all the remaining missing packets (cwnd is stuck at 1 * MSS after the first timeout). This issue is fixed in TCP NewReno, but since we are implementing TCP Reno, we had the opportunity to experience the exact issue that was addressed in TCP NewReno.
From the above reproduction, we saw that the attacks are indeed valid and could potentially be exploited even in modern careless implementation of TCPs (there are places like middle-boxes that might not use the default OS network stacks). Although the attacks were demonstrated back in 1999, it has shown the probability of having security loopholes even in official protocol designs. We hope that when designing new networking protocols, engineers nowadays should not only focus on performance metrics such as throughput and latency, but also have a closer look on the protocols’ security robustness. With security loopholes, protocols with extremely high throughput, low latency, and low memory usage could still be useless in production environments.
6. Reproduction Instructions
Please check out the instructions in our GitHub repo: https://github.com/kreamkorokke/cs244-final-project.
 Savage, Stefan, et al. “TCP congestion control with a misbehaving receiver.” ACM SIGCOMM Computer Communication Review 29.5 (1999): 71-78.
 Allman, M., Paxson, V., and W. Stevens, “TCP Congestion Control”, RFC 2581, DOI 10.17487/RFC2581, April 1999.