CS 244 ‘14: Mosh: An Interactive Remote Shell for Mobile Clients

Wei Shi and Sumitra Narayanan

The original paper, ‘Mosh: An Interactive Remote Shell for Mobile Clients’ by Keith Winstein and Hari Balakrishnan (of the MIT Computer Science and Artificial Intelligence Laboratory) can be viewed here: 

Click to access mosh-paper.pdf

Introduction

This paper introduces a remote shell called Mosh, an alternative to the popular Secure Shell (SSH). Mosh addresses several crucial performance issues of SSH in small-bandwidth and high-latency networks. Among these issues are connection loss due to intermittent connectivity, slow echoing of user keystrokes, and the lag in halting a runaway process with Ctrl+C.

Mosh is built on a novel UDP-based transport protocol called the State Synchronization Protocol (SSP) unlike SSH and other remote shells, which mostly run on TCP. Since UDP is designed for low-latency applications unlike TCP, this makes Mosh much faster than SSH. With SSP, user keystrokes on the sender side can be quickly and speculatively echoed before the server sends back updates of the terminal state by utilizing a regularly-updated ‘diff’ between local host and remote host states at the receiver side. Because it uses stateless UDP packets and tracks the client by its cryptographic credentials (also ensuring user security), SSP facilitates user roaming.

Motivation

We are heavy users of the traditional SSH and face its poor user experience on a daily basis, with frequent keystroke lags and frustrating connection losses eating away at our time. Thus, we are very interested to see what Mosh can offer and test it to see if it really works as advertised. This testing would be fairly simple since the client-server topology required to carry out this experiment is easy to emulate with Mininet. If we can successfully reproduce the paper’s results in our project, we will be glad to switch to Mosh from SSH.

Results achieved by the authors

According to the authors of this paper, Mosh is able to handle intermittent connectivity and changes in IP addresses as well as providing good interactive performance over long-delay network paths. The authors examined 40 hours of keystroke activity including 9,986 total keystrokes from 6 users (these traces included typical, day-to-day use of popular programs such as the bash and zsh shells, the alpine and mutt email clients, the emacs and vim text editors, the irssi and barnowl chat clients, the links text-mode Web browser, and several programs unique to each user). From this, they found that mean and median response times were drastically reduced on different types of connections, compared to SSH; these results can be seen in the graph in our Subset Goal section. Mosh achieved this improvement by accurately predicting the response to 70% of user keystrokes. Their favorable results lead them to conclude that SSP, on which Mosh is built, is an ideal protocol to gracefully handle the roaming and intermittent connectivity prevalent in mobile application use.

Subset Goal

From the paper, we chose to replicate Figure 2: Cumulative distribution of keystroke response times with Sprint EV-DO (3G) Internet service.

Subset Motivation

We chose to replicate Figure 2 (shown above) since it is an easy way to visualize Mosh’s effectiveness when compared to SSH (by way of its relatively quick response to user keystrokes, as seen in the figure).

Subset results

Our reproduction of Figure 2 shows that Mosh has a clear gain over SSH via 3G link. The shapes of both the Mosh and SSH curves match well with the result in the paper. One discrepancy is that in our graph, the median of the Mosh response times is much lower than in the paper. Most of these small response times are due to Mosh’s quick, speculative echoing; they have nothing to do with the network characteristics. Therefore, we believe that the smaller response times we obtained are due to our machine speed. The authors may have run the experiment on a relatively slower machine. Despite that discrepancy, we believe we have successfully reproduced the authors’ result.

Sensitivity analysis

Aside from reproducing Figure 2 from the paper, we also conducted two variations of the experiment (to see how well the authors’ assumptions on Mosh hold under different experimental conditions).

Mosh via WiFi with 10% loss rate

One of the typical uses of SSH is on a mobile laptop via flaky WiFi connectivity. For example, when doing work in a crowded classroom, SSH usually gets stuck due to packet drop via congested WiFi. We wanted to see whether Mosh could help mitigate this problem. Thus, we created a server-to-client link with WiFi characteristics, i.e., 10ms RTT and 10% loss rate. The resulting graph is shown below:

This graph shows that Mosh does not have a clear gain over SSH here; they both have a majority of response times under 50ms. We have also tried higher loss rates, e.g., 30% – 50%. In these cases, the replay of SSH/MOSH will often get stuck, preventing the creation of the final graph. Therefore, we conclude that Mosh does not help mitigate the packet loss problem too much. This conclusion also matches our observation of Mosh on real-world usage: It performs just as badly as SSH via a flaky connection, as can be seen in most public places like cafes or airports.

Mosh via 4G LTE

Since the paper was written before 4G LTE had been widely deployed, the experiment was carried out via the old 3G link. Now that 4G LTE is available in most areas and on most devices, we wanted to see if Mosh still has its advantages via a 4G LTE link. Thus, we created a server-to-client link with 4G LTE characteristics, i.e., 75ms RTT[1] and 50% jitter. The resulting graph is shown below:

This graph shows that Mosh still has a clear gain over SSH via a 4G LTE link. These curves follow a similar trend to those of the original 3G experiment. The majority of the Mosh keystroke responses are returned under 20ms, whereas SSH has to wait for the whole RTT to display the response. However, given that the average response time for both is around 75ms, we believe that human users will not perceive this difference in response times.

Addressing Challenges

1. Active/inactive states of 3G link

A typical 3G link has both active and inactive states. When a client initiates a connection over an inactive link, the RTT is much longer than that of an active link. We did not know of this dynamic link characteristic when we first emulated the result. Thus, we initially used a uniform link delay of 500ms, which is what the paper uses (since almost all SSH delays are 500ms). However, with such a uniform delay, we obtained a graph that differed from Figure 2 in the paper; our plot looked like a step function. After we discovered this unique characteristic of 3G links, we used a delay of 500ms along with a jitter of 250ms to emulate the effect of a link with both active and inactive states. The resulting graph, as can be seen, is much closer to what the authors obtain.

2. Mosh 1.2.3 cannot run headlessly

We started by installing Mosh using apt-get. However, that binary package is outdated. We tried to start Mosh using host.cmd() in Mininet, but this causes the Mosh server to crash. This is because cmd() in Mininet does not create an actual TTY. This issue has been fixed in Mosh 1.2.4-dev. Therefore, to run the emulation, we needed to install Mosh from the authors’ ppa instead of Ubuntu mirror.

3. Replay script does not exit properly

The replay script from the author’s repository has a bug that causes it not to exit after the log is played. This stops our emulation from running other tests. We spent some time debugging this problem, and finally had to patch the client-side replay script.

Critique: Critique on the main thesis that you decided to explore – does the thesis hold well, has it outdated, does it depend on specific assumptions not elucidated in the original paper?

The authors’ thesis, “Mosh (mobile shell) is a remote terminal application that supports intermittent connectivity, allows roaming, and speculatively and safely echoes user keystrokes for better interactive response over high-latency paths”, holds well. As seen from our plot, Mosh is able to quickly and speculatively echo user keystrokes under the authors’ conditions for delay and bandwidth (we emulated on Mininet the Sprint EV-DO (3G) Internet service used by the authors in their experiment).

However, as mentioned earlier in our ‘Challenges’ section, the authors mentioned only the delay (500ms), leaving out the jitter inherent in the Sprint EV-DO (3G) links on which they tested. Lacking this information, we later found out about the active vs inactive states of 3G links and figured out that a jitter of 250ms was required to reproduce their graph.

Platform: Choice of your platform, why that particular choice and how reproducible is the entire setup, what parameters of the setup you think will affect the reproducibility the most?

We chose to replicate the authors’ results in Mininet. With Mininet, it is extremely easy to set up links to emulate the Sprint EV-DO (3G) links used by the authors when carrying out the experiment. All we needed to do was to set up the client node to run the Mosh client, server node to run the Mosh server, and importantly, specify the appropriate delay, jitter, and bandwidth parameters of the client-server link. With these steps, the whole experimental setup was easily reproduced with just several lines of code.

Overall, Mininet experiments can be easily run using simple Python scripts. This made it easy to automate the various steps of our experiment: the process of obtaining the user keystroke log by emulating a Mosh client and server, dumping the delays of all user keystroke echoes to a file, and parsing the file to reproduce the authors’ graph.

README: Instructions on setting up and running the experiment.

A markdown version of the README is available on Github: https://github.com/weishi/cs244-mosh/blob/master/README.md.

Setting up the instance

We used the CS244-Spr14-Mininet image as a starting point. Basically, you need to follow these steps:

1. Create a community instance image: ami-e2b6ded2 (EC2 region: Oregon)
2. Choose m3.medium for Instance Type
3. Open port 22 for SSH
4. **IMPORTANT** Open all UDP ports by adding the rule ‘All UDP’ with source ‘Anywhere’
5. Pick your SSH key pair
6. Launch & login

If you are unclear, follow the guide http://www.stanford.edu/class/cs244/ec2.html, and remember to open all UDP ports.

Running the experiment

Get the source code and run the script:

git clone https://github.com/weishi/cs244-mosh.git
cd cs244-mosh
sudo ./run.sh

The script will first try to install any dependency packages. The script will run 3 sets of experiments. The first one will reproduce Figure 2 from the paper. The other two are for sensitivity analysis: The second one will run the emulation over a WiFi link with 10% loss rate and the third one will run the emulation over a 4G LTE link.

The first replay (SSH/MOSH) may take up to 15min because it uses a large dataset. The other two may take up to 5min. They use a smaller dataset because their loss rates will significantly increase the emulation time. In the worst case, all tests together might take about 30-40min to run.

Sometimes (very rarely), the second replay (sensitivity analysis 1) may get stuck due to the randomness of loss. The problem usually goes away after restarting the script. To see the progress, you can monitor stdoutSSH.txt/stdoutMOSH.txt, etc.

Result

The result graphs are created in the ‘mosh-{timestamp}’ directory.

• graph.png: the main result that should match Figure 2 in the paper.
• graph_sen1.png: the result for sensitivity analysis 1.
• graph_sen2.png: the result for sensitivity analysis 2.

Any feedback on Mininet/EC2 hurdles or any other third-party code you used, or any other suggestions/rants?

We find that Mininet is not well documented; most of the information is buried in the tutorial. The generated Python API reference manual does not contain much useful information. We had some trouble figuring out various parameters in the addLink() method. It is also not clear if the parameter expects a string with a plain number or a number along with units. We ended up digging into the source code of Mininet and working our way through trial and error.

References

[1] 4G RTT from this paper:

Click to access P1_ychen_measuring_cellular_networks_diversity.pdf