When we launch a new app or feature on Facebook, we usually do it from a powerful wireless network at our offices in North America. These networks are typically what we use to test and build services. However, for a large percentage of people around the world, accessing Facebook requires connecting to a slower, less reliable wireless network. We want as many people as possible to be able to access our services at their full potential. So we need to be able to test our features on wireless connections that more accurately reflect the types of connections people on Facebook will ultimately use.
In January 2013, we read a 2600 magazine article about a group of hackers at DEF CON who had used open source technology to create a network for mobile phones at the convention and distributed phones to attendees. We wanted to take certain elements of this idea and adapt it to create networks of varying strength that we could use to optimize our apps for those conditions and allow our software engineers to empathize with people using Facebook in less-than-ideal network environments.
First attempt: re-creating a 2G network
We decided to test our idea at an infrastructure Hackathon. We bought two old, secondhand base stations and some mobile phones to start, and spread the word that we were looking for other people to join our effort.Right away, more engineers signed up to help, and after getting a solid start at the Hackathon, we acquired a good understanding of the cellular software protocols by configuring the software components and analyzing how they communicated with one another. We continued meeting for weekly hack sessions and soon were able to make voice calls and send SMS messages and transmit Internet data. For two months, we worked on simulating a 2G connection, but it proved to be problematic. Acquiring reasonably priced cellular radios that would reliably run open source base station software was time-consuming.
Pivoting to Wi-Fi
It was at another infrastructure Hackathon in May 2013 that we decided to shift our focus. We had done a good deal of the work needed to create a new network, but we didn’t have an easy, efficient way to make it functional. So we pivoted our efforts from creating a 2G network to building a Wi-Fi network that could be set to different simulation speeds by the engineers using it.
Using the measurements and findings from our 2G project, we created a set of simulations with our project and a user interface with Django and Bootstrap. After we demo’d our work during the Infra Hackathon Forum, an engineer from our corporate network team contacted us and helped us put that early version of our effort into the Facebook campus wireless network so every engineer could use it. The result was the start of what we ultimately called Augmented Traffic Control (ATC), and it was our biggest step toward creating test network conditions that simulated what people would most likely experience on different kinds of networks.
In the following weeks, we set up Augmented Traffic Control so it could simulate 2G, Edge, 3G, and LTE networks. We specifically simulated the most common network connectivities in countries such as Brazil, India, Indonesia, Kenya, Nigeria, and the Philippines.
We kept up with our weekly hack sessions through August, and in the fall we were able to begin evangelizing Augmented Traffic Control to internal teams whose work was affected by network conditions. As we continued working on Augmented Traffic Control, we started getting requests from other engineers to use it. Once we had something we could share with them, we wanted to get their feedback.
Augmented Traffic Control in practice
The other teams that used ATC found that it helped save time and pointed out weaknesses in their work. The team working on an update to Messenger that would make the app run faster and more reliably used ATC to check how things would run when the network connection wasn’t as strong. They ran automated tests in which the bandwidth connections were changed in order to look at how long Messenger should wait before timing out and how many retries were optimal to send messages. This information allowed the Messenger team to choose a selection of small A/B tests to run. By zeroing in on where the weaknesses were in different networks, they were able to more quickly identify problems and save development time. The Messenger team also uses ATC APIs in automation to measure call quality on different network configurations.
Engineers who work on download functionality found ATC useful because it was able to reduce their use of A/B tests. They were able to quickly test the way that code behaved when attempting to download files across different network strengths, and used ATC to quickly re-create the results to ensure it was a consistent behavior. With this information, they could improve downloads to make them more adaptable and flexible to the different network strengths people on Facebook encounter.
In some cases, developers were able to re-create issues that would happen only on extremely slow networks, and then investigate and handle those conditions correctly.
This pattern was repeated across many teams that had previously made extensive use of A/B testing. Often, a new app or feature will undergo testing that requires daily tracking for desktop. For mobile, that testing can take weeks or months for updates to ship and for comparisons to be valid. For testing across network strength, this can be even more time-consuming. With Augmented Traffic Control, we can instead switch easily from one connection strength to another and demo the feature in real time on different network simulations. Because ATC is built into the Wi-Fi network our developers have been able to use it to test IOS, Android and feature phones.
Open source
We created Augmented Traffic Control with open source technology, building on the work of others. We want to give the open source community the same chance to improve on our ideas and innovate with their own — so today we are open-sourcing our design for Augmented Traffic Control on GitHub.
Thanks to the following people, who contributed to Augmented Traffic Control across its lifetime: Alessandro Salvatori, Ameesh Goyal, Andrew Pope, Anthony Gargiulo, Chris Vander Mey, Emmanuel Bretelle, Gus Luxton, John Morrow, Patrick Shuff, and Zeal Jagannatha.
