Shipping code updates to the Facebook app, which is used every day by hundreds of millions of people, requires extensive testing to ensure stability and performance. At Facebook’s scale, this process requires checking hundreds of important interactions across numerous types of devices and operating systems for both correctness and speed. Traditionally, this has largely been a manual test design process, during which engineers devote time and resources to designing test cases. But at Facebook, we have developed an intelligent software testing tool called Sapienz to efficiently and effectively design many of the test cases we need. There are undoubtedly still test cases best designed by human engineers, because of their superior domain knowledge compared with a machine’s. But our work with Sapienz shows it is possible to automate much of the tedious, time-consuming process and accelerate the deployment of new features.
Sapienz technology, which grew out of decades of work by the software engineering research community, leverages automated test design to make the testing process faster, more comprehensive, and more effective. This is the first time a search-based automated test design technology has been deployed at this scale into continuous integration. Its rapid deployment, successful intervention rates, and reporting precision have pleasantly surprised both the engineers who designed it and those who are using it for friction-free fault finding.
Sapienz samples the space of all possible tests, using intelligent computational search and an approach called search-based software testing. An important design principle is that Sapienz tests through the UI, so issues Sapienz reports to engineers can be found through the UI. This avoids the false positives that bedevil many test design approaches, but it makes it more challenging to provide guidance to the computational search.
As Sapienz searches, it builds a model of the system under test, through the UI interactions, and harvests good tests for subsequent reuse. Sapienz uses a hybrid approach to search, combining lower-level individual events with higher-level patterns of related events (known as motif events), that captures a connection of related lower-level events into a higher-level motif gene.
We first deployed Sapienz at Facebook in September 2017, and have since scaled it to meet the challenge of testing millions of lines of code in Facebook’s Android app.
Sapienz now automatically designs, runs, and reports the results of tens of thousands of test cases every day on the Facebook Android app. In the first few months since its deployment, the technology has allowed engineers to fix issues within hours (sometimes within minutes) of the code being written. This has had a positive impact on hundreds of millions of people around the world, bringing them new features and enhancements as quickly as possible, without compromising stability or performance.
In addition to speeding up the testing process, Sapienz has delivered an extremely low false-positive rate. Our engineers find that 75 percent of Sapienz reports are actionable, resulting in fixes. Sapienz uses Facebook’s One World platform to scale to many hundreds (and even thousands) of emulators at any one time, in continuous deployment 24 hours a day, seven days a week.
Engineers have benefited from the friction-free way in which Sapienz finds code issues. It checks new code nearly in real time, reports straight into Facebook’s continuous integration system, Phabricator, and adds comments just as a human engineer would. It designs tests to expose potential faults without the testing engineer’s having to intervene at all, making Sapienz unique in its level of friction-free code-checking.
We are just at the beginning of our journey with Sapienz at Facebook and expect over the next few years to implement and share new applications and feature enhancements. We also expect that search-based software engineering will find many new exciting and innovative applicators, both here at Facebook and in the wider tech sector as a whole.
The Sapienz team is strongly committed to increasing engagement between fundamental scientific work and practical industrial-scale software engineering. The tool originated as part of a research paper but moved to Facebook-scale deployment in just 17 months, illustrating how academic researchers and engineers can collaborate to rapidly improve real-world applications used by billions of people.
We are currently extending Sapienz to test other Android and iOS apps in the Facebook product family.
