Since Facebook’s first line of PHP, and its first MySQL INSERT statement, open source has been a huge part of our engineering philosophy.
Nowadays, we use, maintain, and contribute to a significant number of major projects – in areas as diverse as native mobile tools, big data systems, client-side web libraries, backend runtimes and infrastructure, and, through the Open Compute Project, server and storage hardware.
2013 has been a great year for our open source program, with a significant number of new projects that we’re really proud of, a renewed commitment to run and maintain them actively, and a desire to work with the vibrant communities that have built up around them. On our GitHub account alone, we now have more than 90 repos comprising over 40,000 commits and that have collectively been forked 15,000 times.
The end of the year is a great opportunity to look back at some of our major areas of investment, and recap a (non-exhaustive!) list of the projects we’ve been working on.
Mobile
The open source contributions we’re the most proud of are often the technologies we’ve had to develop to meet the scale and performance challenges we experience at Facebook.
Mobile has been no different: Facebook’s mobile apps have been a focus for our product teams this year, and we have yet again needed to develop new capabilities – in this case, those required to move fast, and compile, test, and release our mobile apps ever more efficiently. It’s been natural to open source our work as we’ve done so, and these tools have become the heart of our mobile open source portfolio.
In April, for example, we released Buck, our Android build tool. Developer efficiency is important for us, and naturally speed is Buck’s primary focus: At Facebook, we found it to be more than twice as fast as Ant when building our apps. Since launch, the project has gone from strength to strength and is used throughout the Android community as well as for a number of notable Java projects.
For iOS, we had a similar bottleneck, and also found ourselves needing to accelerate and automate the manual Xcode build process. The result was the extremely popular xctool – also released in April – which makes it easier for engineers (and not to mention continuous integration machines) to compile and test iOS and Mac projects.
We launched Rebound, a physics and animation library for Android, at our Mobile @ Scale event in October. Will Bailey wrote up the project in this recent blog post, and we believe that modeling real world physics is a powerful way to create natural and tactile animations and interactions within apps.
And finally, is there a better way to finish off a great year of mobile tools than by releasing another? Today we’re thrilled to be releasing Origami, a toolkit for Quartz Composer that allows designers to build and prototype mobile interactions faster than ever.
Web
Web technologies remain relevant for Facebook too, on both mobile and desktop clients. On the front end, much of our open source focus has been on supporting our fast and flexible JavaScript library React, which we launched at JSConf in May.
Since then, the React library – and the enthusiasm from its community – has developed considerably. There are now numerous integrations into build tools, server-side environments (like node, Rails, and Python), and other client libraries – as well as a number of high-profile external deployments such as Khan Academy. The team maintains an excellent community round-up blog with many examples of other React projects, demos and tutorials.
We also want to help move both the JavaScript language and the quality of web apps forward in general. Regenerator, for example, is a transformer that brings ECMAScript 6’s yield syntax to today’s browsers, and Huxley, built by the Instagram team in the summer, has quickly become a hugely popular visual regression test tool for web apps.
Data
Facebook has a really strong history of working with open source data infrastructure, stretching back to our contributions to MySQL, Cassandra, Hadoop, Hive and Hbase. 2013 has been no exception, and just in the last few months we’ve launched two new flagship data projects.
The first of these, Presto, is a new distributed SQL query engine, designed for high performance analytics that we use to run interactive queries against our 300PB data warehouse. We announced Presto at our summer Analytics @ Scale event and wrote more about it last month on our blog. Since then we’ve been thrilled to see it adopted by companies like Airbnb and Dropbox, and to have contributions from a diverse and enthusiastic community, including new clients, an Ansible playbook, and debian packaging.
The second project, RocksDB, is data infrastructure of a very different sort: an embedded key-value store library based on LevelDB, and optimized for environments with many CPU cores and fast flash storage. Again, in the few months since its launch, there’s been strong interest in the project, and some great community work already on bindings for other languages.
Our contributions and commitment to data infrastructure projects have stretched well beyond our own too. We’ve made a strong commitment to Giraph, for example, and have scaled it to support graph structures with over a trillion edges.
Infrastructure
Last, but absolutely not least, infrastructure projects remain one of the most important parts of our open source efforts.
HHVM, the HipHop Virtual Machine, is by far the most significant and followed project in our portfolio, and has enjoyed huge support from the PHP ecosystem. 2013 has seen almost 4,000 commits to the project, and great strides made in terms of performance and compatibility with third-party PHP frameworks, important for broader community adoption.
The team has just emerged from a lockdown and yesterday shared news of the progress they’ve made. We’re also excited to have seen the VM integrated into Travis CI for continuous integration tests, and to have added FastCGI support for integration with other popular web servers.
We’ve worked on many other notable infrastructure projects this year too. Of our own, these include pfff, our code analysis toolbox, libPhenom, a high performance eventing framework, and folly, our popular C++ library. We’ve also made significant contributions to Mercurial, LLVM, and GNU grep throughout the year.
The Open Compute Project also continued to go from strength to strength in 2013, with new work on networking hardware, new community and foundation management, and numerous hackathons and workshops throughout the year. Facebook is working hard to help support the project and its goal to develop open source server and data center designs for all kinds of data center technologies – and we’re looking forward to the Open Compute Summit next month.
Our open source program in 2013…
When you’re using, or contributing to, an open source project, we know there’s nothing worse than seeing it stagnate: bugs unaddressed, issues left open, pull requests neglected. Our goal at Facebook is to ensure we’re maintaining strong and responsible community engagement across our portfolio.
We’ve taken some very specific steps to do this. For example, we now encourage our engineering teams to work on open source projects in the open by default, using GitHub, Bitbucket or Apache Software Foundation projects as the source of truth.
We have a new toolchain that helps us keep these repos in sync with our internal systems, code review process, task tracking and so on. And we’ve recently instrumented every repo to ensure they are all being kept healthy: We have internal dashboards of the number of commits, pull requests, and issues for each team. This provides us an early warning system for any communities we are inadvertently neglecting.
We’re also proud to announce that we’re working with Bountysource to help out our communities and motivate and reward involvement in various open source projects. We’ve already provided a range of bounties on projects including HHVM and D.
And just last month, we announced the Facebook Open Academy to help bring open source to computer science curricula at universities around the world.
In 2013, we also started the “@ Scale” program of engineering events, which are designed to bring together developers to talk about the challenges of implementing various technologies at scale and share their solutions and any resulting open source projects. Events this year have included Analytics @, Mobile @, and Data @ Scale, and videos from each are available on our Facebook Developers channel. Stay tuned for many more @ Scale events in the New Year.
And finally, welcome to our new engineering web site, code.facebook.com! We want to make it as easy as possible for you to stay up to date with all our activities, and this is where you’ll find all our engineering blog posts, videos, academic publications – and of course open source projects. Subscribe to our Facebook Page and our open source news for other regular updates.
…and beyond
As the famous Facebook maxim goes, our open source program is still only 1% finished.
We know there is still a huge amount to work on, across each of the major themes above. We’re very lucky to have strong and enthusiastic communities on our projects, and with that comes great responsibility.
Whether you’re in the mobile, web, data or infrastructure communities, we look forward to continuing to working with you all – and we’ll see you in 2014!
