We recently hosted the third annual ZooKeeper Meetup@Facebook in Menlo Park, featuring technical talks focused on new performance, scalability, and security features implemented at large-scale companies with complex distributed systems. Events such as these are important for building community among peers across the industry. Speakers from Cloudera, Facebook, Salesforce, San José State University, and Twitter discussed how they are pushing the limits of ZooKeeper performance and scale, and how they are securing ZooKeeper instances for their organizations.
ZooKeeper@Facebook: Performance
Fangmin Lyu and Jie Huang, Software Engineers, Facebook
To kick off the event, Fangmin shared how Facebook’s latest patches have improved the write throughput on ZooKeeper by as much as 10x, thanks to changes made to the snapshot scheduler and the addition of batch commits support. Jie discussed a method for optimizing semaphores, a popular use case for ZooKeeper, via a new lock API that reduces server round trips and queries per second (QPS), thereby improving performance and throughput.
Watch Fangmin’s full talk.
Watch Jie’s full talk.
ZooKeeper@Facebook: Scalability
Vladimir Ivic and Yisong Yue, Software Engineers, Facebook
Vladimir and Yisong began the scalability portion of the talks with a presentation on improving ZooKeeper’s multitenancy features by implementing demand control, which introduces data size and request rate quotas for ZooKeeper namespaces and client IDs. Yisong discussed how these features help reduce or eliminate noisy neighbor situations. Vladimir walked through how we migrate users from one ZooKeeper ensemble to another for better scalability with zero downtime.
Watch Vladimir’s full talk.
Watch Yisong’s full talk.
ZooKeeper@Facebook: Containerization
Christopher Bunn, Production Engineer, Facebook
Christopher concluded the Facebook talks by discussing how our entire ZooKeeper fleet has been migrated to our in-house containerization system, doing away with our bare-metal deployments. He shared some of the lessons we learned along the way, as well as how this migration has simplified our fleet management and deployments.
Watch Christopher’s full talk.
Upgrading to ZooKeeper 3.5 at Cloudera
Máté Szalay-Bekő, Staff Engineer, Cloudera
Based in Budapest, Máté introduced the community talks with a remote presentation on Cloudera’s upgrade to ZooKeeper 3.5. After Cloudera and Hortonworks merged last year, the new Cloudera Data Platform retained the best of both former distributions — and in many ways much more. One example is ZooKeeper 3.5.5, which is now part of Cloudera’s latest release. Máté summarizes the journey to integrate and test ZooKeeper 3.5.5 with more than 13 different open source components.
Watch Máté’s full talk.
ZooKeeper security at Salesforce
Fiaz Hossain, CTO of Site Reliability, Salesforce
Suhas Dantkale, Lead Software Engineer, Salesforce
Fiaz and Suhas shared the work they’ve done on Federal Information Processing Standard (FIPS) 140-2 compliance and ZooKeeper C Client Binding Enhancements at Salesforce.
Watch Fiaz’s full talk.
Watch Suhas’s full talk.
ZooKeeper@Twitter
Michael Han, Software Engineer, Twitter
Michael discussed the experience and lessons learned while upgrading, running, and maintaining Twitter’s entire ZooKeeper fleet on top of the Apache ZooKeeper master branch. He also talked about the systems Twitter has built around ZooKeeper.
Watch Michael’s full talk.
Historical performance of ZooKeeper
Jonathan Wong and John Tran, former students, San José State University
Current and former students of the systems group at San José State University run by Ben Reed (a founder of ZooKeeper) presented their work on tracking ZooKeeper performance changes over time. Their goal is to reproduce the reference benchmark with the latest codebase (updating from 3.2) and run that across different releases to see how performance has changed and why. By automatically benchmarking new pull requests, they will attempt to understand the performance impact of proposed changes.
Watch Jonathan’s full talk.
Watch John’s full talk.
Follow us on our Facebook Meetup page for more updates and info on our next event.
