Data distillation makes omni-supervised learning possible on, Facebook's Engineering blog


An investigation of omni-supervised learning, a type of semi-supervised learning that uses a combination of data that’s been manually labeled for training purposes (supervised data) and unlabeled data (unsupervised data) and offers the potential to outperform state-of-the-art fully supervised methods. The proposed approach uses data distillation, a simple method of omni-supervision, to train an AI model.


Data distillation is based on the classic idea of self-training, or making predictions on unlabeled data and using them to update the model. In a similar way, the model uses predictions to fill in the gaps in its own training data. The researchers developed a four-step sequence: training a model on a large amount of supervised data; applying the trained model to unsupervised data; generating labels for that unsupervised data; and, finally, incorporating both the supervised and (now automatically labeled) unsupervised data to go back and retrain the model. They found that models trained with data distillation performed better than those trained with supervised data alone.


These experiments prove that it is possible for omni-supervised learning to surpass the results found with large-scale supervised learning. Combining more traditional curated training data with a large amount of unlabeled data is a viable strategy and opens up the possibility of drawing information from real-world sources to speed the creation of AI systems.


Data Distillation: Towards Omni-Supervised Learning

To help personalize content, tailor and measure ads, and provide a safer experience, we use cookies. By clicking or navigating the site, you agree to allow our collection of information on and off Facebook through cookies. Learn more, including about available controls: Cookies Policy