Larry Zitnick and Nafissa Yakubova from the Facebook Artificial Intelligence Research (FAIR) group collaborated with Daniel Sodickson, M.D., Ph.D, Michael Recht, M.D., and Yvonne Lui, M.D., from NYU School of Medicine on this work.
Facebook and NYU School of Medicine’s Department of Radiology are excited to announce fastMRI, a new collaborative research project that will investigate the use of artificial intelligence (AI) to make magnetic resonance imaging (MRI) scans up to 10 times faster. If this effort is successful, it will make MRI technology available to more people, expanding access to this key diagnostic tool.
MRI scanners provide doctors and patients with images that typically show a greater level of detail related to soft tissues — such as organs and blood vessels — than is captured by other forms of medical imaging. But they are relatively slow, taking anywhere from 15 minutes to over an hour, compared with less than a second or up to a minute, respectively, for X-ray and CT scans. These long scan times can make MRI machines challenging for young children, as well as for people who are claustrophobic or for whom lying down is painful. Additionally, there are MRI shortages in many rural areas and in other countries with limited access, resulting in long scheduling backlogs. By boosting the speed of MRI scanners, we can make these devices accessible to a greater number of patients.
Sufficiently accelerated MRI devices could also reduce the amount of time patients must hold their breath during imaging of the heart, liver, or other organs in the abdomen and torso. Increased speed could let MRI machines fill the role of X-ray and CT machines for some applications, allowing patients to avoid the ionizing radiation associated with those scans.
Challenges in advanced medical imaging
This project will initially focus on changing how MRI machines operate. Currently, scanners work by gathering raw numerical data in a series of sequential views and turning the data into cross-sectional images of internal body structures that doctors then use to evaluate a patient’s health. The larger the data set to be gathered, the longer the scan will take.
(L) Raw MRI data before it’s converted to an image. To capture full sets of raw data for a diagnostic study, MRI scans can often take 15-60+ minutes. (R) MRI image of the knee reconstructed from fully sampled raw data.
Using AI, it may be possible to capture less data and therefore scan faster, while preserving or even enhancing the rich information content of magnetic resonance images. The key is to train artificial neural networks to recognize the underlying structure of the images in order to fill in views omitted from the accelerated scan. This approach is similar to how humans process sensory information. When we experience the world, our brains often receive an incomplete picture — as in the case of obscured or dimly lit objects — that we need to turn into actionable information. Early work performed at NYU School of Medicine shows that artificial neural networks can accomplish a similar task, generating high-quality images from far less data than was previously thought to be necessary.
In practice, reconstructing images from partial information poses an exceedingly hard problem. Neural networks must be able to effectively bridge the gaps in scanning data without sacrificing accuracy. A few missing or incorrectly modeled pixels could mean the difference between an all-clear scan and one in which radiologists find a torn ligament or a possible tumor. Conversely, capturing previously inaccessible information in an image can quite literally save lives.
(L) Raw MRI data that has been under-sampled. The MRI scan to capture this data was conducted more quickly than one to capture full data for a diagnostic study, but under-sampling generates noise and artifacts in the resulting MRI image. (R) MRI image of the knee reconstructed from subsampled data. The fastMRI project seeks to use AI to create useful MRI images, without noise and artifacts like those shown here.
The Facebook and NYU collaboration
NYU School of Medicine, a department of NYU Langone Health, has a long-standing history of pushing the boundaries of medical research and education to benefit the lives of patients. The Radiology Department’s Center for Advanced Imaging Innovation and Research (CAI²R) includes a multidisciplinary team of engineers, physicists, mathematicians, radiologists, and other clinicians and scientists with key expertise in rapid image acquisition, parallel imaging, and advanced image reconstruction. Their focus is on developing novel imaging technologies and rapidly translating those technologies into clinical practice.
NYU School of Medicine’s Department of Radiology chair Michael Recht, M.D., Daniel Sodickson, M.D., Ph.D. vice chair for research and director of the Center for Advanced Imaging Innovation and Research and Yvonne Lui, M.D., director of artificial intelligence, watch an MRI exam take place with a technician at NYU Langone Health in August, 2018.
Since 2016, CAI²R investigators have been pursuing the idea of utilizing AI to achieve faster MRI scans. Early studies have suggested that scan times could be reduced by an order of magnitude or more. Realizing those potential gains, however, would require additional AI knowledge and large-scale computing resources.
Around the same time, the Facebook Artificial Intelligence Research (FAIR) group, which focuses on open and foundational research that advances the state of AI, was looking for projects in which AI could have significant real-world impact. CAI²R’s image reconstruction work fit those criteria and provided FAIR with an opportunity to combine its deep learning expertise — particularly in the field of computer vision — and its ability to train models at large scale with the medical school’s leading imaging science expertise.
Project data details
The imaging data set used in the project, collected exclusively by NYU School of Medicine, consists of 10,000 clinical cases and comprises approximately 3 million magnetic resonance images of the knee, brain, and liver.
All data, including both images and raw scanner data, are fully stripped of patient names and all other protected health information. The work is fully HIPAA-compliant and approved under NYU Langone’s Institutional Review Board, which oversees all human subject research at the medical center. The project is governed by strict human subject data protection protocols and supported by the world-class information technology team at NYU Langone.
The magnetic resonance images (which generally represent small targeted regions of anatomy) used for this project have been scrubbed of any potential distinguishing features. Comparisons of the performance between AI-based reconstructions and traditional reconstructions will, likewise, be devoid of any identifying information. No Facebook data of any kind will be used in the project.
Recht, Sodickson, and Lui examine MRI scans of a knee at NYU Langone Health in August 2018.
Advancing the AI and medical communities
Unlike other AI-related projects, which use medical images as a starting point and then attempt to derive anatomical or diagnostic information from them (in emulation of human observers), this collaboration focuses on applying the strengths of machine learning to reconstruct the most high-value images in entirely new ways. With the goal of radically changing the way medical images are acquired in the first place, our aim is not simply enhanced data mining with AI, but rather the generation of fundamentally new capabilities for medical visualization to benefit human health.
In the interest of advancing the state of the art in medical imaging as quickly as possible, we plan to open-source this work to allow the wider research community to build on our developments. As the project progresses, Facebook will share the AI models, baselines, and evaluation metrics associated with this research, and NYU School of Medicine will open-source the image data set. This will help ensure the work’s reproducibility and accelerate adoption of resulting methods in clinical practice.
What’s next
Though this project will initially focus on MRI technology, its long-term impact could extend to many other medical imaging applications. For example, the improvements afforded by AI have the potential to revolutionize CT scans as well. Advanced image reconstruction might enable ultra-low-dose CT scans suitable for vulnerable populations, such as pediatric patients. Such improvements would not only help transform the experience and effectiveness of medical imaging, but they’d also help equalize access to an indispensable element of medical care.
We believe the fastMRI project will demonstrate how domain-specific experts from different fields and industries can work together to produce the kind of open research that will make a far-reaching and lasting positive impact in the world.
