AUTHOR=van den Bergh Jarrett , Chirayath Ved , Li Alan , Torres-Pérez Juan L. , Segal-Rozenhaimer Michal 

TITLE=NeMO-Net – Gamifying 3D Labeling of Multi-Modal Reference Datasets to Support Automated Marine Habitat Mapping

JOURNAL=Frontiers in Marine Science

VOLUME=Volume 8 - 2021

YEAR=2021

URL=https://www.frontiersin.org/journals/marine-science/articles/10.3389/fmars.2021.645408

DOI=10.3389/fmars.2021.645408

ISSN=2296-7745

ABSTRACT=NASA NeMO-Net, the Neural Multimodal Observation and Training Network for global coral reef assessment, is a convolutional neural network (CNN) that generates benthic habitat classification maps for coral reef and other shallow marine ecosystems from 2D satellite and 3D airborne remote sensing imagery. Training CNNs with high-accuracy labels for automated 2D and 3D semantic segmentation is a challenging task in machine learning. To overcome this big data challenge, we present a novel 3D online classification video game for mobile and desktop devices. 
Leveraging the power of citizen science, the NeMO-Net video game is able to generate high-resolution 3D benthic habitat labels at an unprecedented scale. The NeMO-Net video game trains users to accurately identify coral reef families and semantically segment 3D scenes captured using NASA FluidCam, the first remote sensing system capable of mitigating refractive ocean wave distortion. An active learning framework is used to allow users to rate and edit other user classifications. Data labels from the game are used to train the NeMO-Net CNN to autonomously map shallow marine systems, significantly augmenting satellite habitat mapping accuracy in these regions.
We share the NeMO-Net approach to user training and retention, outline the 3D labeling technique developed to accurately label complex coral reef imagery, and present preliminary results from over 70,000 user classifications as well as criteria for evaluating and filtering user data, a vital step in overcoming the inherent variability of citizen science. Finally, we examine how future citizen science and machine learning approaches might benefit from label training in 3D space using an active learning framework.
Within 7 months of its launch, NeMO-Net has reached over 300 million people globally and enabled a new generation to directly participate in a scientific campaign, uninhibited by geography, language, or physical ability. As the NeMO-Net video game reaches the needed training threshold for the NeMO-Net CNN, anticipated in early 2021, it will help produce the first cm-scale trained global shallow marine habitat mapping products later in 2021. These multimodal, multidecadal, high-resolution global data products will enable novel conservation applications by the UN (SDG 14), IUCN, federal, state, indigenous, and non-profit organizations.