Is AI currently capable of identifying wild oysters? A comparison of human annotators against the AI model, ODYSSEE

Brendan Campbell^1*Alan Williams^2*Kleio Baxevani^1,3*Alyssa Campbell^1,4Rushabh Dhoke⁵Rileigh Hudock¹Xiaomin Lin^6,7Vivek Mange^5,6Bernhard Neuberger⁸Arjun Suresh⁶Alhim Vera⁹Arthur C Trembanis¹Herbert Glenn Tanner⁵Edward Hale^1,4

• ¹School of Marine Science and Policy, College of Earth, Ocean, and Environment, University of Delaware, Newark, Delaware, United States
• ²Laboratory of Horn Point, Center for Environmental Science, University of Maryland, College Park, Cambridge, Maryland, United States
• ³Department of Aerospace Engineering, A. James Clark School of Engineering, University of Maryland, College Park, College Park, Maryland, United States
• ⁴Sea Grant Delaware, University of Delaware, Lewes, Delaware, United States
• ⁵Center for Autonomous and Robotic Systems, College of Engineering, University of Delaware, Newark, Delaware, United States
• ⁶Maryland Robotics Center, A. James Clark School of Engineering, University of Maryland, College Park, College Park, Maryland, United States
• ⁷Institute for Assured Autonomy, Whiting School of Engineering, Johns Hopkins University, Baltimore, Maryland, United States
• ⁸University of Applied Science Technikum, Department of Industrial Engineering, Wien, Austria
• ⁹Department of Aerospace Engineering and Engineering Mechanics, CEAS, University of Cincinnati, Cincinnati, Ohio, United States

Oysters are ecologically and commercially important species that require frequent monitoring to track population demographics (e.g. abundance, growth, mortality). Current methods of monitoring oyster reefs often require destructive sampling methods and extensive manual effort. Therefore, they are suboptimal for small-scale or sensitive environments. A recent alternative, the ODYSSEE model, was developed to use deep learning techniques to identify live oysters using video or images taken in the field of oyster reefs to assess abundance. The validity of this model in identifying live oysters on a reef was compared to expert and non-expert annotators. In addition, we identified potential sources of prediction error. Although the model can make inferences 1 Campbell et al.significantly faster than expert and non-expert annotators (39.6 s, 2.34 ± 0.61 h, 4.50 ± 1.46 h, respectively), the model overpredicted the number of live oysters, achieving lower accuracy (63%) in identifying live oysters compared to experts (74%) and non-experts (75%) alike. Image quality was an important factor in determining the accuracy of the model and the annotators. Better quality images improved human accuracy and worsened model accuracy.Although ODYSSEE was not sufficiently accurate, we anticipate that future training on higher-quality images, utilizing additional live imagery, and incorporating additional annotation training classes will greatly improve the model's predictive power based on the results of this analysis. Future research should address methods that improve the detection of living vs. dead oysters.